2247
MR-to-CT Synthesis in MR-only Radiotherapy Based on Deep Learning1Shanghai Jiao Tong University, Shanghai, China, 2Fudan University Shanghai Cancer Center, Shanghai, China
Synopsis
Keywords: Analysis/Processing, Liver, MR radiotherapy, medical image synthesis，
Motivation: MRI-only radiotherapy requires synthesizing MR images into CT-equivalent images to calculate the radiation dose. However, the current synthesis methods are limited when applied to small anatomical regions, such as tumors.
Goal(s): Our goal was to develop a novel MR-to-CT synthesis algorithm that produces better results for small anatomical structures.
Approach: We introduced a multi-branch hybrid perceptual generative model incorporating an attention mechanism to synthesize different scales anatomical structures.
Results: Our proposed algorithms yield favorable results for small anatomical structures based on physician feedback and quantitative assessments.
Impact: The proposed synthesis algorithm simplifies and speeds up MRI-only radiotherapy workflow. It is also applicable to other fields of medical image synthesis, such as multi-modal image diagnosis treatment.
INTRODUCTION
Medical imaging has become increasingly important in the diagnosis and treatment of oncological patients, particularly in radiotherapy (RT). Traditionally, CT-based imaging is widely adopted in RT for patient positioning and monitoring before, during, or after the dose delivery [1].Over the past few decades, magnetic resonance imaging (MRI) has demonstrated its efficacy in enhancing tumor and organ-at-risk delineation, primarily owing to its superb soft-tissue contrast [2]. Hence, some researchers consider using MRI for treatment planning simulation, patient positioning alignment with the treatment plan, and real-time monitoring[3-5]. To benefit from the complementary advantages offered by different imaging modalities, MRI is generally registered to CT. This workflow necessitates the acquisition of a CT scan, which increases the doctor's workload and exposes the patient to additional radiation.
Recently, MRI-only based RT has been proposed to simplify and speed up the workflow while reducing patients' exposure to ionizing radiation. This is particularly significant in repeated simulations or with more vulnerable populations, such as children. The primary challenge in implementing MRI-only RT is the absence of tissue attenuation information necessary for precise dose calculations. Various methods have been proposed to convert MRI data into CT-equivalent images, thereby generating synthetic CT (sCT) for treatment planning and dose calculation.
In recent years, there has been a growing interest in synthesizing sCT from MRI data using artificial intelligence (AI) algorithms [6]. However, the current AI models have limitations in generating small anatomical structures, like tumors, which can negatively affect the precision of dose calculations. This study proposes a new model for synthesizing medical images with high quality at different anatomical scales.
METHOD
Given the abundance of unpaired, unlabeled medical datasets, we designed an unsupervised deep learning model based on the GAN architecture [7]. The overall structure of the model is shown in Figure 1. We developed a hyper-perceptual feature extraction module, ACP, within the generator bottleneck. It mainly consists of an external attention branch and multiple convolution branches. The convolution branches learn local image features, while the attention branch focuses on long context information[8]. This multi-branch design allows for information extraction from different anatomical scales of the medical image. Additionally, the proposed model incorporates skip-connection to convey low-dimensional structural information within the image.For the discriminator, we utilized the Patch GAN [9]. We also introduced a novel objective function known as the contrastive loss [10]. This objective is to maximize the mutual information between corresponding patches in the input and the output images. For example, if we consider a patch depicting the liver in a generated CT image, we aim to strengthen the association between this patch and the liver patch in the input MRI image, compared to the rest of the patches in the MRI.
RESULTS
The experiments used two datasets. The first was a proprietary dataset containing liver tumor images of different modalities. It included a total of 43 MR images and 35 CT images. The second dataset was the public dataset BraTS, which primarily consists of different sequences of brain images. In this experiment, we selected paired T1 and T2 images of 80 patients.We evaluated our model using several vital parameters, including Peak Signal-to-Noise Ratio (PSNR), Fréchet Inception Distance (FID), and the Dice Coefficient, which we segmented the liver in the synthesized CT and compared with the liver in the input MR image as shown in Fig.2. We also compared it with other popular baseline models. The results were presented in Table 1. Based on the numerical results of the evaluation metrics, our model outperformed all others. Based on the synthetic images presented, as shown in Fig.3 and Fig.4, our proposed model demonstrated excellent performance in reproducing small anatomical structures while also maintaining the highest quality in terms of texture synthesis.
DISCUSSION
We contend that the multi-branch feature extraction module within the generator effectively extracts image features across different anatomical scales, enabling high-quality image synthesis in the target domain.CONCLUSION
We proposed a multi-branch medical image synthesis model to facilitate MR-to-CT conversion for radiotherapy. Simultaneously, this model offers versatile applications in other fields of medical image synthesis, including registration, multi-modal image diagnosis, and treatment.Acknowledgements
This work was supported in part by the Ministry of Science and Technology of the People's Republic of China under Grant 2023YFC2411400, in part by Shanghai Jiao Tong University Medical Engineering Cross Research Funds under Grant YG2021ZD05, in part by Shanghai Hospital Development Center Foundation under Grant SHDC12021112, in part by Shanghai Zhangjiang National Independent Innovation Demonstration Zone Special Development Fund Maior Project under Grant ZJ2021-ZD-007.References
[1] Ahmad S S, Duke S, Jena R, et al. Advances in radiotherapy[J]. Bmj, 2012, 345.
[2] Schmidt M A, Payne G S. Radiotherapy planning using MRI[J]. Physics in Medicine & Biology, 2015, 60(22): R323.
[3] Raaymakers B W, Raaijmakers A J E, Kotte A, et al. Integrating a MRI scanner with a 6 MV radiotherapy accelerator: dose deposition in a transverse magnetic field[J]. Physics in Medicine & Biology, 2004, 49(17): 4109.
[4] Mutic S, Dempsey J F. The ViewRay system: magnetic resonance–guided and controlled radiotherapy[C]//Seminars in radiation oncology. WB Saunders, 2014, 24(3): 196-199.
[5] Raaymakers B W, Jürgenliemk-Schulz I M, Bol G H, et al. First patients treated with a 1.5 T MRI-Linac: clinical proof of concept of a high-precision, high-field MRI guided radiotherapy treatment[J]. Physics in Medicine & Biology, 2017, 62(23): L41.
[6] Huynh E, Hosny A, Guthier C, et al. Artificial intelligence in radiation oncology[J]. Nature Reviews Clinical Oncology, 2020, 17(12): 771-781.
[7] Creswell A, White T, Dumoulin V, et al. Generative adversarial networks: An overview[J]. IEEE signal processing magazine, 2018, 35(1): 53-65.
[8] Dalmaz O, Yurt M, Çukur T. ResViT: Residual vision transformers for multimodal medical image synthesis[J]. IEEE Transactions on Medical Imaging, 2022, 41(10): 2598-2614.
[9] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedingsof the IEEE conference on computer vision and pattern recognition,pages 1125–1134, 2017
[10] T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, ‘Contrastive Learning for Unpaired Image-to-Image Translation’. arXiv, Aug. 20, 2020. Accessed: Feb. 06, 2023. [Online]. Available: http://arxiv.org/abs/2007.1565
Figures
