2237
Code-Aware Transformation from NAC PET to AC PET, MRI, or CT Imaging1Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese, Shenzhen, China, 2Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China, Shenzhen, China, 3Department of Nuclear Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China, 4Central Research Institute, United Imaging Healthcare Group, Shanghai, China, 5Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences., Beijing, China
Synopsis
Keywords: Other AI/ML, Multimodal, modality transformation, dynamicly transformation, NAC PET, AC PET, MRI, CT
Motivation: In radiation therapy with PET images, CT and MR images are used for precise targeting, but acquiring them is expensive, time-consuming and increases radiation risk.
Goal(s): Developing a deep learning model capable of dynamically switching to a specified mode enhances flexibility beyond traditional one-to-one cross-modal conversion methods.
Approach: We developed a deep learning model with dynamic modality translation capabilities by the incorporation of switch layers within the decoder module.
Results: The evaluations showed that our model excels at converting non-attenuation corrected PET images to attenuation corrected PET, MR, or CT images, making it easier to obtain additional modality images for radiation therapy.
Impact: Dynamic conversion from NAC PET to desired modalities like AC PET, CT, or MRI on demand is more efficient, saving on data storage and processing, and offers customized imaging for specific clinical needs, enhancing workflow efficiency.
Introduction
In radiation therapy, PET images guide the customization of radiation doses, while CT or MR imaging provides precise tumor localization when used alongside PET. However, acquiring additional CT or MR scans solely for targeting increases radiation exposure and extends scan times, just as attenuation correction for AC PET images does. Generating AC PET, synthetic CT, and MRI images from NAC PET data through cross-modal transformation offers a safer, more efficient alternative. Researchers have spent a lot of efforts on using deep learning to transform one type of image [1-9], like non-attenuation corrected (NAC) PET, into another, such as attenuation corrected (AC) PET [10,11], synthetic CT [12,13], or MRI [14]. However, existing cross-modal transformation methods mainly focus on one-to-one model conversions, which may restrict the flexibility of the transformation process. To address this limitation, we developed a switch-based network for multi-modality medical image translation. The proposed model has the capability to generate targeted modality images from NAC PET data based on user input, encompassing sAC PET, sCT, and sMRI images.Materials and methods
Patient studies:A total of 119 patients scanned with the uEXPLORER scanner and 225 patients scanned with the uPMR scanner were retrospectively enrolled. Data were gathered using the uEXPLORER and uPMR scanners manufactured by UIH from 2020 to 2022 in Shanghai. We selected 13 patient cases as an external validation set for quantitative analysis, and the remaining 108 PET/CT cases and 215 PET/MR cases were used as experimental data to train the proposed network.
Method Implements:
We propose a modified UNet to realize adaptive multimodality translation. The overall framework of the proposed model is shown in Figure 1. The network consists of three parts: an encoder, several residual blocks, and a decoder with adaptive model translation layers. The input of the proposed network is non-attenuation correction PET (NAC PET) and the switch code. The output is the corresponding modality image that is specified by the switch code.
The adaptive multimodality translation is realized by the following equation,
$$A(x,s)=F^1(s)\frac{x-\mu(x)}{\sigma(x)}+F^2(s)$$
where $$$F^1(s)$$$ and $$$F^2(s)$$$ are two fully connected layers that learns the scale and shift parameters for controlling the adaptive modality translation. $$$\mu(x)$$$ and $$$\sigma(x)$$$ are the mean and standard deviation of $$$x$$$. $$$x$$$ is the feature map of the input and $$$s$$$ is the switch code after one-hot coding.
Data analysis:
To better train the proposed model, we mix and slice these data along the axial orientation to obtain two-dimensional images. After excluding 6547 slices for negative samples, we obtained 104286 slice samples in total. We randomly select 83429 samples for training, 10428 samples for validation, and 10429 samples for testing. The quantitative performance is evaluated by PSNR, MAE, and SSIM. The qualitative performance is measured by the error map between the model output and the ground truth.
Results
Figure 2 shows the AC PET, CT and MRI translated from NAC PET under corresponding switch code. The error maps are used to evaluate the visual effect of the proposed model. The profiles of the translated CT and MRI image from NAC PET also show that the proposed model can well perform the translation task. Table 1 illustrates the quantitative outcomes for the test dataset. While variations in bias, PSNR, and SSIM values are evident across different anatomical sites, the quantitative results are remarkably positive. Figure 3 shows the outcomes of modality translations without actual references and the clinical evaluation by two radiologists. According to the radiologists, the proposed model excels in noise suppression and yields outcomes of notable comprehensive quality.Discussion and conclusion:
This work proposed a deep learning model that utilizes switch layers to achieve multi-modal image transformation with modality code awareness. This approach enabled dynamic translation across multiple modalities by incorporating switch codes that encapsulate users' specific modality preferences. The performance of the proposed method was thoroughly assessed using both objective and subjective evaluation metrics on a dataset comprising 336 patient data instances. The results not only validated the efficacy of the adaptive multi-modality translation approach but also underscored its potential for minimizing scanning time, patient radiation exposure, acquisition complexities, and modality misalignment. Future endeavors could explore arbitrary multi-modality translation and the extension of 3D network capabilities.Acknowledgements
This work was supported by the National Natural Science Foundation of China (12305409, 82372038 and 62101540), the Shenzhen Excellent Technological Innovation Talent Training Project of China (RCJC20200714114436080), the Key Laboratory for Magnetic Resonance and Multimodality Imaging of Guangdong Province (2023B1212060052), the Shenzhen Science and Technology Program (JCYJ20220818101804009 and RCBS20210706092218043), and the Guangdong Basic and Applied Basic Research Foundation (2022A1515110696).References
[1] S Poonkodi and M Kanchana. 3d-medtrancsgan: 3d medical image transformation using csgan. Computers in Biology and Medicine, 153:106541, 2023.
[2] Feifei Li, Mirjam Schöneck, Oya Beyan, and Liliana Lourenco Caldeira. Voxel-wise medical imaging transformation and adaption based on cyclegan and score-based diffusion. CARING IS SHARING–EXPLOITING THE VALUE IN DATA FOR HEALTH AND INNOVATION, page 1027, 2023.
[3] Yang Lei, Joseph Harms, Tonghe Wang, Yingzi Liu, Hui-Kuo Shu, Ashesh B Jani, Walter J Curran, Hui Mao,Tian Liu, and Xiaofeng Yang. Mri-only based synthetic ct generation using dense cycle consistent generative adversarial networks. Medical physics, 46(8):3565–3581, 2019.
[4] Yao Zhao, He Wang, Cenji Yu, Laurence E Court, Xin Wang, Qianxia Wang, Tinsu Pan, Yao Ding, Jack Phan, and Jinzhong Yang. Compensation cycle consistent generative adversarial networks (comp-gan) for synthetic ct generation from mr scans with truncated anatomy. Medical physics, 2023.
[5] Amir Jabbarpour, Seied Rabi Mahdavi, Alireza Vafaei Sadr, Golbarg Esmaili, Isaac Shiri, and Habib Zaidi. Unsupervised pseudo ct generation using heterogenous multicentric ct/mr images and cyclegan: Dosimetric assessment for 3d conformal radiotherapy. Computers in biology and medicine, 143:105277, 2022.
[6] Chun-Chieh Wang, Pei-Huan Wu, Gigin Lin, Yen-Ling Huang, Yu-Chun Lin, Yi-Peng Chang, and Jun-Cheng Weng. Magnetic resonance-based synthetic computed tomography using generative adversarial networks for intracranial tumor radiotherapy treatment planning. Journal of personalized medicine, 12(3):361, 2022.
[7] Jacopo Lenkowicz, Claudio Votta, Matteo Nardini, Flaviovincenzo Quaranta, Francesco Catucci, Luca Boldrini, Marica Vagni, Sebastiano Menna, Lorenzo Placidi, Angela Romano, et al. A deep learning approach to generate synthetic ct in low field mr-guided radiotherapy for lung cases. Radiotherapy and Oncology, 176:31–38, 2022.
[8] Wen Li, Yafen Li, Wenjian Qin, Xiaokun Liang, Jianyang Xu, Jing Xiong, and Yaoqin Xie. Magnetic resonance image (mri) synthesis from brain computed tomography (ct) images based on deep learning methods for magnetic resonance (mr)-guided radiotherapy. Quantitative imaging in medicine and surgery, 10(6):1223, 2020.
[9] Eryan Feng, Pinle Qin, Rui Chai, Jianchao Zeng, Qi Wang, Yanfeng Meng, and Peng Wang. Mri generated from ct for acute ischemic stroke combining radiomics and generative adversarial networks. IEEE Journal of Biomedical and Health Informatics, 26(12):6047–6057, 2022.
[10] Xue Dong, Yang Lei, Tonghe Wang, Kristin Higgins, Tian Liu, Walter J Curran, Hui Mao, Jonathon A Nye, and Xiaofeng Yang. Deep learning-based attenuation correction in the absence of structural information for whole-body positron emission tomography imaging. Physics in Medicine & Biology, 65(5):055011, 2020.
[11] Isaac Shiri, Pardis Ghafarian, Parham Geramifar, Kevin Ho-Yin Leung, Mostafa Ghelichoghli, Mehrdad Oveisi, Arman Rahmim, and Mohammad Reza Ay. Direct attenuation correction of brain pet images using only emission data via a deep convolutional encoder-decoder (deep-dac). European radiology, 29:6867–6879, 2019.
[12] Zhanli Hu, Yongchang Li, Sijuan Zou, Hengzhi Xue, Ziru Sang, Xin Liu, Yongfeng Yang, Xiaohua Zhu, Dong Liang, and Hairong Zheng. Obtaining pet/ct images from non-attenuation corrected pet images in a single pet system using wasserstein generative adversarial networks. Physics in Medicine & Biology, 65(21):215010, 2020.
[13] Xue Dong, Tonghe Wang, Yang Lei, Kristin Higgins, Tian Liu, Walter J Curran, Hui Mao, Jonathon A Nye, and Xiaofeng Yang. Synthetic ct generation from non-attenuation corrected pet images for whole-body pet imaging. Physics in Medicine & Biology, 64(21):215016, 2019.
[14] Changhui Jiang, Xu Zhang, Na Zhang, Qiyang Zhang, Chao Zhou, Jianmin Yuan, Qiang He, Yongfeng Yang, Xin Liu, Hairong Zheng, et al. Synthesizing pet/mr (t1-weighted) images from non-attenuation-corrected pet images. Physics in Medicine & Biology, 66(13):135006, 2021.
Figures
