2240
Zero-Dose PET Synthesis for Patients with Glioblastoma by Mixture-of-Experts-Based TransUNet1Stanford CAFN Lab, Santa Clara, CA, United States
Synopsis
Keywords: AI/ML Image Reconstruction, PET/MR
Motivation: Through the utilization of MRI images: T1, T1c, ASL, and T2-FLAIR, the PET image can be synthesized without requiring the patients to face radiation exposure.
Goal(s): To synthesize high-quality FDG-PET images by multi-contrast MRI, using TransUNet with mixture-of-experts (MoE) in order to ensemble both local and global feature maps.
Approach: TransUNet was utilized as the backbone to synthesize PET from multi-contrast MRI. A mixture-of-experts (MoE) was designed to assign a weight map to each layer's feature.
Results: Multi-contrast MRIs can be used to synthesize FDG-PET images for GBM cases by the proposed MoE-based TransUNet model, and incorporating MoE in TransUNet yields solid results.
Impact: To synthesize high-quality FDG-PET images by multi-contrast MR via deep learning, this zero-dose project can be transformative in today’s modern field of medicine, It would not require the patients to face radiation exposure while improving the accessibility of PET information.
Target Audience
Clinicians, researchers, staff, technologists and students working on PET/MRI.Introduction
Positron emission tomography (PET) is a widely used imaging technique in many clinical applications including tumor detection and neurological disorder diagnosis.However, PET has drawbacks: (1) risk of primary or secondary cancer in scanned subjects, (2) expensive and (3) not offered in the majority of medical centers in the world.
The purpose of this research is to synthesize high-quality PET images from radiation-free multi-contrast MR images, using TransUNet with mixture-of-experts (MoE) in order to ensemble both local and global feature maps.
Methods
Data Acquisition and Preprocessing: 38 patients with glioblastoma with paired FDG PET and MRI (T1, T1 with contrast [T1c], T2-FLAIR, and arterial spin label [ASL]) exams were acquired. MRI images were co-registered to PET and further normalized to a standard template. The volumes were normalized by the mean of the non-zero regions. Flipping along the X axis was used for data augmentation.Models: The overview of the method is demonstrated in Figure (a). TransUNet [1] was utilized as the backbone, which merits both U-Net (for local context) and Transformer (for global context) to synthesize PET from multi-contrast MRI. Each MRI contrast was fed separately to the same model. Then, a mixture-of-experts (MoE) [2] was designed to assign a weight map to each layer's feature before fusing them as the final output of the deep learning model. Specifically, it treats each layer as an expert by upsampling its feature map (after pixel-wise MLP) to the full size, capturing both local and global dependencies. Lastly, a simple late fusion is then taken to combine synthesized PET images that are independently synthesized from one of the MRI contrasts. It removes the dependency on the complete four MRI contrasts as the input, which is especially helpful when one or more contrasts are missing due to different imaging protocols or corruption.
Results
As shown in Table (b), TransUNet based model achieves consistent synthesis performance. Compared to the vanilla TransUNet, incorporating MoE in TransUNet yields better results (improved 19.0% in PSNR, 5.6% in SSIM).Discussion
A deep-dive on the MoE’s effects in visualization is performed, by plotting the MoE weight maps (W1, W2, W3, W4) for each expert feature of each resolution-layer (from high resolution to low resolution). The MoE weight map for each expert feature is visualized in Figure (c). It presents the different weights’ focus on the different contexts, in which W1 depicts the local details, W2 on the global context information inside the brain, W3 on the brain’s boundaries, and W4 on complementary to the brain. The visualization shows the abnormal regions (red boxes) are mapped with the darker-colored heavy weight areas (green boxes).Conclusion
Multi-contrast MRIs can be used to synthesize high-quality FDG-PET images for GBM cases by the proposed MoE-based TransUNet AI model.Acknowledgements
No acknowledgement found.References
[1] Ouyang, J., Chen, K. T., Duarte Armindo, R., Davidzon, G. A., Hawk, K. E., Moradi, F., ... & Zaharchuk, G. (2023). Predicting FDG‐PET Images From Multi‐Contrast MRI Using Deep Learning in Patients With Brain Neoplasms. Journal of Magnetic Resonance Imaging.
[2] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
Figures
(a) Overview of the proposed methods.
(b) Qualitative results.
(c) Results of Synthesized PET vs Acquired PET in Abnormal Regions.