4488

Paired Conditional Generative Adversarial Network for Highly Accelerated Liver 4D MRI

Di Xu¹, Xin Miao², Yang Yang³, Hengjie Liu⁴, Jessica E. Scholey¹, Wensha Yang¹, Mary Feng¹, Michael Ohliger¹, Yi Lao⁴, and Ke Sheng¹
¹Radiation Oncology, UCSF, San Francisco, CA, United States, ²Siemens Healthineers, Boston, MA, United States, ³Radiology, UCSF, San Francisco, CA, United States, ⁴Radiation Oncology, UCLA, Los Angeles, CA, United States

Synopsis

Keywords: AI/ML Image Reconstruction, Radiotherapy, 4D MRI

Motivation: Densely sampled k-space leads to high-quality MR but can be impractical due to lengthy scanning time. Accelerating MR acquisition by reducing sampling density can decrease image quality and/or increase reconstruction complexity and time.

Goal(s): This work aims to design an algorithm for efficient and high-quality reconstruction of highly accelerated radial-sampling liver 4D MR.

Approach: We proposed a novel Paired Conditional generative adversarial network term Re-Con-GAN, evaluated on a 4D liver MR dataset at 3x, 6x, and 10x acceleration ratios.

Results: Re-Con-GAN achieved better PSNR, SSIM, and RMSE with sub-second inference speed (0.15s) than compressed sensing (120s) and non-GAN deep learning methods (0.15s-0.73s).

Impact: A robust and efficient framework, Re-Con-GAN, is proposed in the current work with sub-second inference speed (0.15s) and promising reconstruction results demonstrated on an in-house curated 4D liver MRI dataset.

Introduction

4D MRI is a powerful tool to quantify tumor boundary and motion for image-guided liver radiation therapy (RT) owing to its superior soft tissue contrast^1–6. In a free-breathing liver RT treatment planning workflow, the tumor contours are delineated at each respiratory phase of 4D MRI to form an internal target volume (ITV) for the ultimate target region of radiation. Full-liver coverage and high spatiotemporal resolutions are required for imaging to ensure accurate ITV definition. Continuous free-breathing scanning of 3D golden angle stack-of-star sequence with self-navigation is commonly used for acquiring 4D MRI data⁵. Though radial sampling is relatively robust to motion sensitivity⁷ and acceleration⁸, there are a few limitations of this technique, including long scanning time (8-10 min⁵), limited inter-slice resolution⁹, and under-sampling streaking artifacts that are evident with less regular breathing patterns.
Parallel imaging^10,11 and compressed sensing (CS)^12–14 have been extensively studied. However, CS reconstruction tends to be slow and shows worsening performance at high under-sampling rates^15–18. Recent deep learning (DL) advances have offered a data-driven approach for efficient 4D MRI reconstruction. Existing works have explored 4D MRI reconstruction with convolutional¹⁹ and recurrent^20,21 neural networks as well as Transformers²². Promising results were reported using these models with explicitly defined loss functions, yet detail loss was evident at high acceleration ratios. Since balancing image fidelity, noise suppression, and detail retention in an explicitly defined loss function is difficult, we postulate that including implicit loss terms through Generative Adversarial Networks (GANs)²³ training can circumvent the challenge. Though some previous studies have demonstrated superior MRI reconstruction using GANs^24–27, none of them explored the capability of GANs in 4D MRI temporal profiling and reconstruction. The current study explored the feasibility of using GANs for 4D MRI reconstruction. We developed a novel architecture termed reconstruct conditional GAN (Re-Con-GAN).

Methods

17 patients who underwent liver contrast-enhanced 4D MRI were included in the study. A prototype free-breathing T1-weighted volumetric golden angle stack-of-stars sequence was used for acquisition. The scanning parameters were - TE=1.5ms, TR=3 ms, matrix size=288 × 288 , FOV=374 mm × 374 mm, in-plane resolution=1.3 mm × 1.3 mm, slice thickness=3 mm, radial views (RV) per partition=3000, number of slices per partition=64-75, acquisition time=8-10 min. The pulse sequence ran continuously over multiple respiratory cycles. Retrospective under-sampling was performed by keeping the first 1000, 500, and 300 spokes from the total 3000 spokes (3×, 6× and 10×). For initial image reconstruction, data sorting based on self-gating signal was performed to divide the contiguously acquired k-space data into 8 respiratory phases. Nonuniform fast Fourier transform (nuFFT) was applied to reconstruct each phase. We organized the images in 2D+t. The images were resized to 256 × 256 and normalized using Z-score normalization. Data augmentation was employed, including random rotation, flipping, and cropping.
Re-Con-GAN trains input (nuFFT reconstruction of under-sampled data) and output (fully sampled image series) in pairs. Re-Con-GAN is a versatile architecture with plug-and-play generator, discriminator, and loss objective, as shown in Figure 1 and 2. Three types of networks, including 3D ResNet9²⁸, UNet²³, and reconstruction swin transformer (RST)²², were explored as generators. PatchGAN was selected as the discriminator. A novel loss objective fused with L₁, L₂, and multi-scale structure similarity indexed measurement was designed for network training.

Results

A conventional CS approach was included as the baseline. Ablation studies that solely tune generators are also conducted to underpin the improvement from Re-Con-GAN. Statistical results and visualization are reported in Table 1 Figure 3 and Figure 4. Re-Con-GAN based models is consistently superior to that of CS and ablated generators across all acceleration levels. Re-Con-GAN with 3D ResNet9 generator performs moderately better than other generators. The inference time of Re-Con-GAN with 3D ResNet9, 3D ResNet9 and CS are 0.15s, 0.15s and 120s.

Discussion

RT provides effective tumor suppression for non-surgical hepatocellular carcinoma patients and the patients waiting for liver transplantation. The success of liver RT largely depends on the efficiency and quality of its imaging guidance. Therefore, we developed Re-Con-GAN for sub-second 4D reconstruction of highly accelerated MR sequences. The platform using implicit loss function is conducive to testing more aggressive acceleration ratios and future extension into raw coil signal reconstruction.

Conclusion

A robust and efficient framework, Re-Con-GAN, is proposed in the current work with promising and efficient reconstruction results demonstrated on an in-house curated 4D liver MRI dataset. Re-Con-GAN has demonstrated superior performance than CS and non-GANs DL methods. Re-Con-GAN can potentially benefit the advance of online adaptive MR-guided liver RT.

Acknowledgements

The study is supported in part by NIH R01CA259008 and DOD W81XWH2210044.

References

1. Witt JS, Rosenberg SA, Bassetti MF. MRI-guided adaptive radiotherapy for liver tumours: visualising the future. Lancet Oncol. 2020;21(2):e74-e82. doi:10.1016/S1470-2045(20)30034-6

2. Hussain SM, Semelka RC. Hepatic imaging: comparison of modalities. Radiol Clin North Am. 2005;43(5):929-947, ix. doi:10.1016/j.rcl.2005.05.006

3. Voroney JP, Brock KK, Eccles C, Haider M, Dawson LA. Prospective comparison of computed tomography and magnetic resonance imaging for liver cancer delineation using deformable image registration. Int J Radiat Oncol Biol Phys. 2006;66(3):780-791. doi:10.1016/j.ijrobp.2006.05.035

4. Stemkens B, Paulson ES, Tijssen RHN. Nuts and bolts of 4D-MRI for radiotherapy. Phys Med Biol. 2018;63(21):21TR01. doi:10.1088/1361-6560/aae56d

5. Thomas HR, Miao X, Ferguson D, et al. Contrast-enhanced 4D-MRI for internal target volume generation in treatment planning for liver tumors. Radiother Oncol. 2022;173:69-76 doi:10.1016/j.radonc.2022.05.037

6. Paulson ES, Ahunbay E, Chen X, et al. 4D-MRI driven MR-guided online adaptive radiotherapy for abdominal stereotactic body radiation therapy on a high field MR-Linac: Implementation and initial clinical experience. Clin Transl Radiat Oncol. 2020;23:72-79. doi:10.1016/j.ctro.2020.05.002

7. Glover GH, Pauly JM. Projection reconstruction techniques for reduction of motion effects in MRI. Magn Reson Med. 1992;28(2):275-289. doi:10.1002/mrm.1910280209

8. Winkelmann S, Schaeffter T, Koehler T, Eggers H, Doessel O. An optimal radial profile order based on the Golden Ratio for time-resolved MRI. IEEE Trans Med Imaging. 2007;26(1):68-76. doi:10.1109/TMI.2006.885337

9. Li Z, Huang C, Tong A, Chandarana H, Feng L. Kz-accelerated variable-density stack-of-stars MRI. Magnetic Resonance Imaging. 2023;97:56-67. doi:10.1016/j.mri.2022.12.017

10. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sensitivity encoding for fast MRI. Magn Reson Med. 1999;42(5):952-962.

11. Griswold MA, Jakob PM, Heidemann RM, et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn Reson Med. 2002;47(6):1202-1210. doi:10.1002/mrm.10171

12. Feng L, Grimm R, Block KT, et al. Golden-angle radial sparse parallel MRI: combination of compressed sensing, parallel imaging, and golden-angle radial sampling for fast and flexible dynamic volumetric MRI. Magn Reson Med. 2014;72(3):707-717. doi:10.1002/mrm.24980

13. Feng L, Benkert T, Block KT, Sodickson DK, Otazo R, Chandarana H. Compressed sensing for body MRI: Compressed Sensing for Body MRI. J Magn Reson Imaging. 2017;45(4):966-987. doi:10.1002/jmri.25547

14. Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med. 2007;58(6):1182-1195. doi:10.1002/mrm.21391

15. Gamper U, Boesiger P, Kozerke S. Compressed sensing in dynamic MRI. Magn Reson Med. 2008;59(2):365-373. doi:10.1002/mrm.21477

16. Chen L, Schabel MC, DiBella EVR. Reconstruction of dynamic contrast enhanced magnetic resonance imaging of the breast with temporal constraints. Magn Reson Imaging. 2010;28(5):637-645. doi:10.1016/j.mri.2010.03.001

17. Adluru G, McGann C, Speier P, Kholmovski EG, Shaaban A, Dibella EVR. Acquisition and reconstruction of undersampled radial data for myocardial perfusion magnetic resonance imaging. J Magn Reson Imaging. 2009;29(2):466-473. doi:10.1002/jmri.21585

18. Lingala SG, Hu Y, DiBella E, Jacob M. Accelerated dynamic MRI exploiting sparsity and low-rank structure: k-t SLR. IEEE Trans Med Imaging. 2011;30(5):1042-1054. doi:10.1109/TMI.2010.2100850

19. Schlemper J, Caballero J, Hajnal JV, Price AN, Rueckert D. A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction. IEEE Trans Med Imaging. 2018;37(2):491-503. doi:10.1109/TMI.2017.2760978

20. Qin C, Schlemper J, Caballero J, Price AN, Hajnal JV, Rueckert D. Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction. IEEE Trans Med Imaging. 2019;38(1):280-290. doi:10.1109/TMI.2018.2863670

21. Huang Q, Xian Y, Yang D, et al. Dynamic MRI reconstruction with end-to-end motion-guided network. Medical Image Analysis. 2021;68:101901. doi:10.1016/j.media.2020.101901

22. Xu D, Liu H, Ruan D, Sheng K. Learning Dynamic MRI Reconstruction with Convolutional Network Assisted Reconstruction Swin Transformer. Published online September 18, 2023. Accessed September 19, 2023. http://arxiv.org/abs/2309.10227

23. Isola P, Zhu JY, Zhou T, Efros AA. Image-to-Image Translation with Conditional Adversarial Networks. Published online 2016. doi:10.48550/ARXIV.1611.07004

24. Yang G, Yu S, Dong H, et al. DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction. IEEE Trans Med Imaging. 2018;37(6):1310-1321. doi:10.1109/TMI.2017.2785879

25. Mardani M, Gong E, Cheng JY, et al. Deep Generative Adversarial Neural Networks for Compressive Sensing MRI. IEEE Trans Med Imaging. 2019;38(1):167-179. doi:10.1109/TMI.2018.2858752

26. Liu F, Samsonov A, Chen L, Kijowski R, Feng L. SANTIS: Sampling‐Augmented Neural neTwork with Incoherent Structure for MR image reconstruction. Magnetic Resonance in Med. 2019;82(5):1890-1904. doi:10.1002/mrm.27827

27. Gao C, Ghodrati V, Shih SF, et al. Undersampling artifact reduction for free-breathing 3D stack-of-radial MRI based on a deep adversarial learning network. Magnetic Resonance Imaging. 2023;95:70-79. doi:10.1016/j.mri.2022.10.010

28. Johnson J, Alahi A, Fei-Fei L. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Published online 2016. doi:10.48550/ARXIV.1603.08155

Figures

Figure 1: Training Re-Con-GAN Discriminator to map under sampled 4D MRI image series to fully sampled ones. D is designed to learn to distinguish between fake and real images.

Figure 2: The design of our proposed Re-Con-GAN. The input to Re-Con-GAN is a random noise vector z and under-sampled image series x. The supervision is a fully sampled image series y. The network is trained by combining generator G and discriminator D.

Figure 3: Visualization of 3x, 6x and 10x reconstruction results of an axial view slice from a patient in validation set. Reconstruction visualization, zoomed-in region of interest as well as residual between prediction and the fully sampled nuFFT reconstructed image are visualized. US nuFFT refers to the under-sampled nuFFT image series (input), US R9 GAN refers to the Re-Con-GAN with ResNet9 generator, U256 refers to the 3D UNet, CS refers to the compressed sensing reconstruction and FS nuFFT refers to the fully sampled image series.

Figure 4: Visualization of a selected temporal profile (motion binning = 1, 3, 5, 7) from a patient in the validation set. 3x, 6x, and 10x reconstruction results from input, ground truth and our proposed method are visualized. Red arrows points towards the patient’s liver tumor.

Table 1: Statistical results from our proposed Re-Con-GAN under 3x, 6x and 10x acceleration rate and their corresponding baselines are presented. The best score and the worst score under each acceleration is bolded and wavy underlined, respectively.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)

4488

DOI: https://doi.org/10.58530/2024/4488