4484
Computationally Efficient IMplicit Training Strategy for UNrolled NEtworks (IMUNNE) for Reconstruction of Accelerated Cardiac Dynamic MRI1Radiology, Northwestern University, Chicago, IL, United States, 2Computer Science, Northwestern University, Evanston, IL, United States, 3Electrical Engineering, Northwestern University, Evanston, IL, United States, 4Biomedical Engineering, Northwestern University, Evanston, IL, United States, 5Medicine, Cardiology Division, Northwestern University, Chicago, IL, United States
Synopsis
Keywords: AI/ML Image Reconstruction, Machine Learning/Artificial Intelligence, Compressed Sensing, Unrolled Network, Implicit Network
Motivation: Unrolled networks (UN) achieve state-of-the-art performance in undersampled dynamic MRI reconstruction but suffer from long training times and extensive GPU memory cost.
Goal(s): To apply an implicit training strategy for UNs (IMUNNE) in combination with transfer learning to develop an efficient and versatile reconstruction technique for accelerated dynamic cardiac MRI.
Approach: We compare IMUNNE with a complex denoiser U-Net and an end-to-end UN on three different highly undersampled dynamic cardiac MRI datasets.
Results: For all datasets, we observed that: (1) both unrolled architectures outperform CU-Net with respect to image quality; (2) compared to end-to-end UN, IMUNNE significantly reduced both training and inference times.
Impact: This work has the potential to facilitate a more widespread adoption of highly-accelerated, cardiac MRI by reducing training time, inference time and memory cost of state-of-the-art unrolled reconstruction methods, thereby lowering the clinical hardware requirements and the requisite energy consumption.
Introduction
While cardiovascular MRI is a powerful modality, it suffers from lengthy scan time and sensitivity to arrhythmia and/or dyspnea. One approach to addressing said weaknesses is to highly accelerate the acquisition using compressed sensing (CS) and perform imaging during free breathing. Regrettably, the clinical adoption of CS has been slow due to its lengthy iterative, nonlinear process. This may be addressed using deep learning. Unrolled networks (UN)1–3 have shown state-of-the-art performance for accelerated MRI reconstruction. However, the end-to-end training of UNs is memory-intensive and time-consuming even with top-notch GPU hardware, so that the application of UNs generally remains limited to academic settings. Recently, a novel training strategy for implicit unrolled networks (IMUNNE)4,5 was proposed to reduce memory and training time while maintaining the advantages of UN. In this study, we adapt IMUNNE on three disparate cardiac MRI datasets (real-time cine with balanced steady-state free precession [b-SSFP], real-time cine with gradient recalled echo [GRE], and first-pass perfusion with GRE) obtained using radial k-space sampling, in order to demonstrate the versatility of IMUNNE. By performing the main training on one of the datasets and subsequent transfer learning on the remaining datasets, we further reduce the computational cost. We compare IMUNNE to a complex U-Net and a UN with alternating data-consistency and regularizer blocks (Alt-UN)3 in terms of image quality and training and evaluation times.Methods
Data preparation: We used three disparate cardiac MRI datasets for training and evaluation, comprising: (1) real-time cine with b-SSFP, (2) real-time cine with GRE and (3) first-pass cardiac perfusion MRI with GRE. Table 1 contains the relevant imaging parameters for each dataset. The sample unit for training was a single cine series. Due to reasons listed in6, we obtained the ground truth using CS7.Implementation: The core idea of IMUNNE is to train the UN “implicitly” i.e., the gradient backpropagation is performed only through the last unrolled iteration4,5, thereby approximating the correct gradient to calculate the update of network weights. We compared IMUNNE to a standalone denoising CU-Net with the same configuration as used within IMUNNE (21.7K trainable weights; Fig. 1). We further compared IMUNNE to Alt-UN to showcase the acceleration achieved compared to conventional end-to-end training methods.
For CU-Net and IMUNNE, we used the same implementation as originally proposed4. For Alt-UN, we used the publicly available code base without the originally proposed skip connection since our datasets were reconstructed using an adaptable regularization weight, leading to different scaling magnitude between the zero-filled inputs and the reconstructed images.
Network training and testing: The main training of each of the architectures was performed on the b-SSFP cine dataset. We used the learning rate of 5e-04 with learning rate decay factor of 0.95 every 30 epochs for all architectures. CU-Net and IMUNNE were trained for 500 epochs, respectively. Alt-UN was trained according to the training scheme prescribed by the authors of the original publication3: the regularization block was pretrained followed by 150 epochs of fine-tuning of the end-to-end architecture. Subsequently, transfer learning with a limited number of training epochs was performed for the GRE cine and the perfusion datasets. Each architecture was initialized with the weights obtained through the main training. Other relevant training parameters are described in Table 1.
To compare the performance of IMUNNE with other methods, we calculated the Structural similarity index (SSIM)8, normalized mean-square error (NMSE), peak signal-to-noise ratio (PSNR) and a reference-free blur metric (0: sharpest; 1: blurriest)9.
All architectures were implemented using PyTorch with GPU acceleration (NVIDIA A100-PCIE-40GB) and the TorchKbNufft10 library for backpropagatable non-uniform FFT11 implementation.
Results
Figure 2 shows representative examples from each dataset. Both unrolled approaches outperformed the standalone CU-Net, which in general produced more streaking artifacts and blurrier images. For a summary of image quality metrics of all patients, see Table 2. IMUNNE and Alt-UN outperformed CU-Net by a significant margin (p<0.05 with Bonferroni correction) with respect to all reported image quality metrics for all datasets. The image quality metrics were not significantly different (p>0.05) between IMUNNE and Alt-UN, but the training times were shorter for IMUNNE: 41%, 37% and 32% speedup for b-SSFP cine, GRE cine, and perfusion datasets, respectively. Since our approach uses less unrolled iterations, the inference time was also reduced by more than 80% for all datasets.Conclusion
IMUNNE outperformed CU-Net in terms of image quality and a fully unrolled network in terms of computational efficiency. IMUNNE was tested on three disparate cardiac dynamic datasets. A future study is warranted to implement IMUNNE for inline image reconstruction using standard GPU hardware.Acknowledgements
This work is supported by the National Institutes of Health (R01HL116895, 1R01HL167148‐01A1, R01HL151079, R21EB030806A1), the Radiological Society of North America (EILTC2302) and the American Heart Association (19IPLOI34760317, 949899).References
1. Hammernik K, Klatzer T, Kobler E, et al. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med. 2018;79(6):3055-3071.
2. Küstner T, Fuin N, Hammernik K, et al. CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions. Sci Rep. 2020;10(1).
3. Kofler A, Haltmeier M, Schaeffter T, Kolbitsch C. An end-to-end-trainable iterative network architecture for accelerated radial multi-coil 2D cine MR image reconstruction. Med Phys. 2021;48(5):2412-2425.
4. Iakovlev N, Schiffers F, Tapia SL, et al. Memory-Efficient
IMplicit Training Strategy for UNrolled NEtworks (IMUNNE) for Real-Time Cine
Cardiac MR Reconstruction. Proceedings from the 35th Annual Society for
Magnetic Resonance Angiography (SMRA). 2023.
5. Fung SW, Heaton H, Li Q, Mckenzie D, Osher S, Yin W. JFB: Jacobian-Free Backpropagation for Implicit Networks. AAAI. 2022;36(6):6648-56.
6. Shen D, Ghosh S, Haji-Valizadeh H, et al. Rapid reconstruction of highly undersampled, non-Cartesian real-time cine k-space data using a perceptual complex neural network (PCNN). NMR Biomed. 2021;34(1):e4405.
7. Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med. 2007;58(6):1182-1195.
8. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing. 2004;13(4):600-612.
9. Crete F, Dolmiere T, Ladret P, Nicolas M. The blur effect: perception and estimation with a new no-reference perceptual blur metric. Proc. SPIE. 2007;6492.
10. Muckley MJ, Stern R, Murrell T, Knoll F. TorchKbNufft: A High-Level, Hardware-Agnostic Non-Uniform Fast Fourier Transform. ISMRM Workshop on Data Sampling & Image Reconstruction. 2020.
11. Fessler JA, Sutton BP. Nonuniform fast Fourier transforms using min-max interpolation. IEEE Transactions on Signal Processing. 2003;51(2):560-574.
Figures
Figure 1: a) Unrolled network used in the experiments: 4 unrolled blocks containing data consistency and a learned regularizer U-Net (see b)). The architecture is trained “implicitly”, i.e., error is propagated through the last block only, thereby approximating the true gradient.
b) Complex U-Net (CU-Net) architecture: 3 encoding/decoding stages with skip connections. Each stage consists of 2D spatial followed by 1D temporal convolutional layers. Final 3D convolution layer produces the network output. We employed CU-Net both as IMUNNE regularizer and as a standalone network.
Table 2: Qualitative evaluation metrics and times for b-SSFP cine (full training), GRE cine (transfer learning) and the perfusion (transfer learning) datasets.
*Training involves pretraining the CNN block and fine-tuning the end-to-end network.
+#P>0.05 corresponds to non-significant difference in pair.