0017

Low-Latency Reconstruction of Real-Time Cine MRI Using an Unrolled Network
Marc Vornehm1,2, Jens Wetzl2, Florian Fürnrohr1, Daniel Giese2,3, Rizwan Ahmad4, and Florian Knoll1
1Computational Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, 2Magnetic Resonance, Siemens Healthcare GmbH, Erlangen, Germany, 3Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, 4Biomedical Engineering, The Ohio State University, Columbus, OH, United States

Synopsis

Keywords: Machine Learning/Artificial Intelligence, Image Reconstruction

Motivation: Interactive real-time MRI requires low reconstruction latencies. Deep learning-based methods are promising, but unrolled networks like the Variational Network have longer inference times than purely image-based methods.

Goal(s): Design and train a Variational Network with high reconstruction quality and inference times suitable for interactive real-time applications.

Approach: Modify the Variational Network architecture such that few unrolling steps are sufficient for high reconstruction quality with short inference times.

Results: The proposed architectural modifications allowed to halve the number of unrolling steps without compromising image quality, therefore enabling considerably shortened reconstruction times.

Impact: Two modifications to an unrolled Variational Network architecture for MRI reconstruction are proposed. These enable reconstructing interactive real-time cardiac cine MRI with high reconstruction quality while maintaining minimal reconstruction latency.

Introduction

Interactive real-time MRI is used in MRI-guided cardiac interventions, posing high demands on reconstruction latency1. For device navigation, for instance, a maximum latency of 200$$$\,$$$ms is considered tolerable2. Achieving both high temporal and spatial resolutions in these applications is challenging because it requires high acceleration rates, in turn necessitating advanced reconstruction techniques like compressed sensing (CS). These, however, may be infeasible due to their reconstruction latency.

Deep learning has gained interest for MRI reconstruction due to its performance and speed. Current approaches for interventional applications focus on artifact suppression in image space3,4,5. State-of-the-art unrolled networks like the Variational Network6,7 (VN) also consider measurement data during reconstruction. They contain several unrolling steps (cascades), each of which increases reconstruction time. We present two modifications to a VN that allow to halve the number of cascades without sacrificing image quality, therefore considerably reducing reconstruction latency.

Methods

The network architecture is based on a VN with residual U-Nets in each refinement block and spatiotemporal convolutions9. It takes undersampled $$$k$$$-space data from the $$$n$$$ most recent frames as input and generates $$$n$$$ frames as output of the last cascade, where $$$n$$$ is chosen as 7. The last frame is then extracted from this cine series. During training, a combination of SSIM-loss and $$$\perp$$$-loss10 is computed on this frame, and during inference, it serves as the network’s output.

We propose two changes to the VN:
  1. $$$k$$$-$$$t$$$-weighted refinement11: A learnable weighting function $$$W_n(k_x,k_y,t)$$$ is multiplied to the output of each cascade’s refinement block. This function is piecewise linear and defined by 10x10x10 control points optimized during training. It is tri-linearly interpolated to the required size in $$$k$$$-$$$t$$$-space before multiplication.
  2. Conjugate gradient (CG) initialization: $$$k$$$-space data is first reconstructed using five iterations of CG with Tikhonov regularization, where the regularization weight is optimized during training.
The architecture is depicted in Figure 1. We trained networks with five VN cascades with and without the described modifications. We further trained networks with 6-10 cascades to demonstrate network performance improvement with the number of unrolling steps.

All networks were trained on fully sampled cine data from the OCMR dataset12 with a training/validation split of 186/44 slices. From each cine, every window of $$$n$$$ subsequent frames was considered as a sample and a golden ratio variable density cartesian sampling mask13 was used for undersampling. For testing, 14 prospectively undersampled real-time cine series from the OCMR dataset were used. These were acquired using the same undersampling pattern as used for retrospective undersampling of the training data. Acceleration rates varied between six and ten. The test datasets were input into the network using a sliding window approach and the final reconstructed real-time cine was obtained by concatenating the last frame of each reconstructed window.

Reference reconstructions of test data were obtained with CS with temporal total variation regularization using Bart14, reconstructing all frames at once. Structural similarity (SSIM) was computed between concatenated VN reconstructions and CS reconstructions.

Training and testing were conducted on NVIDIA A100 GPUs and reconstruction time per frame was measured, including network inference and data transfer to/from the GPU for the latest frame. Student’s $$$t$$$-tests were performed comparing results of experiments with five cascades and different combinations of the proposed modifications, and between experiments with differing number of cascades without modifications. $$$p$$$-values below 0.05 were considered significant.

Results

Quantitative results are presented in Figure 2, exemplary reconstructions in Figures 3 and 4. Each architectural modification, both individually and combined, significantly improved SSIM values. Increasing the number of cascades also significantly improved reconstructions, while also increasing reconstruction time.

Discussion

Both $$$k$$$-$$$t$$$-weighted refinement and CG initialization significantly improved reconstruction quality for a small VN with only five cascades. This improvement is evident in the SSIM values and the presented reconstructions, where flickering is reduced in the modified versions. Without these modifications, similar reconstruction quality was only achieved at ~9 VN cascades, which resulted in reconstruction times of 103$$$\,$$$ms per frame, compared to 69$$$\,$$$ms for five cascades with both modifications.

CG initialization provides an initial estimate for the optimization problem solved iteratively by the VN, therefore reducing the number of required unrolling steps. The $$$k$$$-$$$t$$$-weighted refinement can be interpreted as a convolutional layer with a large convolution kernel appended to the U-Net in image space, but leveraging the convolution theorem for computational efficiency.

Conclusion

We presented a neural network for reconstructing interactive real-time cardiac cine MRI. Two architectural modifications were proposed, allowing to reduce the number of cascades in the network and hence reconstruction time. Our findings indicate the feasibility of unrolled networks for interactive MRI reconstruction.

Acknowledgements

This research was supported by NIH/NIBIB grant R01EB029957. In addition, we gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the NHR project b143dc. NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) – 440719683.

References

  1. Nayak KS, Lim Y, Campbell‐Washburn AE, Steeden J. Real‐Time Magnetic Resonance Imaging. J Magn Reson Imaging. 2022;55(1):81-99.
  2. Campbell-Washburn AE, Tavallaei MA, Pop M, et al. Real-time MRI guidance of cardiac interventions. J Magn Reson Imaging. 2017;46(4):935-950.
  3. Hauptmann A, Arridge S, Lucka F, Muthurangu V, Steeden JA. Real‐time cardiovascular MR with spatio‐temporal artifact suppression using deep learning–proof of concept in congenital heart disease. Magn Reson Med. 2019;81(2):1143-1156.
  4. Jaubert O, Montalt‐Tordera J, Knight D, et al. Real‐time deep artifact suppression using recurrent U‐Nets for low‐latency cardiac MRI. Magn Reson Med. 2021;86(4):1904-1916.
  5. Jaubert O, Montalt‐Tordera J, Knight D, Arridge S, Steeden J, Muthurangu V. HyperSLICE: HyperBand optimized spiral for low‐latency interactive cardiac examination. Magn Reson Med. 2023:mrm.29855.
  6. Hammernik K, Klatzer T, Kobler E, et al. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med. 2018;79(6):3055-3071.
  7. Sriram A, Zbontar J, Murrell T, et al. End-to-End Variational Networks for Accelerated MRI Reconstruction. In: Martel AL, Abolmaesumi P, Stoyanov D, et al., eds. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Vol 12262. Lecture Notes in Computer Science. Cham: Springer International Publishing; 2020:64-73.
  8. Muckley MJ, Riemenschneider B, Radmanesh A, et al. Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction. IEEE Trans Med Imaging. 2021;40(9):2306-2317.
  9. Vornehm M, Wetzl J, Giese D, Ahmad R, Knoll F. Spatiotemporal variational neural network for reconstruction of highly accelerated cardiac cine MRI. European Heart Journal - Cardiovascular Imaging. 2022;23(Supplement_2):jeac141.018.
  10. Terpstra ML, Maspero M, Sbrizzi A, van den Berg CAT. ⊥-loss: A symmetric loss function for magnetic resonance imaging reconstruction and image registration with deep learning. Medical Image Analysis. 2022;80:102509.
  11. Vornehm M, Wetzl J, Giese D, Knoll F. k-t adaptive Regularization in Variational Networks for Cardiac Cine Reconstruction. In: Proc ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, AZ, USA; 2023.
  12. Chen C, Liu Y, Schniter P, et al. OCMR (v1.0)—Open-Access Multi-Coil k-Space Dataset for Cardiovascular Magnetic Resonance Imaging. arXiv:2008.03410 [eess.IV]. 2020.
  13. Joshi M, Pruitt A, Chen C, Liu Y, Ahmad R. Technical Report (v1.0)—Pseudo-random Cartesian Sampling for Dynamic MRI. arXiv:2206.03630v1 [eess.SP]. 2022.
  14. Blumenthal M, Holme C, Roeloffs V, et al. mrirecon/bart: version 0.8.00. 2022.

Figures

Fig. 1: Network architecture with conjugate gradient (CG) initialization and $$$k$$$-$$$t$$$-weighted refinement. The Variational Network contains $$$n$$$ cascades, each consisting of data consistency (DC) and a refinement block (R). Each refinement block contains a residual U-Net $$$f_{\Phi,n}$$$. Learnable parameters (marked in red) are the parameters of the U-Nets $$$f_{\Phi,n}$$$, the data consistency weights $$$\lambda_n$$$, the refinement weighting functions $$$W_n$$$, and the CG regularization parameter $$$\mu$$$.

Fig. 2: Structural similarity (SSIM) and reconstruction time per frame for the conducted experiments. Student’s $$$t$$$-tests were performed comparing results of experiments 1-4 with each other and comparing results of experiments 5-9 with experiment 1.

Fig. 3: Exemplary reconstructions with five VN cascades and different combinations of CG initialization and $$$k$$$-$$$t$$$-weighted refinement.

Fig. 4: Exemplary reconstructions with five VN cascades and different combinations of CG initialization and $$$k$$$-$$$t$$$-weighted refinement.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
0017
DOI: https://doi.org/10.58530/2024/0017