Untrained Learning for Image Acquisition & Reconstruction
1Department of Radiology, University of Pennsylvania, Philadelphia, PA, United States
Synopsis
Keywords: Image acquisition: Reconstruction, Image acquisition: Machine learning, Image acquisition: Fast imaging
Model-based reconstruction via deep learning techniques has shown great promise for reconstructing MR images from highly accelerated acquisition schemes, resulting in substantial scan time reductions. Aside from supervised and self-supervised trained methods that are optimized over large datasets and fast at inference, untrained methods are gaining prominence as a method for image reconstruction in the absence of training datasets. This talk will introduce untrained methods and will focus on their utility for reconstructing multi-contrast MR images in the context of quantitative imaging.Objectives
As a result of attending this talk, participants should take away the following:1. An understanding of the distinction between end-to-end supervised, end-to-end self-supervised, and untrained (scan-specific) deep learning methods for MR image reconstruction.
2. An understanding of the role of untrained methods in quantitative imaging when sufficient training data is not available or accessible.
3. An overview of an untrained method from the literature applied to the task of multi-contrast image reconstruction for quantitative imaging.
Introduction
Compressed sensing was a ground-breaking discovery for MRI reconstruction that demonstrated the feasibility of representing MR images in a sparse basis such that only a subset of k-space data is required for reconstructing an image1,2. As long as the k-space samples are collected in a random fashion, under-sampling schemes below the Nyquist sampling criterion can be achieved, resulting in accelerated imaging. The possible accelerations from compressed sensing, however, are either limited by noise amplification or the reconstructions suffer from residual artifacts regardless of the randomness of the under-sampling pattern3.To circumvent the limitations of compressed sensing, research efforts have shifted toward model-based reconstruction viadeep learning strategies. There are three relevant categories of deep learning approaches toward faster MRI: end-to-end supervised, end-to-end self-supervised, and untrained (scan-specific). Studies relying on supervised methods, however, are often limited by the availability of sufficiently large, curated training datasets. This is an issue that self-supervised4and untrained methods5,6 circumvent by relaxing the need for fully sampled ground truth data. Nevertheless, self-supervised methods still require a dataset for training and validation. Conversely, untrained generative neural networks, instead, operate on a single under-sampled input, relying on the neural network structure for implicit regularization.
The purpose of this talk is to review untrained model-based deep learning methods for accelerated MRI reconstruction, discuss the benefits and drawbacks of untrained methods compared to supervised and self-supervised trained methods, and review examples of untrained methods for the purpose of quantitative imaging.
Methods
Deep learning methods for model-based reconstruction7–10 have shown promise for reconstructing images from highly accelerated raw MRI data. This talk will first review supervised trained methods, like Model-Based Deep Learning (MoDL)8; self-supervised trained methods, like Self-Supervision via Data Under-sampling10; and untrained methods, like ConvDecoder5, DeepDecoder11, and zero-shot learning for physics-guided deep learning12.Next, this talk will focus on a specific untrained method – ConvDecoder with physics-based regularization13 (Figure 1) – for the purpose of recovering multi-contrast images for quantitative imaging and will introduce an early stopping condition informed by this regularization term. At the end, different approaches for improving untrained methods will be discussed, such as reducing optimization time by warm starting the weights of the architecture with weights from similar tissue anatomy.
Discussion
By attending this talk, audience members will understand the distinction between trained and untrained methods. Importantly, attendees will be able to identify suitable uses for untrained methods in the absence of available training datasets. Moreover, attendees will learn how untrained methods rely on early stopping and will review different types of early stopping conditions from the literature.Conclusion
Untrained (i.e. scan specific) deep learning methods for MRI reconstruction are ideal methods when sufficiently large datasets are not available for training self-supervised or supervised approaches. In the untrained regime, the network architecture is iteratively optimized over one example to recover higher frequency features lost during accelerated data acquisition, with physics-based regularization schemes showing promise as an early stopping condition and for simultaneous parameter mapping.Acknowledgements
For the work performed in Slavkova et al. (2023) by the speaker of this talk and collaborators, we thank the National Institutes of Health for funding through NCI U01CA142565, U01CA174706, U01CA253540, U24CA226110, and R01CA240589, as well as NIH U24EB029240. We also extend our gratitude to the American Cancer Society for support through RSG-18-006-01-CCE and the Cancer Prevention and Research Institute of Texas for support through CPRIT RR160005. An Amazon Web Services Machine Learning Research Award was an integral funding component in enabling the use of GPUs. The senior author of Slavkova et al. (2023), T.E. Yankeelov, is a CPRIT Scholar in Cancer Research. The speaker of this talk also extends her gratitude to J.I. Tamir for his mentorship during her graduate school career and his guidance during the preparation of this talk.
References
1. Lustig, M. & Donoho, D. Compressed sensing MRI. IEEE Signal Processing Magazine 72–82 (2008).
2. Lustig, M., Donoho, D. & Pauly, J. M. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med 58, 1182–1195 (2007).
3. Yaman, B., Amir Hossein Hosseini, S. & Akcakaya, M. Zero-Shot Self-Supervised Learning for MRI Reconstruction. in International Conference on Learning Representation (OpenReview, 2022).
4. Liu, F., Kijowski, R., El Fakhri, G. & Feng, L. Magnetic resonance parameter mapping using model‐guided self‐supervised deep learning. Magn Reson Med 85, 3211–3226 (2021).
5. Zalbagi Darestani, M. & Heckel, R. Accelerated MRI With Un-Trained Neural Networks. IEEE Trans Comput Imaging 7, 724–733 (2021).
6. Van Veen, D., Jalal, A., Price, E., Vishwanath, S. & Dimakis, A. G. Compressed Sensing with Deep Image Prior and Learned Regularization. 1–18 (2018).
7. Fessler, J. A. Model-based reconstruction for MRI. IEEE Signal Processing Magazine 27, 81–89 (2010).
8. Aggarwal, H. K., Mani, M. P. & Jacob, M. MoDL: Model-Based Deep Learning Architecture for Inverse Problems. IEEE Trans Med Imaging 38, 394–405 (2019).
9. Hammernik, K. et al. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med 79, 3055–3071 (2018).
10. Yaman, B. et al. Ground-Truth Free Multi-Mask Self-Supervised Physics-Guided Deep Learning in Highly Accelerated MRI. in 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) 1850–1854 (IEEE, 2021). doi:10.1109/ISBI48211.2021.9433924.
11. Heckel, R. & Hand, P. Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks. arXiv preprint (2018).
12. Yaman, B., Amir, S., Hosseini, H. & Akçakaya, M. Zero-Shot Physics-Guided Deep Learning for Subject-Specific MRI Reconstruction. in NeurIPS Workshop on Deep Learning and Inverse Problems (2021).
13. Slavkova, K. P. et al. An untrained deep learning method for reconstructing dynamic MR images from accelerated model‐based data. Magn Reson Med 89, 1617–1633 (2023).
Figures
Figure 1. Overview figure of the untrained framework with physics-based regularization. At each optimization step, the generative model, G(w) (parameterized by weights w), generates a 3D image from a low-dimensional latent space. The image generated by G(w) is then fit to the signal model, yielding image xm. The first term in L(w) enforces data consistency (where yu represents multi-coil, under-sampled k-space data) through image encoding using the MRI forward operator, A; and the second term in L(w) enforces signal model consistency through xm. The reconstructed image is x*.