In this educational, we give an overview of the current developments in deep learning-based MRI reconstruction of undersampled k-space data. We show the advantages of deep learning-based approaches over compressed sensing approaches in terms of improved image quality and suppressed artifacts. We will also discuss several challenges that are encountered during learning covering the design of a training database, deep network architectures and image quality measures.
Compressed Sensing (CS)1 allows for accelerated data acquisition below the Nyquist rate under three conditions. These are incoherent undersampling artifacts, sparsity in some transform domain and a nonlinear reconstruction algorithm. However, translating CS approaches to clinical practice is challenging: Many clinical protocols are based on a Cartesian sampling scheme2, which violates the incoherence assumption. We also have to choose a fixed sparsifying transform that makes simplified assumptions on the underlying tissue. In the non-linear reconstruction approach, we have to make a trade-off between the sparsifying transform and the influence of the raw data which means that each CS based reconstruction is treated as an individual optimization problem.
In CS, no explicit knowledge of the appearance of undersampling artifacts and the imaged anatomy is used, which is in strong contrast to human learning. Translating human learning to accelerated MRI reconstruction means that we can learn from suitable training data how to distinguish true image content from undersampling artifacts. This shifts the optimization to the time-consuming learning stage. Once a model is learned, we use it to reconstruct previously unseen data efficiently. The immediate benefits are that we do not require additional parameter tuning and the image quality is significantly improved once the model is learned.
While there are enormous developments in using deep learning3 for computer vision, such as classification4, optical flow5 or image restoration6, the application to medical image reconstruction is still at the very beginning. Starting in 2016, first pilot studies that use deep learning for MRI reconstruction7-13 were introduced. We will give an overview of these developments and also discuss challenges that are encountered when using deep learning for accelerated MRI reconstruction.
In general, we require three ingredients for successful learning approaches: Suitable training data, a deep network architecture and a loss function to measure the similarity between a clean and a corrupted image. However, when translating the knowledge of deep learning from computer vision to MRI reconstruction, we experience a number of challenges that we will discuss during this educational.
Training Data: In a breakthrough paper, Krizhevsky et al.4 showed excellent results on the ImageNet 2012 classification benchmark that consists of a large scale image classification task with more than 10.000 different object classes. It is known from classification that extremely large data sets with millions of labeled images are necessary in order to obtain outstanding results. However, in MRI reconstruction no such database is available. In the case of reconstruction of undersampled data, artifact-free reference data generated from fully sampled data are required during training, however, defining and acquiring proper reference data is not a trivial task.
Deep Network Design: When training data is available, we need to find a suitable network architecture to reconstruct artifact-free images. To form a deep network architecture, relatively simple layers that represent weight vectors or convolution kernels followed by non-linear activation functions (e.g. ReLu) are stacked together. Typically, such models have several millions of free model parameters, which have to be properly tuned during a training stage in order to make the network applicable for a certain task. In MRI reconstruction, we additionally have to consider the complex domain of MRI images in the network design. Depending on the used data, both magnitude and/or phase images are of great interest.
Loss Functions: For successful training of deep network architectures, it is essential to use a suitable loss function. The loss defines the goal of the network training and strongly influences the resulting performance. One of the most popular loss functions for training a deep network architecture is the mean-squared error. However, this choice does not always lead to the desired results for MRI reconstruction, especially when dealing with low SNR levels of certain MRI sequences.
1. M. Lustig, D. Donoho, and J. M. Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med, 2007, no. 58, pp. 1182-1195.
2. K. G. Hollingsworth. Reducing acquisition time in clinical MRI by data undersampling and compressed sensing reconstruction. Phys Med Biol, 2015, vol. 60, no. 21, pp. R297-R322.
3. Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 2015, vol. 521, no. 7553, pp. 436-444.
4. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.
5. A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. FlowNet: Learning Optical Flow with Convolutional Networks. IEEE International Conference on Computer Vision, 2015, pp. 2758–2766.
6. Y. Chen, W. Yu, and T. Pock. On learning optimized reaction diffusion processes for effective image restoration. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, pp. 5261–5269.
7. K. Hammernik, F. Knoll, D. K. Sodickson, and T. Pock. Learning a Variational Model for Compressed Sensing MRI Reconstruction. Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), 2016, no. 24, p. 1088.
8. G. Wang. Perspective on Deep Imaging. IEEE Access, 2016, pp. 8914-8924.
9. S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, D. Liang. Accelerating magnetic resonance imaging via deep learning. IEEE International Symposium on Biomedical Imaging, 2016, pp. 514-517.
10. K. Kwon, D. Kim, H. Seo, J. Cho, B. Kim, H.W. Park. Learning-based Reconstruction using Artificial Neural Network for Higher Acceleration. Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), 2016, no. 24, p. 1801.
11. Y. Yang, J. Sun, H. Li, Z. Xu. Deep ADMM-Net for Compressive Sensing MRI. Advances in Neural Information Processing Systems, 2016, pp. 10-18.
12. Y. S. Han, J. Yoo and J. C. Ye. Deep Learning with Domain Adaptation for Accelerated Projection Reconstruction MR. arxiv:1703.01135, 2017.
13. D. Lee, J. Yoo and J. C. Ye. Deep artifact learning for compressed sensing and parallel MRI. arxiv:1703.01120, 2017.