Synopsis

This talk will provide an introduction to the use of machine learning and neural networks in the field of MR image reconstruction. We will use the example of reconstruction from undersampled data from accelerated acquisitions throughout the talk and will base our formulation on iterative reconstruction methods as used in compressed sensing (CS). We will formulate a network architecture based reconstruction that can be seen as a generalization of CS, and explain how we can learn an entire image reconstruction procedure. Using selected examples, we will discuss both advantages and challenges, covering topics like reconstruction time, design of the training procedure, error metrics and training efficiency and validation of image quality.

Highlights

• Describe how recent developments in machine learning can be used for MR image reconstruction.
• Discuss advantages and challenges, in particular in the light of clinical application.
  

Target audience

Outcome/Objective

To provide an overview of the opportunities and challenges associated with the use of neural networks for MR image reconstruction from accelerated acquisitions.

Purpose

Advantages and Challenges for Machine Learning based Image Reconstruction

Advantages:

• CS relies on incoherence of aliasing artifacts to separate them from the underlying image content, using penalty terms like Total Variation (TV) or l1-wavelet-sparsity⁹. For several clinical sequences, most notably 2D-Cartesian acquisitions achieving incoherence is challenging because of the limited degree of randomness that can be introduced in the pulse sequence¹⁶. In contrast, neural network-based reconstructions can be trained according to a given undersampling trajectory and learn to separate the introduced artifacts from the true image content, thus removing restrictions on the sequence design. The resulting spatial filters and neuron influence functions, which take the place of conventional sparsifying transforms and error norms in CS, also have a much higher complexity. The consequence of this are image models that can result in more natural looking reconstructions.
• Another advantage is that while the training step is usually time consuming (depending on the size of the training data set, it can take several days), the network can be designed that the truly time critical of application of the learned network to new data is extremely efficient. For the examples that will be shown in this talk, the computation times will be in the order of milliseconds.
• Finally, paradoxically, even though the number of tunable parameters is substantially higher than in a conventional CS reconstruction (depending on the network architecture, the number of parameters can be in the order of tens of thousands), the task of regularization parameter tuning is actually simplified because all free parameters are learned during the training phase.

Challenges:

• While recent developments in artificial intelligence, machine learning, computer vision and image restoration can be used as inspirations for developments in image reconstruction, this particular application faces several unique challenges. One particular challenge is the ground-truth data that is used in the training phase. Image processing scientists have access to databases like imageNet (http://image-net.org) that provide millions of examples for training and validation. While hospital PACS system could in principle provide a similar amount of data, they only archive dicom images and not the rawdata which is necessary in the context of image reconstruction. Despite ongoing initiatives by the ISMRM like MR-Hub (http://www.ismrm.org/MR-Hub/) and MRI UNBOUND (http://www.ismrm.org/mri_unbound/), a large scale publicly available archive for MR-rawdata and corresponding reference reconstructions is still an unmet need. In addition, acquisition of these uncorrupted training data is sometimes non-trivial in itself. In the case of undersampling to accelerate data acquisition, training examples correspond to fully sampled acquisitions. These longer scantimes can introduce additional artifacts, e.g. due to subject motion. Especially for applications like dynamic imaging, which would benefit substantially from accelerated data acquisition it is unclear how to generate a ground truth with both high spatial and temporal resolution.
• Robustness and generalization potential are essential for translation of neural network based image reconstruction from a research environment to clinical routine use. A trained network must be general enough such that it can deal with anatomical variations, additional artifacts in the images (e.g. metal implants) and case-by case modifications of the scan protocols, which can result in changes of contrast, aliasing artifacts and matrix voxel size and. While initial results are promising¹⁷, generalization with respect to diagnostic content has yet to be evaluated in clinical patient studies.
• The design of the loss function that is used to train a system for iterative image reconstruction is essential for its success. Image metrics like pixel-wise mean-squared error or the structural similarity index that are commonly used in image processing have several shortcomings when it comes to medical images. They substantially penalize changes in contrast and noise, but are not particularly sensitive towards loss of small low contrast structures, which are often the essential diagnostic content of the images. The design of the loss function must also take the complex nature of MRI data into account.

Discussion and conclusion

Acknowledgements

References

[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.

[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.

[3] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs,” in International Conference on Learning Representations, 2015.

[4] A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazirbas ¸, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox, “FlowNet: Learning Optical Flow with Convolutional Networks,” in IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2758–2766.

[5] Y. Chen, W. Yu, and T. Pock, “On learning optimized reaction diffusion processes for effective image restoration,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, pp. 5261–5269.

[6] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016.

[7] K. Hammernik, F. Knoll, D. K. Sodickson, and T. Pock, “Learning a Variational Model for Compressed Sensing MRI Reconstruction,” in Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), 2016, no. 24, p. 1088.

[8] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, D. Liang, “Accelerating magnetic resonance imaging via deep learning”, ISBI 514-517 (2016).

[9] K.H. Jin, M.T. McCann, E. Froustey, M, Unser, “Deep Convolutional Neural Network for Inverse Problems in Imaging”, https://arxiv.org/abs/1611.03679 (2016).

[10] K. Kwon, D. Kim, H. Seo, J. Cho, B. Kim, H.W. Park, “Learning-based Reconstruction using Artificial Neural Network for Higher Acceleration”, in Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), 2016, no. 24, p. 1801.

[11] G. Wang, “Perspective on Deep Imaging”, IEEE Access 8914-8924 (2016).

[12] V Golkov, A Dosovitskiy, J.I. Sperl, M.I. Menzel, M. Czisch, P. Saemann, T. Brox, D. Cremers, “q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI Scans”, IEEE TMI 35: 1344-1351 (2016).

[13] K. P. Pruessmann, M. Weiger, P. Boernert, and P. Boesiger, “Advances in sensitivity encoding with arbitrary k-space trajectories,” Magn Reson Med, vol. 46, no. 4, pp. 638–651, 2001.

[14] K. T. Block, M. Uecker, and J. Frahm, “Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint,” Magn Reson Med, vol. 57, no. 6, pp. 1086–1098, Jun. 2007.

[15] M. Lustig, D. Donoho, and J. M. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging.,” Magn Reson Med, vol. 58, no. 6, pp. 1182–1195, 2007.

[16] K. G. Hollingsworth, “Reducing acquisition time in clinical MRI by data undersampling and compressed sensing reconstruction,” Phys Med Biol, vol. 60, no. 21, pp. R297--R322, Nov. 2015.

[17] F Knoll, K Hammernik, E Garwood, A Hirschmann, L Rybak, M Bruno, T Block, J Babb, T Pock, DK Sodickson and MP Recht, “Accelerated knee imaging using a deep learning based reconstruction” in Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), 2017, no. 25 (in press).