4760
SUPER-RESOLUTION RECONSTRUCTION OF LATE GADOLINIUM ENHANCEMENT CARDIOVASCULAR MAGNETIC RESONANCE IMAGES USING A RESIDUAL CONVOLUTIONAL NEURAL NETWORK1NIHR Cardiovascular Biomedical Research Unit, Royal Brompton Hospital, London, United Kingdom, 2National Heart & Lung Institute, Imperial College London, London, United Kingdom, 3Biomedical Image Analysis Group, Imperial College London, London, United Kingdom, 4Department of Cardiac, Thoracic and Vascular Sciences, University of Padua
Synopsis
Late gadolinium enhancement cardiac magnetic resonance (LGE-CMR) has enabled the accurate myocardial tissue characterization. Due to practical considerations, the acquisition of anisotropic two-dimensional (2D) stack volumes, with low through-plane resolution, still prevails in the clinical routine. We propose a deep learning-based method for reconstructing a super-resolved three-dimensional LGE-CMR data-set from a low resolution 2D short-axis stack volume. The method directly learns the residuals between the high and low resolution images. Results on clinical data-sets show that the proposed technique outperforms the state-of-the-art with regard to image quality. The fast speed of our model furthers facilitates its adoption for practical usage.
PURPOSE
Late gadolinium enhancement cardiovascular magnetic resonance (LGE-CMR)1 has allowed the differentiation between normal and diseased myocardial tissue. This information may be exploited2 to perform risk stratification and guide interventions in various cardiac pathologies. Current clinical LGE-CMR protocols typically involve the acquisition of anisotropic two-dimensional (2D) stack volumes, with low through-plane resolution (≈6-10mm) compared to the high in-plane resolution (≈1.5mm)3,4. This ensues as an effect of signal-to-noise ratio (SNR) and slice profile considerations4. However, the accuracy of the LGE-CMR-based myocardial tissue characterization closely depends on image resolution3. The goal of the single image super-resolution (SISR) methods is to reconstruct a high resolution (HR) image from a single low-resolution (LR) input image. More recently, convolutional neural networks (CNNs) have been brought forward to perform the SISR task on colour natural images5. Due to their inherent flexibility to learn complex non-linear relationships between the input and output, CNNs outperformed the state-of-the-art. We propose a framework that adopts a residual CNN to perform the challenging task of reconstructing an HR LGE-CMR data-set from a LR 2D short-axis (SA) stack volume. To the best of our knowledge, this work is the first study to carry out deep learning-based SR reconstruction on LGE-CMR data-sets.METHODS
(i) Single image network architecture: We adopt a deep CNN architecture6 to perform the SISR task for the LGE-CMR data. A deconvolution layer is applied directly to the LR input image to learn the optimal initial up-scaling filters for the specific SR application. The output of the deconvolution layer is then fed to a concatenation of six convolutional layers and rectified linear units (ReLUs)7 to estimate the non-linear mapping. The intermediate feature maps hj(n) at layer n are computed as
$$h_j^{(n)}=max(0,\sum_{k=1}^Kh_k^{(n-1)}*w_{kj}^n)$$
where * is the convolution operator, k is the filter index, j represents the specific feature map, and $$$w_{kj}^n$$$ are the convolutional layer kernels (hidden units). Our model learns the residuals between the LR input and the HR ground truth, rather than predicting the HR image itself. Hence, a simplified regression function is learnt that mostly contains high frequency components. Finally, the predicted residuals are added to the up-scaled output of the deconvolution layer to reconstruct the desired HR output (Fig. 1).
(ii) Clinical data: 28 adult congenital heart disease patients underwent transverse navigator-gated three-dimensional (3D) LGE-CMR imaging2,8 using a segmented gradient echo sequence. Imaging was performed ~15 min after Gd administration. The spatial resolution of the images was 1.25x1.25x2mm. All data were acquired on a Siemens Avanto 1.5T.
(iii) Implementation: To evaluate the SR reconstruction performance of the adopted up-scaling method, synthetic LR LGE-CMR data-sets (resolution=1.25x1.25x10mm) were generated from the acquired HR LGE-CMR data-sets using an established technique9,10. The method was compared against other conventional (linear and cubic-spline) and state-of-the-art (multi-atlas patch-match (MAPM)11) image up-scaling techniques. For the model evaluation, we relied on peak signal to noise ratio (PSNR) and structural similarity index (SSIM)12, where the navigator-gated free-breathing 3D LGE-CMR data-sets with high through-plane resolution were used as a reference for the reconstructed images. The proposed model was evaluated using two rounds of leave-four-out cross-validation. The fidelity of the super-resolved reconstructed images was evaluated using the l1 norm of the error13.
RESULTS
The quantitative comparison is summarized in Fig. 2. The proposed CNN model achieved superior image quality of the reconstructed HR images compared with the state-of-the-art MAPM method (p=0.009, Mann-Whitney). A visual comparison of the super-resolved images is given in Fig. 3. Finally, Fig. 4 shows that the proposed SISR network creates sharper images than the MAPM technique which is prone to over-smoothness.DISCUSSION & CONCLUSIONS
CNN-based SR technology may be of assistance in the quest to achieve 3D isotropic CMR without compromising SNR or requiring long acquisition times. The proposed framework is not computationally demanding, unlike other state-of-the-art techniques11 that require hours for a single reconstruction. In general, more accurate SR reconstruction techniques for LGE-CMR lead to a more truthful characterization of myocardial scar through addressing issues such as enhancement over-estimation due to partial volume effects3. Therefore, the outcomes of this paper may have important clinical implications by the time that LGE-CMR-derived parameters have been shown14 to outperform other traditional function parameters (such as left ventricular ejection fraction) in predicting adverse cardiac events. The SR method could also find application in other low-level vision problems (e.g. de-noising) or as a pre-processing step for other image analysis tasks (e.g. enhanced tissue classification). Future work will involve the extension of the CNN model to allow the use of multiple LGE-CMR input images (acquired in different viewing planes) to improve performance6.Acknowledgements
This work was supported by funds from the NIHR Cardiovascular Biomedical Research Unit of Royal Brompton & Harefield NHS Foundation Trust and Imperial College London. Ozan Oktay is funded by the Engineering and Physical Sciences Research Council (EPSRC), UK (EPSRC grant: EP/H046410/1). Sonya Babu-Narayan is supported by a British Heart Foundation Intermediate Clinical Research Fellowship. The authors would like to thank Steve Collins and Yasin Karanfil for their help. CMR imaging took place at Royal Brompton Hospital. The SR reconstruction experiments were executed at BiomedIA Group of Imperial College London.References
[1] R. J. Kim, E. Wu, A. Rafael, E. L. Chen, M. A. Parker, O. Simonetti, F. J. Klocke, R. O. Bonow, and R. M. Judd. The use of contrast-enhanced magnetic resonance imaging to identify reversible myocardial dysfunction, N Engl J Med 2000;343(20): 1445–53.
[2] A. Giannakidis, E. Nyktari, J. Keegan, I. Pierce, I. Suman- Horduna, S. Haldar, D. J. Pennell, R. Mohiaddin, T. Wong, and D. N. Firmin. Rapid automatic segmentation of abnormal tissue in late gadolinium enhancement cardiovascular magnetic resonance images for improved management of long-standing persistent atrial fibrillation. Biomed Eng OnLine 2015;14:88.
[3] O. Dzyubachyk, Q. Tao, D. H. J. Poot, H. J. Lamb, K. Zeppenfeld, B. P. F. Lelieveldt, and R. J. van der Geest. Super-resolution reconstruction of late gadolinium-enhanced MRI for improved myocardial scar assessment. JMRI 2015;42:160–7.
[4] P. Kellman, and A. E. Arai. Cardiac imaging techniques for physicians: Late enhancement. JMRI 2012;36:529–42.
[5] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Trans PAMI 2016;38(2):295–307.
[6] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero, A. de Marvao, S. Cook, D. O’Regan, and D. Rueckert. Multi-input cardiac image super-resolution using convolutional neural netwroks. In: MICCAI (2016).
[7] V. Nair, and G. E. Hinton, Rectified linear units improve restricted Boltzmann machines. In: Proc. Int. Conf. Mach. Learn., 2010, pp. 807–14.
[8] J. Keegan, J. Jhooti, S. V. Babu-Narayan, P. Drivas, S. Ernst, and D. N. Firmin. Improved respiratory efficiency of 3D late gadolinium enhancement imaging using the continuously adaptive windowing strategy (CLAWS). Magn Reson Med 2014;71(3): 1064–74.
[9] H. Greenspan. Super-resolution in medical imaging. Computer J 2009;52(1):43–63.
[10] A. Gholipour, J. A. Estroff, and S. K. Warfield. Robust super-resolution volume reconstruction from slice acquisitions: Application to fetal brain MRI. IEEE Trans Med Imaging 2010;29(10):1739–58.
[11] W. Shi, J. Caballero, C. Ledig, X. Zhuang, W. Bai, K. Bhatia, A. de Marvao, T. Dawes, D. O'Regan, and D. Rueckert. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: MICCAI. pp. 9–16 (2013).
[12] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP 2004;13(4): 600–612.
[13] H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for neural networks for image processing. 2015. arXiv preprint arXiv:1511.08861.
[14] A. T. Yan, A. J. Shayne, K. A. Brown, S. N. Gupta, C. W. Chan, T. M. Luu, M. F. Di Carli, H. G. Reynolds, W. G. Stevenson, and R. Y. Kwong. Characterization of the peri-infarct zone by contrast-enhanced cardiac magnetic resonance imaging is a powerful predictor of post-myocardial infarction mortality. Circulation. 2006;114(1):32-9.
Figures
