4760

SUPER-RESOLUTION RECONSTRUCTION OF LATE GADOLINIUM ENHANCEMENT CARDIOVASCULAR MAGNETIC RESONANCE IMAGES USING A RESIDUAL CONVOLUTIONAL NEURAL NETWORK

Archontis Giannakidis^1,2, Ozan Oktay³, Jennifer Keegan^1,2, Veronica Spadotto^1,4, Inga Voges¹, Gillian Smith¹, Iain Pierce¹, Wenjia Bai³, Daniel Rueckert³, Sabine Ernst¹, Michael A Gatzoulis¹, Dudley J Pennell^1,2, Sonya Babu-Narayan¹, and David N Firmin^1,2

¹NIHR Cardiovascular Biomedical Research Unit, Royal Brompton Hospital, London, United Kingdom, ²National Heart & Lung Institute, Imperial College London, London, United Kingdom, ³Biomedical Image Analysis Group, Imperial College London, London, United Kingdom, ⁴Department of Cardiac, Thoracic and Vascular Sciences, University of Padua

Synopsis

Late gadolinium enhancement cardiac magnetic resonance (LGE-CMR) has enabled the accurate myocardial tissue characterization. Due to practical considerations, the acquisition of anisotropic two-dimensional (2D) stack volumes, with low through-plane resolution, still prevails in the clinical routine. We propose a deep learning-based method for reconstructing a super-resolved three-dimensional LGE-CMR data-set from a low resolution 2D short-axis stack volume. The method directly learns the residuals between the high and low resolution images. Results on clinical data-sets show that the proposed technique outperforms the state-of-the-art with regard to image quality. The fast speed of our model furthers facilitates its adoption for practical usage.

PURPOSE

Late gadolinium enhancement cardiovascular magnetic resonance (LGE-CMR)¹ has allowed the differentiation between normal and diseased myocardial tissue. This information may be exploited² to perform risk stratification and guide interventions in various cardiac pathologies. Current clinical LGE-CMR protocols typically involve the acquisition of anisotropic two-dimensional (2D) stack volumes, with low through-plane resolution (≈6-10mm) compared to the high in-plane resolution (≈1.5mm)^3,4. This ensues as an effect of signal-to-noise ratio (SNR) and slice profile considerations⁴. However, the accuracy of the LGE-CMR-based myocardial tissue characterization closely depends on image resolution³. The goal of the single image super-resolution (SISR) methods is to reconstruct a high resolution (HR) image from a single low-resolution (LR) input image. More recently, convolutional neural networks (CNNs) have been brought forward to perform the SISR task on colour natural images⁵. Due to their inherent flexibility to learn complex non-linear relationships between the input and output, CNNs outperformed the state-of-the-art. We propose a framework that adopts a residual CNN to perform the challenging task of reconstructing an HR LGE-CMR data-set from a LR 2D short-axis (SA) stack volume. To the best of our knowledge, this work is the first study to carry out deep learning-based SR reconstruction on LGE-CMR data-sets.

METHODS

(i) Single image network architecture: We adopt a deep CNN architecture⁶ to perform the SISR task for the LGE-CMR data. A deconvolution layer is applied directly to the LR input image to learn the optimal initial up-scaling filters for the specific SR application. The output of the deconvolution layer is then fed to a concatenation of six convolutional layers and rectified linear units (ReLUs)⁷ to estimate the non-linear mapping. The intermediate feature maps h_j⁽ⁿ⁾ at layer n are computed as

$h_j^{(n)}=max(0,\sum_{k=1}^Kh_k^{(n-1)}*w_{kj}^n)$

where * is the convolution operator, k is the filter index, j represents the specific feature map, and $w_{kj}^n$ are the convolutional layer kernels (hidden units). Our model learns the residuals between the LR input and the HR ground truth, rather than predicting the HR image itself. Hence, a simplified regression function is learnt that mostly contains high frequency components. Finally, the predicted residuals are added to the up-scaled output of the deconvolution layer to reconstruct the desired HR output (Fig. 1).

(ii) Clinical data: 28 adult congenital heart disease patients underwent transverse navigator-gated three-dimensional (3D) LGE-CMR imaging^2,8 using a segmented gradient echo sequence. Imaging was performed ~15 min after Gd administration. The spatial resolution of the images was 1.25x1.25x2mm. All data were acquired on a Siemens Avanto 1.5T.

(iii) Implementation: To evaluate the SR reconstruction performance of the adopted up-scaling method, synthetic LR LGE-CMR data-sets (resolution=1.25x1.25x10mm) were generated from the acquired HR LGE-CMR data-sets using an established technique^9,10. The method was compared against other conventional (linear and cubic-spline) and state-of-the-art (multi-atlas patch-match (MAPM)¹¹) image up-scaling techniques. For the model evaluation, we relied on peak signal to noise ratio (PSNR) and structural similarity index (SSIM)¹², where the navigator-gated free-breathing 3D LGE-CMR data-sets with high through-plane resolution were used as a reference for the reconstructed images. The proposed model was evaluated using two rounds of leave-four-out cross-validation. The fidelity of the super-resolved reconstructed images was evaluated using the l1 norm of the error¹³.

RESULTS

The quantitative comparison is summarized in Fig. 2. The proposed CNN model achieved superior image quality of the reconstructed HR images compared with the state-of-the-art MAPM method (p=0.009, Mann-Whitney). A visual comparison of the super-resolved images is given in Fig. 3. Finally, Fig. 4 shows that the proposed SISR network creates sharper images than the MAPM technique which is prone to over-smoothness.

DISCUSSION & CONCLUSIONS

CNN-based SR technology may be of assistance in the quest to achieve 3D isotropic CMR without compromising SNR or requiring long acquisition times. The proposed framework is not computationally demanding, unlike other state-of-the-art techniques¹¹ that require hours for a single reconstruction. In general, more accurate SR reconstruction techniques for LGE-CMR lead to a more truthful characterization of myocardial scar through addressing issues such as enhancement over-estimation due to partial volume effects³. Therefore, the outcomes of this paper may have important clinical implications by the time that LGE-CMR-derived parameters have been shown¹⁴ to outperform other traditional function parameters (such as left ventricular ejection fraction) in predicting adverse cardiac events. The SR method could also find application in other low-level vision problems (e.g. de-noising) or as a pre-processing step for other image analysis tasks (e.g. enhanced tissue classification). Future work will involve the extension of the CNN model to allow the use of multiple LGE-CMR input images (acquired in different viewing planes) to improve performance⁶.

Acknowledgements

This work was supported by funds from the NIHR Cardiovascular Biomedical Research Unit of Royal Brompton & Harefield NHS Foundation Trust and Imperial College London. Ozan Oktay is funded by the Engineering and Physical Sciences Research Council (EPSRC), UK (EPSRC grant: EP/H046410/1). Sonya Babu-Narayan is supported by a British Heart Foundation Intermediate Clinical Research Fellowship. The authors would like to thank Steve Collins and Yasin Karanfil for their help. CMR imaging took place at Royal Brompton Hospital. The SR reconstruction experiments were executed at BiomedIA Group of Imperial College London.

References

[1] R. J. Kim, E. Wu, A. Rafael, E. L. Chen, M. A. Parker, O. Simonetti, F. J. Klocke, R. O. Bonow, and R. M. Judd. The use of contrast-enhanced magnetic resonance imaging to identify reversible myocardial dysfunction, N Engl J Med 2000;343(20): 1445–53.

[2] A. Giannakidis, E. Nyktari, J. Keegan, I. Pierce, I. Suman- Horduna, S. Haldar, D. J. Pennell, R. Mohiaddin, T. Wong, and D. N. Firmin. Rapid automatic segmentation of abnormal tissue in late gadolinium enhancement cardiovascular magnetic resonance images for improved management of long-standing persistent atrial fibrillation. Biomed Eng OnLine 2015;14:88.

[3] O. Dzyubachyk, Q. Tao, D. H. J. Poot, H. J. Lamb, K. Zeppenfeld, B. P. F. Lelieveldt, and R. J. van der Geest. Super-resolution reconstruction of late gadolinium-enhanced MRI for improved myocardial scar assessment. JMRI 2015;42:160–7.

[4] P. Kellman, and A. E. Arai. Cardiac imaging techniques for physicians: Late enhancement. JMRI 2012;36:529–42.

[5] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Trans PAMI 2016;38(2):295–307.

[6] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero, A. de Marvao, S. Cook, D. O’Regan, and D. Rueckert. Multi-input cardiac image super-resolution using convolutional neural netwroks. In: MICCAI (2016).

[7] V. Nair, and G. E. Hinton, Rectified linear units improve restricted Boltzmann machines. In: Proc. Int. Conf. Mach. Learn., 2010, pp. 807–14.

[8] J. Keegan, J. Jhooti, S. V. Babu-Narayan, P. Drivas, S. Ernst, and D. N. Firmin. Improved respiratory efficiency of 3D late gadolinium enhancement imaging using the continuously adaptive windowing strategy (CLAWS). Magn Reson Med 2014;71(3): 1064–74.

[9] H. Greenspan. Super-resolution in medical imaging. Computer J 2009;52(1):43–63.

[10] A. Gholipour, J. A. Estroff, and S. K. Warfield. Robust super-resolution volume reconstruction from slice acquisitions: Application to fetal brain MRI. IEEE Trans Med Imaging 2010;29(10):1739–58.

[11] W. Shi, J. Caballero, C. Ledig, X. Zhuang, W. Bai, K. Bhatia, A. de Marvao, T. Dawes, D. O'Regan, and D. Rueckert. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: MICCAI. pp. 9–16 (2013).

[12] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP 2004;13(4): 600–612.

[13] H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for neural networks for image processing. 2015. arXiv preprint arXiv:1511.08861.

[14] A. T. Yan, A. J. Shayne, K. A. Brown, S. N. Gupta, C. W. Chan, T. M. Luu, M. F. Di Carli, H. G. Reynolds, W. G. Stevenson, and R. Y. Kwong. Characterization of the peri-infarct zone by contrast-enhanced cardiac magnetic resonance imaging is a powerful predictor of post-myocardial infarction mortality. Circulation. 2006;114(1):32-9.

Figures

Fig. 1 The proposed single image super resolution network architecture for up-scaling the LGE-CMR data. Using a deconvolution layer for the initial up-scaling task means that the model is more data-adaptive than other networks which used fixed filters for the same task. Its deep architecture allows for greater accuracy in the estimations due to the additional non-linearities. The number of filters for the six convolutional layers was 64, 64, 32, 16, 16, and 1, respectively. Small 3³ kernels were used to address over-fitting. Residual learning makes the LR-HR mapping problem easier. The Caffe package was employed for the implementation.

Fig. 2 Quantitative comparison of the different image up-scaling methods. Training one fold (24 data-sets) took approximately 6 ½ hours on a NVIDIA Tesla Kepler K80. Inference on a single 3D volume can be performed in 15-20 seconds for the same data-set size and GPU. The Batch Normalization technique was adopted for all hidden layers to facilitate convergence at deeper layers. The loss function was minimized using the Adam Solver. For the MAPM method, 24 atlases were used for each test subject, so that the number of atlases matches the number of data-sets used during the CNN training.

Fig. 3 (a) The ground truth high-resolution image, (b) the low resolution image, (c) the image up-scaled using linear interpolation, (d) the image up-scaled using cubic spline interpolation, (e) the image up-scaled using the proposed CNN-based method.

Fig. 4 The red arrow indicates the typical over-smoothing performed by MAPM in areas closer to boundaries.

Proc. Intl. Soc. Mag. Reson. Med. 25 (2017)

4760