4271
Unsupervised MRI Super-Resolution Reconstruction Using a Hybrid Regularizer Powered Deep Image Prior1Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China, 2Department of Radiology, The Affiliated Drum Tower Hospital of Nanjing University Medical School, Nanjing, China, 3Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, Shenzhen, China, 4United Imaging Research Institute of Innovative Medical Equipment, Shenzhen, China
Synopsis
Keywords: Image Reconstruction, Brain
Motivation: While deep networks have shown great effectiveness for post-acquisition MRI resolution enhancement, their training requires an enormous of datasets. Deep Image Prior (DIP) is a novel approach that leverages the inductive bias of deep convolutional architecture, allowing for MRI super-resolution without the need for training.
Goal(s): We aim to improve the capabilities of DIP and thus achieve resolution enhancement.
Approach: We introduced a hybrid regularizer that integrates total variation with a neural network denoiser into the DIP framework.
Results: Validated on 5T MR datasets, our method further improved on DIP and generated high-resolution MRI with realistic details, outshining several competing methods.
Impact: The proposed unsupervised method offered a robust framework for MRI super-resolution reconstruction that leverages intrinsic image structure to ensure resolution enhancement without the need for training data, thus boosting the efficiency of medical imaging and potentially benefiting clinical diagnostics.
Introduction
The enhancement of spatial resolution in MRI leads to longer scan time and decreased signal-to-noise ratio1-3. While deep networks have shown great effectiveness for post-acquisition MRI resolution enhancement4-6, their training requires many low-high resolution pairs, which are not always available. The Deep Image Prior (DIP) provides a novel solution to this issue7. It is a recent regularization framework that leverages the architecture of a convolution neural network (CNN), with training tailored to a low-resolution image. In this work, we enhanced the capabilities of DIP by integrating total variation (TV) and deep denoiser regularizers, leveraging the Half Quadratic Splitting (HQS) scheme. The TV regularizer excels at preserving edges, while the CNN denoiser learned from vast amounts of data captures intricate image details. Furthermore, we validated the effectiveness of the proposed method for enhancing the resolution of 5T MR images.Methods
In DIP framework, a high-resolution (HR) MRI x^∈R(H×W) is estimated by a CNN G(θ*;z) with weights θ* and random-noise input z:$\[\hat x = G\left( {{\theta ^*};z} \right)\].(1)$
Given a low-resolution (LR) image x∈R(H/t×W/t), the weights are obtained via:
$\[\theta^*=\arg\min_{\theta}\frac{1}{2}||AG(\theta;z)-x||_2^2\],(2)$
where A:R(H×W)→R^(H/t×W/t) is a downsampling operator with factor t. To boost the capabilities of DIP, we develop a hybrid scheme which combines TV and CNN denoiser regularizations into the optimization. Therefore, we obtain:
$\[\theta^{\star}=\arg\min_{\theta,y,t}||AG(\theta;z)-x||_2^2+\lambda_{1}||y||_{TV}+\lambda_{2}\varphi(t) s.t. y=G(\theta;z), t=G(\theta;z)\]. (3)$
Where ‖∙‖TV is TV norm8 and φ(∙) is a learned regularization, λ1 and λ2 are penalty parameters.
To solve this problem, we use the augmented Lagrangian with scaled dual variable y, t and penalty parameters μ1, μ2
$\[L=\arg\min_{\theta,y,t}\frac{1}{2}||AG(\theta;z)-x||_2^2+\lambda_{1}||y||_{TV}+\lambda_{2}\varphi(t)+\frac{\mu_{1}}{2}||y-G(\theta;z)||_2^2+\frac{\mu_{2}}{2}||t-G(\theta;z)||_2^2\]. (4)$
The above minimization problem is solved by the iterative HQS method9. In addition, we choose a state-of-the-art deep denoiser from the literature10 into the alternating iterations. The detailed steps for HQS are summarized in Fig.1.
The backbone of our network G is a U-Net-like7 architecture (Fig.2). Downsampling of the outout is performed by Lanczos interpolation11, which is differentiable.
Results
Dataset: We evaluated our method on both T1 and T2 weighted MR images acquired by a 5T MRI scanner (uMR Jupiter, United Imaging, Shanghai, China). Specifically, we randomly selected 50 slices from these data to evaluate the proposed method. Note that our training required only LR images. Meanwhile, we employed the HR images as the ground truth for evaluation.Experimental Results: We considered an undersampling factor of two and compared our method with Bicubic, DIP and TV8. Both PSNR and SSIM results (Fig.3) show that our method outperforms various competing methods, especially for T1w images. In addition, visual outcomes (Fig.4) indicate the superiority of our method, with revealing clearer anatomical details that closely approximate the ground truth.
Conclusion
We proposed the improved DIP framework and validated its effectiveness for super-resolution reconstruction of 5T MR images. Our method required only a LR image for training, and transformed low-resolution images into their high-resolution counterparts, yielding super-resolved images that closely approximate the ground truth.Acknowledgements
The study was partially supported by Natural Science Foundation of Guangdong Province-Outstanding Youth Project (2023B1515020002), National Key Technology Research and Development Program of China (2021YFF0501502), Key Laboratory for Magnetic Resonance and Multimodality Imaging of Guangdong Province (2023B1212060052), and Central guidance for local science and technology development project (ZYYD2023D02).References
1.Berggren K, Ryd D, Heiberg E, Aletras AH, Hedström E. Super-Resolution Cine Image Enhancement for Fetal Cardiac Magnetic Resonance Imaging. J Magn Reson Imaging. 2022 Jul;56(1):223-231. doi: 10.1002/jmri.27956.
2. Song L, Wang Q, Liu T, Li H, Fan J, Yang J, Hu B. Deep robust residual network for super-resolution of 2D fetal brain MRI. Sci Rep. 2022 Jan 10;12(1):406. doi: 10.1038/s41598-021-03979-1. PMID: 35013383; PMCID: PMC8748749.
3. Chaudhari AS, Fang Z, Kogan F, Wood J, Stevens KJ, Gibbons EK, Lee JH, Gold GE, Hargreaves BA. Super-resolution musculoskeletal MRI using deep learning. Magn Reson Med. 2018 Nov;80(5):2139-2154. doi: 10.1002/mrm.27178. Epub 2018 Mar 26. PMID: 29582464; PMCID: PMC6107420.
4.Zhou Z, Ma A, Feng Q, Wang R, Cheng L, Chen X, Yang X, Liao K, Miao Y, Qiu Y. Super-resolution of brain tumor MRI images based on deep learning. J Appl Clin Med Phys. 2022 Nov;23(11):e13758. doi: 10.1002/acm2.13758. Epub 2022 Sep 15. PMID: 36107021; PMCID: PMC9680577.
5.Masutani EM, Bahrami N, Hsiao A. Deep Learning Single-Frame and Multiframe Super-Resolution for Cardiac MRI. Radiology. 2020 Jun;295(3):552-561. doi: 10.1148/radiol.2020192173. Epub 2020 Apr 14. PMID: 32286192; PMCID: PMC7263289.
6.de Leeuw den Bouter ML, Ippolito G, O'Reilly TPA, Remis RF, van Gijzen MB, Webb AG. Deep learning-based single image super-resolution for low-field MR brain images. Sci Rep. 2022 Apr 16;12(1):6362. doi: 10.1038/s41598-022-10298-6. PMID: 35430586; PMCID: PMC9013376.
7. Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. "Deep image prior." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
8. Jiaming Liu,Yu Sun,Xiaojian Xu,and Ulugbek S Kamilov,“Image restoration using total variation regularized deep image prior,”in ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP).IEEE,2019,pp.7715–7719.
9. Geman D, Yang C. Nonlinear image recovery with half-quadratic regularization. IEEE Trans Image Process. 1995;4(7):932-46. doi: 10.1109/83.392335. PMID: 18290044.
10. Zhang, K., Li, Y., Liang, J. et al. Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis. Mach. Intell. Res. (2023).
11. Roszkowiak L, Korzynska A, Zak J, Pijanowska D, Swiderska-Chadaj Z, Markiewicz T. Survey: interpolation methods for whole slide image processing. J Microsc. 2017 Feb;265(2):148-158. doi: 10.1111/jmi.12477. Epub 2016 Sep 29. PMID: 27681946.
Figures
Figure 3. The quantitative analysis results of performance of the Bicubic interpolation, DIP with TV and the proposed method for super-resolution reconstruction of 5T MR images. (The best values are highlighted in bold.)
