2810
Accelerated Multi-Contrast Parallel Imaging Reconstruction with Implicit Neural Representation1Industrial Engineering and Management Systems, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran (Islamic Republic of), 2Department of Computer Engineering, Hongik University, Seoul, Korea, Republic of, 3Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, United States, 4Radiology, Harvard Medical School, Boston, MA, United States, 5Harvard/MIT Health Sciences and Technology, Cambridge, MA, United States
Synopsis
Keywords: Quantitative Imaging, Image Reconstruction
Motivation: Multi-contrast MR scans provide rich information for clinical diagnosis and research studies. However, long scan time is a limitation.
Goal(s): an implicit neural representation is proposed for accelerated multi-contrast parallel imaging reconstruction. The proposed scan-specific method obviates the need for fully sampled priors.
Approach: The spatial and temporal feature maps of an initial reconstruction are implicitly represented into the weights of a prior network. It exploits the physics-based parallel imaging forward model of sparsely sampled measurements.
Results: The proposed method outperforms the evaluated parallel imaging techniques at acceleration rates as high as R=16 in both reconstructed echo images and parameter mapping.
Impact: The proposed scan-specific method reconstructs multi-contrast images by implicit representation of the feature maps learned from interim reconstructions and exploitation of parallel imaging forward model in the training stage. It outperforms evaluated parallel imaging techniques.
Introduction
Multi-contrast MR scans can provide rich information for clinical diagnosis and research studies [1]. However, the long scan time is a limitation. To address this problem, many accelerated multi-contrast reconstruction techniques have been developed [2-6] including dictionary learning [7], CNN [8-10], GAN [9,11,12], and unrolled networks [5,6,13,14]. They mostly need prior fully sampled data for their learning paradigm [15].Implicit neural representation emerged as a powerful paradigm in image reconstruction [16,17]. NeRP [18] framework integrates implicit neural networks for reconstructing undersampled medical images. NeRP needs fully-sampled priors for training and does not incorporate parallel imaging forward model.
In this work, we extend NeRP to be capable of multi-contrast reconstruction of undersampled multi-echo gradient-echo (MEGRE) data without the need for fully sampled priors and name it MCNeRP. The performance of MCNeRP is compared with multi-contrast low rank reconstruction methods in both echo images and parameter mapping. Results show that MCNeRP outperforms existing techniques in both settings.
code/data: https://anonymous.4open.science/r/MCNeRP-722F
Theory and Methods
Fig. 1. shows the main structure of MCNeRP. In the first step (part a) SENSE [19] reconstructs the selected undersampled MEGRE measurements to serve as the prior contrasts. A multi-layer perceptron (MLP) learns to map the spatial coordinates and echo times (TEs) of the reconstructed prior contrasts into the parameters of the network. Before entering the dense layers, a Fourier feature layer [20] embeds the spatial and temporal coordinates separately as $$$γspatial[x,y]$$$ and $$$γtemporal[TE]$$$ with [21]:$$γspatial[x,y]=[cos(2πB[x,y]),sin(2πB[x,y])]T , γtemporal[TE]=[cos(2πB[TE]),sin(2πB[TE])]T$$
where B is the coefficient for Fourier feature transformation and its components are randomly sampled for Gaussian distribution.
In the second step (part b), the reconstructed echo image passes through a forward model consisting of Fourier transform, coil sensitivities, and undersampling mask to generate corresponding k-space data. This data matches the sparsely acquired k-space of the selected echo to form the new loss function for the network. Optimizing the new loss function, the parameters of the network are tuned with the information in the physics of the acquired multi-channel data.
In the inferencing step (part c), the trained network reconstructs the echo image related to the input grid and the echo time of the contrast.
Dataset: A fully sampled in-vivo 3D whole brain dataset was acquired using a Siemens 3T Prisma scanner with a 32-channel head-coil, where one slice was selected for the experiments. The slice thickness is 2 mm. The dataset consists of six different TEs with FA=4o. Coil sensitivity maps were generated from the central 24x24 ACS using ESPIRiT [22].
Results
In the first experiment (Fig. 2) we compared the performance of MCNeRP to standard SENSE and Locally Low Rank regularization [2,23] and LORAKS [4,24]. As shown, the proposed method outperforms other evaluated techniques in all six contrasts in terms of NRMSE.The next experiment (Fig. 3) compares parameter mapping performance of MCNeRP and other techniques. Here, multi-contrast reconstructions are followed by a parameter fitting optimization for T2*, frequency, and proton density mapping. As shown, MCNeRP offers better reconstructions in all T2*, frequency, and proton density maps.
Finally, we compare the performance of the original NeRP and MCNeRP. A fully sampled contrast was selected as the prior image for the original NeRP and other sub-sampled contrasts were inferred from the trained network. Fig. 4 shows that the proposed method significantly improved the performance of the NeRP in multi-contrast reconstruction.
In MCNeRP, the prior step takes ~30 minutes and the inferencing step takes ~7 minutes for each contrast with the V100 GPU available at Google Collaboratory.
Discussion and Conclusion
NeRP is extended for accelerated scan-specific multi-contrast parallel imaging reconstruction. The reconstruction obviates the need for fully sampled priors. The proposed network outperforms multi-contrast low rank parallel imaging methods in both image reconstruction and MR parameter mapping. The evaluated average error is below 7% in acceleration rate as high as R=16. The new input feature map, separating the Fourier feature embedding for spatial and temporal inputs, employing complex images, and using the information of the physics of parallel imaging allow the proposed network to outperform the original NeRP in multi-contrast reconstruction.Acknowledgements
This work was supported by research grants NIH R01 EB028797, U01 EB025162, P41 EB030006, U01 EB026996, R03 EB031175, R01 EB032378, UG3 EB034875
NVidia Corporation for computing support and National Research Foundation of Korea (NRF) Grant funded by the Korea government (MSIT) (No. 2022R1F1A1074786).
References
[1] Huang J, Chen C, Axel L. “Fast multi-contrast MRI reconstruction.” Magn Reson Imaging. 2014;32(10):1344-1352.
[2] Zhang T, Pauly JM, Levesque IR. “Accelerating parameter mapping with a locally low rank constraint.” Magn Reson Med. 2015;73(2):655-661.
[3] Kim TH, Garg P, Haldar JP. “LORAKI: Autocalibrated recurrent neural networks for autoregressive MRI reconstruction in k-space”. arXiv Prepr arXiv190409390. Published online 2019.
[4] Haldar JP. “Low-rank modeling of local k-space neighborhoods (LORAKS) for constrained MRI.” IEEE Trans Med Imaging. 2013;33(3):668-681.
[5] Polak, D., Cauley, S., Bilgic, B., Gong, E., Bachert, P., Adalsteinsson, E., & Setsompop, K. “Joint multi‐contrast variational network reconstruction (jVN) with application to rapid 2D and 3D imaging.” Magnetic resonance in medicine, 2020;84(3):1456-1469.
[6] Dar SUH, Yurt M, Shahdloo M, Ildız ME, Tınaz B, Çukur T. “Prior-guided image reconstruction for accelerated multi-contrast MRI via generative adversarial networks.” IEEE J Sel Top Signal Process, 2020;14:1072–87.
[7] Song P, Weizman L, Mota JFC, Eldar YC, Rodrigues MRD. “Coupled dictionary learning for multi-contrast MRI reconstruction.” IEEE Trans Med Imaging 2020;39(3):621–33.
[8] Zeng, K., Zheng, H., Cai, C., Yang, Y., Zhang, K., & Chen, Z. “Simultaneous single-and multi-contrast super-resolution for brain MRI images based on a convolutional neural network.”, Computers in biology and medicine, 2018;99:133-141.
[9] Do, W. J., Seo, S., Han, Y., Ye, J. C., Choi, S. H., & Park, S. H. “Reconstruction of multicontrast MR images through deep learning.” Magn Reson Med. 2020;47(3):983-997.
[10] Srinivasan, P., Kaur, P., Nigam, A., & Bhavsar, A. “Semantic Features Aided Multi-Scale Reconstruction of Inter-Modality Magnetic Resonance Images.”, In 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS). IEEE. 110-113.
[11] Kim, K. H., Do, W. J., & Park, S. H. “Improving resolution of MR images with an adversarial network incorporating images with different contrast.” Medical physics, 2018;45(7):3120-3131.
[12] Lyu, Q., Shan, H., Steber, C., Helis, C., Whitlow, C., Chan, M., & Wang, G. “Multi-contrast super-resolution MRI through a progressive network.”, IEEE transactions on medical imaging, 2020;39(9):2738-2749.
[13] Sun, L., Fan, Z., Fu, X., Huang, Y., Ding, X., & Paisley, J. “A deep information sharing network for multi-contrast compressed sensing MRI reconstruction.” IEEE Transactions on Image Processing, 2019;28(12):6141-6153.
[14] Liu X, Wang J, Sun H, Chandra SS, Crozier S, Liu F. “On the regularization of feature fusion and mapping for fast MR multi-contrast imaging via iterative networks.” Magn Reson Imaging, 2021;77:159–68.
[15] Chandra, S. S., Bran Lorenzana, M., Liu, X., Liu, S., Bollmann, S., & Crozier, S. “Deep learning in magnetic resonance image reconstruction.” Journal of Medical Imaging and Radiation Oncology, 2021;65(5):564-577.
[16] Molaei, A., Aminimehr, A., Tavakoli, A., Kazerouni, A., Azad, B., Azad, R., & Merhof, D. “Implicit neural representation in medical imaging: A comparative survey.” In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE;2023:2381-2391.
[17] McGinnis, J., Shit, S., Li, H. B., Sideri-Lampretsa, V., Graf, R., Dannecker, M., ... & Wiestler, B. “Single-subject Multi-contrast MRI Super-resolution via Implicit Neural Representations.” arXiv preprint arXiv:2303.15065. Published online 2023.
[18] Shen L, Pauly J, Xing L. “NeRP: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction.” IEEE Trans Neural Netw Learn Syst. 2022;33(6):2439-2454.
[19] Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. “SENSE: sensitivity encoding for fast MRI.”, Magn Reson Med An Off J Int Soc Magn Reson Med. 1999;42(5):952-962.
[20] M. Tancik et al., “Fourier features let networks learn high frequency functions in low dimensional domains,” Advances in Neural Information Processing Systems, 2020;33:10515-10526.
[21] Kunz, Johannes F., Stefan Ruschke, and Reinhard Heckel. “Implicit Neural Networks with Fourier-Feature Inputs for Free-breathing Cardiac MRI Reconstruction,” arXiv preprint arXiv:2305.06822, Published online 2023.
[22] Uecker M, Lai P, Murphy MJ, et al. “ESPIRiT—an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA”. Magn Reson Med. 2014;71(3):990-1001.
[23] Tamir JI, Uecker M, Chen W, et al. “T2 shuffling: sharp, multicontrast, volumetric fast spin‐echo imaging.”, Magn Reson Med. 2017;77(1):180-195.
[24] Kim TH, Setsompop K, Haldar JP. “LORAKS makes better SENSE: phase‐constrained partial fourier SENSE reconstruction without phase calibration.”, Magn Reson Med. 2017;77(3):1021-1035.
Figures
Fig.1. Framework of MCNeRP. a) spatial and temporal feature maps go to the Fourier mapping layer to separately be transformed into Fourier values. The network embeds the inputs of the prior echoes reconstructed by SENSE into the prior parameters θprior, b) the pre-trained network tunes its parameters by optimization of the loss function from matching the sparse acquired k-space of the echo and the k-space related to the network output after passing through the forward model, c) the tuned network infers the echo image taking its spatial and temporal feature map as input.
Fig. 2. Performance of the proposed MCNeRP vs standard (SENSE) and low rank (LLR and LORAKS) parallel reconstruction techniques. A uniform sampling pattern with acceleration rate of R=16x(4x4) is applied. The prior echoes for MCNeRP are TE=3.6, 13.6, and 23.6(ms) initialized with SENSE. Normalized RMSE (NRMSE) with a brain mask is used as the evaluation measurement. MCNeRP outperforms evaluated parallel reconstruction techniques in all reconstructed echo images.
Fig. 3. T2*, frequency and proton density parameter mapping performance of the proposed MCNeRP vs standard (SENSE) and low rank (LLR and LORAKS) parallel reconstruction techniques. A uniform sampling pattern with acceleration rate of R=16x(4x4) is applied. To control the effect of very large T2* values (due to extremely R2* values), we saturated them within the range of 0-100 ms for the purpose of NRMSE computation. MCNeRP outperforms evaluated parallel reconstruction techniques in terms of NRMSE values and visual maps in all T2*, frequency, and proton density maps.
Fig. 4. Comparison between the original NeRP and proposed MCNeRP in multi-contrast reconstruction. NeRP takes a fully-sampled reconstructed echo image (TE=13.6 (ms)) as the prior and infers other contrasts using a tuned network at R=16x uniform sampling. MCNeRP uses sub-sampled initially reconstructed echo images of TE=3.6, 13.6, and 23.6 (ms) as priors and infers all contrasts in the same acceleration rate and sampling pattern. MCNeRP offers significant improvement in all echo images in comparison to its original version.