1198

Unpaired Image-to-Image Translation of ULF-MRI using Vision Transformers to Advance Volumetric Analyses
Peter Hsu1,2, Elisa Marchetto1,3, Samantha Sanger1, Hersh Chandarana1,3, Jakob Asslaender1,3, Daniel Sodickson1,2,3, Patricia Johnson1,2,3, and Jelle Veraart1,3
1Bernard and Irene Schwartz Center for Biomedical Imaging, Department of Radiology, New York University Grossman School of Medicine, New York, NY, United States, 2Vilcek Institute of Graduate Biomedical Sciences, New York University Grossman School of Medicine, New York, NY, United States, 3Center for Advanced Imaging Innovation and Research (CAI2R), Department of Radiology, New York University Grossman School of Medicine, New York, NY, United States

Synopsis

Keywords: Analysis/Processing, Low-Field MRI, ULF MRI, Ultra-Low-Field MRI, Deep Learning, Unpaired Image Translation, Brain Segmentation, Vision Transformers, CycleGAN

Motivation: The image quality of ultra-low-field MRI impacts the reliability of volumetric analysis in the brain. Existing techniques that address this issue learn from synthetically generated images, leading to a domain shift problem when presented with real images.

Goal(s): Development of a deep learning method trained with real ULF and HF images to robustly generate an image that can be segmented with routine software tools.

Approach: We introduce a CycleGAN framework with Residual Vision Transformers to improve super-resolved images compared to existing methods.

Results: The accuracy of volumetric estimations improves using our method compared to others based on clinical correlations and test-retest reliability metrics.

Impact: Our new image enhancement method should allow reliable volumetric evaluation using ULF-MRI. This will allow investigators in regions with access to ULF systems to monitor brain health in a way that was previously unattainable.

Introduction

Ultra low-field (ULF) MRI1,2 (<100mT) provides an innovative pathway to more accessible neuroimaging by mitigating various logistical, financial, and safety considerations. Unfortunately, ULF scans have (a) significantly lower SNR and spatial resolution per unit time, and (b) significantly different relaxation contrast than traditional high-field (HF) clinical scans. These factors pose a significant challenge in the use of brain segmentation tools for automated volumetric analysis3-6, which have been developed and validated for HF images. Our goal is to enable reliable brain segmentation of ULF-MRI.

We can leverage recent deep learning advances for super-resolution and contrast enhancement to improve ULF-MRI image quality, enabling use of existing segmentation protocols. Although advancements in this field7,8, such as SynthSR9,10, have shown promising results, these methods generally rely on synthetic training images which might limit model performance on real ULF images because of domain shift11. Here, we propose a deep learning method trained with experimental ULF-MRI and HF MP-RAGE scans to create images that return more accurate and consistent segmentations (Figure 1).

Methods

MRI Data (Figure 2):
ULF images from 39 subjects (20 female) were acquired with a Hyperfine1,2 Swoop scanner (64mT). Subjects were healthy controls aged 23-72 years. Image acquisition used 3D FSE with T1-weighted (T1w) and T2-weighted (T2w) contrast in axial, coronal, and sagittal directions. 25 of the 39 subjects had previously received HF MP-RAGE scans. These scans were retrospectively analyzed, leading to minor variations in FoV and spatial resolution.

Model (Figure 1):
Our method follows a CycleGAN12 framework with modified Residual Vision Transformers (ResViT13). This architecture combines the strengths of ResNet14 convolutional layers and transformer15 attention mechanisms. This allows local feature processing with convolutions and global context integration using multi-head self-attention. Contextual processing is furthered by residual connections which help preserve higher-order information throughout network layers. The discriminators are 3 layer convolutional networks.

Datasets:
  1. Pre-Training: Unpaired image-to-image translation for 400 epochs
    a. M4Raw16: 468 low-field (0.3T) T1w & T2w images
    b. HCP17: 1100 3T MP-RAGE & T2w images

  2. Training: Unpaired image-to-image translation for 1200 epochs
    a. NYU: 28 ULF T1w & T2w images
    b. HCP17: 1100 3T MP-RAGE images

  3. Testing:
    a. NYU: 22 ULF images (test-retest for 11 subjects)
    b. NYU: 8 paired HF MP-RAGE images.

Pre-processing:
Images were registered to the MNI18 template using rigid transformation19 and resampled20 to 1mm isotropic voxel size with 256x256x256 dimensions.

Training Strategy:
We trained a model using publicly-available data across field strengths (M4Raw16, a low-field (0.3T) MRI dataset, and HCP17) enabling transfer learning from a similar problem to models trained on our smaller ULF dataset. We further explored the impact of multi-directional ULF information by separately training models with [axial T1w, axial T2w] or [axial T1w, coronal T2w] inputs and HF MP-RAGE targets. Patch-based (128x128x128) learning was implemented to reduce computational constraints.

Segmentation & Statistics:
Brain segmentation of gray matter (GM), white matter (WM), hippocampus (HC), and lateral ventricles (LV) was performed with FastSurfer21,22. Estimation of intracranial volume (iCV) utilized masks acquired through HD-BET23. Pearson correlation coefficient (PCC) was computed using synthetic and subject-matched HF MP-RAGEs. Test-retest variability (TRV24) was computed from synthetic MP-RAGEs of subjects with repeat scans. Comparisons were made with the publicly available state-of-the-art, SynthSR.

Results

Figure 3 showcases a representative example of an ULF image, its model output, and FastSurfer segmentation. Figures 4 and 5 compare PCC and TRV, respectively, for volumetric estimates made using our method and SynthSR.

Discussion & Conclusion

Our results demonstrate that our method improves volumetric estimation for ULF images of healthy brains using automated techniques. First, we observe a higher correlation between subject-matched ULF-MRI and clinical data compared to SynthSR for most volumetric estimates. Second, the test-retest reproducibility increased as shown by reduced TRV across most volumetric estimates using our method compared with SynthSR. Interestingly, correlation performance using axial-coronal input mostly outperformed axial-axial input, but with loss of reproducibility across most estimates. We hypothesize this improvement in accuracy stems from the complementary “k-space” coverage provided by an additional acquisition direction. The lower TRV of axial-coronal inputs may result from more challenging motion correction between low resolution images acquired with orthogonal FoV.

There are some limitations. First, our testing dataset is small as many of our images were dedicated to model training. Second, our model architecture is computationally expensive, necessitating a patch-based approach which potentially limits the long-range contextual information the model can learn. We believe improvements in estimation could be achieved through a segmentation-based loss function as specified by SynthSR9,10. Future directions include expanding our dataset, leveraging synthetically generated ULF images for training, and exploring age-related volumetric changes utilizing our method.

Acknowledgements

This work has grant support through NIH P41 EB017183. This work was performed under the rubric of the Center for Advanced Imaging Innovation and Research (CAI2R, www.cai2r.net), an NIBIB National Center for Biomedical Imaging and Bioengineering.

References

1. Chetcuti, K., Chilingulo, C., Goyal, M. S., et al. Implementation of a Low-Field Portable MRI Scanner in a Resource-Constrained Environment: Our Experience in Malawi. AJNR: American Journal of Neuroradiology, 2022;43(5):670–674.

2. Mazurek, M. H., Cahn, B. A., Yuen, M. M., et al. Portable, bedside, low-field magnetic resonance imaging for evaluation of intracerebral hemorrhage. Nature Communications, 2021;12(1), Article 1.

3. Bermel, R. A., & Bakshi, R. The measurement and clinical relevance of brain atrophy in multiple sclerosis. The Lancet Neurology, 2006;5(2):158–170.

4. Fox, N. C., & Schott, J. M. Imaging cerebral atrophy: Normal aging to Alzheimer’s disease. The Lancet, 2004;363(9406):392–394.

5. Swanson, C. J., Zhang, Y., Dhadda, S., et al. A randomized, double-blind, phase 2b proof-of-concept clinical trial in early Alzheimer’s disease with lecanemab, an anti-Aβ protofibril antibody. Alzheimer’s Research & Therapy, 2021;13(1):80.

6. van Dyck, C. H., Swanson, C. J., Aisen, P., et al. Lecanemab in Early Alzheimer’s Disease. The New England Journal of Medicine, 2023;388(1):9–21.

7. Lau, V., Xiao, L., Zhao, Y., et al. Pushing the limits of low-cost ultra-low-field MRI by dual-acquisition deep learning 3D superresolution. Magnetic Resonance in Medicine, 2023;90(2):400–416.

8. Lin, H., Figini, M., D’Arco, F., et al. Low-field magnetic resonance image enhancement via stochastic image quality transfer. Medical Image Analysis, 2023;87:102807.

9. Iglesias, J. E., Schleicher, R., Laguna, S., et al. (Accurate super-resolution low-field brain MRI. arXiv. 2022;2202.03564.

10. Iglesias, J. E., Billot, B., Balbastre, Y., et al. SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry. Science Advances, 2023;9(5):eadd3607.

11. Kimberly, W. T., Sorby-Adams, A. J., Webb, A. G., et al. Brain imaging with portable low-field MRI. Nature Reviews Bioengineering, 2023;1(9):Article 9.

12. Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv. 2020;1703.10593.

13. Dalmaz, O., Yurt, M., & Çukur, T. ResViT: Residual vision transformers for multi-modal medical image synthesis. IEEE Transactions on Medical Imaging, 2022;41(10):2598–2614.

14. He, K., Zhang, X., Ren, S., & Sun, J. Deep Residual Learning for Image Recognition. arXiv. 2015;1512.03385

15. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv. 2021;2010.11929.

16. Lyu, M., Mei, L., Huang, S. et al. M4Raw: A multi-contrast, multi-repetition, multi-channel MRI k-space dataset for low-field MRI research. Scientific Data, 2023;10(1):Article 1.

17. Van Essen D. C., Smith S. M, Barch D. M., et al. The WU-Minn Human Connectome Project: An overview. NeuroImage 2013;80(2013):62-79.

18. Mazziotta, J. C., Toga, A. W., Evans, A., et al. A Probabilistic Atlas of the Human Brain: Theory and Rationale for Its Development: The International Consortium for Brain Mapping (ICBM). NeuroImage, 1995;2(2, Part A):89–101.

19. Avants, B. B., Tustison, N. J., Song, G.,et al. A reproducible evaluation of ANTs similarity metric performance in brain image registration. NeuroImage, 2011;54(3):2033–2044.

20. Tournier, J.-D., Smith R. E., Raffelt D., et al. MRtrix3: A fast, flexible and open software framework for medical image processing and visualization. NeuroImage 2019;202(2019):116–37.

21. Henschel, L., Conjeti, S., Estrada, S., et al. FastSurfer—A fast and accurate deep learning based neuroimaging pipeline. NeuroImage, 2020;219:117012.

22. van Nederpelt, D. R., Amiri, H., Brouwer, I., Noteboom, S., et al. Reliability of brain atrophy measurements in multiple sclerosis using MRI: An assessment of six freely available software packages for cross-sectional analyses. Neuroradiology, 2023;65(10):1459–1472.

23. Isensee, F., Schell, M., Tursunova, I., et al. Automated brain extraction of multi-sequence MRI using artificial neural networks. Human Brain Mapping, 2019;40(17):4952–4964.

24. McGraw, K. O., & Wong, S. P. Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1996;1(1):30–46.

Figures

Figure 1. Outline of our proposed framework using axial ULF T1w and axial ULF T2w images as an example. Model A takes two ULF images as input and produces a HF MP-RAGE image as output. Model B takes an MP-RAGE image as input and produces two ULF images as output. Discriminator A (DA) predicts real vs fake ULF images and Discriminator B (DB) predicts real vs fake MP-RAGE images. Cycle consistent loss (LCycle) is used to regularize model output.

Figure 2. A. Acquisition parameters of ULF T1w and T2w images including Echo Time (TE), Repetition Time (TR), Inversion Time (TI), Echo Train Length (ETL), and Scan Time per acquisition direction (axial, coronal, and sagittal). B. Acquisition information for ULF T1w, ULF T2w, and MP-RAGE images.

Figure 3. Example input ULF images [axial T1w, coronal T2w] and the output MP-RAGE from our proposed method. The output MP-RAGE is then compatible with FastSurfer which provides ROI labels for volumetric analysis. The labels we use for statistical analysis are shown as “Evaluated Labels” where gray matter is red, white matter is green, hippocampus is blue, and lateral ventricles are yellow.

Figure 4. Correlation analysis between volumetric estimations from Synthetic MP-RAGE images and paired clinical MP-RAGE images using the first scan time for ULF data (N=7 excluding outliers). Results are shown for SynthSR (top row; black) and our proposed method (middle row; orange, bottom row; purple) for axial-axial and axial-coronal input, respectively. Gray matter is denoted with red, white matter with green, hippocampus with blue, and lateral ventricles with yellow. Highest Pearson correlation coefficients are highlighted with bold.

Figure 5. Table comparing Test-Retest Variability (TRV) for SynthSR and our proposed method (N=10 after outlier exclusion). Intracranial Volume (iCV) was calculated using brain masks from HD-BET. Estimates for Gray Matter (GM), White Matter (WM), Hippocampus (HC), and Lateral Ventricles (LV) were provided by FastSurfer segmentation. The highest performing results (lowest variability) are highlighted with bold.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
1198
DOI: https://doi.org/10.58530/2024/1198