2421
Harmonization of multi-site T1 data using CycleGAN with segmentation loss (CycleGANs)1Brigham and Women's Hospital and Harvard Medical School, Boston, MA, United States, 2Brigham and Women's Hospital, Boston, MA, United States, 3Ahmanson-Lovelace Brain Mapping Center, UCLA, Los Angeles, CA, United States, 4University of Utah, Salt Lake City, UT, United States, 5U.S. Air Force School of Aerospace Medicine, San Antonio, TX, United States
Synopsis
This study aims to tackle the structural MRI (T1) data harmonization problem by presenting a novel multi-site T1 data harmonization, which uses the CycleGAN network with segmentation loss (CycleGANs). CycleGANs aims to learn an efficient mapping of T1 data across scanners from the same set of subjects while simultaneously learning the mapping of free surfer parcellations. We demonstrated the efficacy of the method with the Dice overlap scores between FreeSurfer parcellations across two datasets before and after harmonization.
Introduction
Structural brain changes have been well-studied in neuroimaging studies to characterize the impact of aging, neurological disorders, as well as brain development. A large number of neuroimaging studies have reported findings from a single-site dataset with small sample sizes and homogeneous demographics, thereby leading to poor generalizability and reproducibility. To tackle the reproducibility problem, in recent years there is an increasing trend towards data sharing between neuroimaging research communities, i.e., collaborative efforts have collected large-scale, comprehensive, and diverse multi-site neuroimaging datasets. It is, however, not advisable to naively combine neuroimaging data due to the significant scanner- or acquisition-related measurement effects on the data. “Harmonization” is a way to mitigate the measurement differences attributed to the scanner-, protocol-, or other site-related differences. Thus, the harmonization of multi-site structural MRI (e.g., T1) datasets can increase the statistical power of multi-site neuroimaging studies and enable comparative studies pertaining to several brain disorders. Given the importance of the problem, initially, several methods have been published which are based on removing statistical differences from regions of interest (ROI) volumes (Pomponio et al. 2020; Fortin et al. 2018). Recently, non-linear deep learning methods have been proposed which can be used to harmonize neuroimaging data across sites (Ning et al. 2020). In this work, we propose multi-site structural MRI (T1) data harmonization using CycleGAN (Zhang, Yang, and Zheng 2018; Jiang et al. 2018) with segmentation loss (CycleGANs) (Zhang, Yang, and Zheng 2018; Jiang et al. 2018) which aims to learn an efficient mapping of the structural data from the same set of subjects across sites, while also learning the mapping of FreeSurfer parcellations.Methods
Dataset and preprocessing.Twenty-five male subjects were scanned using 3T Siemens Trio and Verio scanners with an isotropic resolution of 0.8 mm3. Scans were completed on two scanners in a very quick succession, i.e., there was only an hour difference between scans. N4 bias field correction and skull stripping were run on the T1 data using in-house software as part of the Luigi pipeline (Billah et al. 2020). FreeSurfer v. 7.1.0 was run on T1 data to create the white matter and gray matter anatomical labels. Next, mri_label2vol (part of the FreeSurfer pipeline) was run to create a volume of anatomical labels. T1 data from two scanners were affinely registered using antsRegistration (Avants et al. 2011). The affine transformations were applied to FreeSurfer maps (Figure 1).
CycleGAN with segmentation loss (CycleGANs) for harmonization of two T1 datasets.
CycleGAN has been widely used for unpaired image-to-image translation in real-world images (Zhu et al. 2017). Although CycleGAN can be potentially used for harmonization of multi-site T1 data, there are still some restrictions that exist. The cycle consistency and adversarial loss have been shown to only constrain the model to learn a global mapping that matches the marginal distribution but not the conditional distribution pertaining to the tissue variabilities. Thus the standard CycleGAN network proposed by (Zhu et al. 2017) cannot be directly used for robust multi-site T1 data harmonization. We, therefore, construct a CycleGAN network (Zhang, Yang, and Zheng 2018; Jiang et al. 2018) with a new loss function for the harmonization of multi-site T1 data that maintains the tissue variabilities while learning mapping of T1 data across scanners together with the mapping of FreeSurfer label maps (Figure 2). We set the aligned T1 data of one scanner with its corresponding FreeSurfer label volume as input to our network. The aligned T1 data from the other scanner was set as output. In addition to adversarial and cycle-consistency loss, we included “segmentation loss” as a multi-class cross-entropy loss to the CycleGAN network, which can regularize the generators: $$$L_{segm}(S_1,S_2,G_1,G_2)=-Y_1 log(S_1(G_1(x \in 2)))+-Y_2 log(S_2(G_2(x \in 1)))$$$.
We supervised the segmentation loss function by the FreeSurfer labels. $$$Y$$$ denotes the ground-truth FreeSurfer labels, G is the generator, 1 and 2 are referred to as first and second datasets respectively. Our method also introduces two auxiliary mappings, $$$S_1:1→Y$$$ and $$$S_2: 2→Y$$$, to constrain the labels. They map the harmonized data from respective domain generators into a shared space $$$Y$$$ (i.e., ground truth FreeSurfer label map), $$$Y_1, Y_2 \in Y$$$ are the ground truth FreeSurfer label maps for dataset 1 and dataset 2.
Results
We used 5-fold cross-validation in our experiments. We computed the Dice score between the FreeSurfer labels before and after harmonization to demonstrate the performance of the harmonization. While the average Dice score was 91% between the T1 data from two scanners prior to harmonization, the Dice score increased to 98% after harmonization in the subcortical regions.Discussion and Conclusion
In this study, we presented a multi-site T1 data harmonization approach, which uses CycleGAN with segmentation loss (CycleGANs). While our network learns the mapping of T1 data from one scanner to another, it also corrects the FreeSurfer label maps. We demonstrated the efficacy of the method in the subcortical regions with Dice overlap scores between FreeSurfer parcellations across two datasets before and after harmonization. We are still testing the performance of CycleGANs in other regions. We note that this work is preliminary and extensive validation will be done in the future to further understand the power and limitation of this technique.Acknowledgements
The authors have been supported by the following grants: NIH R01 MH119222 (Rathi). The project also acknowledges that the research has partly been supported by the BWH Program for Interdisciplinary Neuroscience, through a gift from Lawrence and Tiina Rand (Cetin-Karayumak).References
Avants, Brian B., Nicholas J. Tustison, Gang Song, Philip A. Cook, Arno Klein, and James C. Gee. 2011. “A Reproducible Evaluation of ANTs Similarity Metric Performance in Brain Image Registration.” NeuroImage. https://doi.org/10.1016/j.neuroimage.2010.09.025.
Fortin, Jean-Philippe, Nicholas Cullen, Yvette I. Sheline, Warren D. Taylor, Irem Aselcioglu, Philip A. Cook, Phil Adams, et al. 2018. “Harmonization of Cortical Thickness Measurements across Scanners and Sites.” NeuroImage 167 (February): 104–20.
Jiang, Jue, Yu-Chi Hu, Neelam Tyagi, Pengpeng Zhang, Andreas Rimner, Gig S. Mageras, Joseph O. Deasy, and Harini Veeraraghavan. 2018. “Tumor-Aware, Adversarial Domain Adaptation from CT to MRI for Lung Cancer Segmentation.” Medical Image Computing and Computer-Assisted Intervention: MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention 11071 (September): 777–85.
Ning, Lipeng, Elisenda Bonet-Carne, Francesco Grussu, Farshid Sepehrband, Enrico Kaden, Jelle Veraart, Stefano B. Blumberg, et al. 2020. “Cross-Scanner and Cross-Protocol Multi-Shell Diffusion MRI Data Harmonization: Algorithms and Results.” NeuroImage. https://doi.org/10.1016/j.neuroimage.2020.117128.
Pomponio, Raymond, Guray Erus, Mohamad Habes, Jimit Doshi, Dhivya Srinivasan, Elizabeth Mamourian, Vishnu Bashyam, et al. 2020. “Harmonization of Large MRI Datasets for the Analysis of Brain Imaging Patterns throughout the Lifespan.” NeuroImage 208 (March): 116450.
Zhang, Zizhao, Lin Yang, and Yefeng Zheng. 2018. “Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/cvpr.2018.00963.
Zhu, Jun-Yan, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.” 2017 IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv.2017.244.
Billah, Tashrif; Bouix, Sylvain, A Luigi workflow joining individual modules of an MRI processing pipeline, https://github.com/pnlbwh/luigi-pnlpipe, 2020, DOI: 10.5281/zenodo.3666802
Figures
