Synopsis

Keywords: Quantitative Imaging, Relaxometry, Reproducibility, challenge

A collaborative reproducibility challenge was launched to explore if an imaging protocol independently-implemented at multiple centers can reliably measure T₁ using inversion recovery in a standardized quantitative MRI phantom (ISMRM/NIST). A total of 19 submissions were accepted, totalling 41 phantom T₁ mapping datasets. Errors relative to the temperature-corrected reference T₁ values were under 10% for the range of values expected in the human brain in vivo. All submitted phantom data, code, pipelines, and scripts were shared on open platforms.

Introduction

Quantitative MRI has a reproducibility problem. Even the most fundamental MR parameters such as T₁, which has been mapped since the dawn of MRI (Pykett and Mansfield, 1978), still cannot be reliably measured in vivo across quantitative techniques or site-specific implementations (Stikov et al., 2015). Inversion recovery is the gold standard for T₁ mapping (Barral et al., 2010), and although it is impractically slow for whole-organ imaging, it can be useful to evaluate the variability of T₁ values across site-specific implementations. The 2020 ISMRM joint study group (Reproducible Research and Quantitative MR) reproducibility challenge posed the following question: will an imaging protocol independently-implemented at multiple centers reliably measure what is considered one of the fundamental MR parameters (T₁) using the most robust technique (inversion recovery) in a standardized phantom (ISMRM/NIST system phantom)?

Methods

Submissions required a set of inversion recovery acquisitions of the ISMRM/NIST system phantom (Stupic et al., 2021) (Premium System Phantom, CaliberMRI, Boulder, Colorado), which has a T₁ plate with 14 fiduciary spheres (see Figure 1 for reference values). Challenge participants were instructed to acquire data using the spin-echo inversion recovery protocol for T₁ mapping as reported in (Barral et al., 2010), the details of which are listed in Figure 2. Note that the short repetition time (TR ~ T₁) is possible because the T₁ fitting model used in (Barral et al., 2010) does not make the conventional long TR assumption. Also, note that the four inversion times (TIs) from this publication were optimized for mapping brain T₁ values. As the focus of this challenge was on reproducibility, the challenge design put emphasis on reproducible research practices by sharing the submitted data, code, pipelines, and scripts to reproduce figures. Data submissions for the challenge were handled through the GitHub issue tracking framework in a repository dedicated to this purpose (https://github.com/rrsg2020/data_submission). Each submission was reviewed by the challenge organizers to ensure guidelines were followed, and then datasets and YAML files containing additional details of the submissions were uploaded to OSF.io (https://osf.io/ywc9g/). T₁ maps were generated using a pipeline (https://github.com/rrsg2020/t1_fitting_pipeline) using Barral’s original open-source code built into qMRLab (Karakuzu et al., 2020). ROI labels of each of the 14 T₁ spheres were created using a numerical mask template created in MATLAB and shared by NIST. The diameters were reduced to be 60% of the expected sphere diameter in an effort to avoid edge effects in the ROIs. A pipeline to register the ROI labels to each submitted T₁ map dataset was written using ANTs (Avants, Tustison and Song, 2009). The reference T₁ values for the phantom were adjusted using the measured internal temperature of the phantoms as reported by the participants, prior to data plotting and analyses.

Results

A total of 41 phantom T₁ datasets were submitted and approved, collected at 20 imaging sites. All submitted datasets were acquired at 3 T except for one submission at 0.35 T, and they were all acquired on MRI systems from three major clinical vendors (GE, Philips, Siemens). Minor protocol deviations were reported across sites; in particular, the maximum TI wasn't always achievable (eg., 2400 instead of 2500 ms), and the bandwidth varied substantially (120 to 400 Hz/px). Figure 3 shows the variability in orientation, field-of-view, noise, distortions, and artefacts due to the different site-specific implementations by participants all of whom followed the same inversion recovery T₁ mapping protocol (Figure 2). Figure 4 shows the mean T₁ estimates relative to the reference T₁ values for each sphere on a single representative submission (a,b,c) and all submissions (d,e,f). Each row represents the same data with different axes representations: linear (a,d), log-log (b,e), and error (%) relative to the reference (c,f). Overall, in the range of brain T₁ values (500-2000ms) there is good agreement with small measurement error (within 10%) and only a few severe outliers (4 sites had data points with >10% error). For short T₁ values (< 200 ms), the fit breaks down as expected due to the choice of inversion times; for these short relaxation times, near full signal relaxation occurs by the second inversion time (400 ms).

In addition to sharing all the data, pipelines, and analysis notebooks, an interactive dashboard was developed to present a summary of the data and submission details (Figure 5). From this dashboard, we found that there was a mean intra-site coefficient of variation (CoV) of 2.4% across T₁ values within the human range (spheres 1 to 5, 2000 ms to 500 ms) indicating low T₁ variability within ROIs due to noise and other confounding imaging issues. The inter-site mean CoV for this range of T₁ was 6%, which estimates the level of variability from the site-specific implementations of this protocol and technique.

Conclusions

This challenge reports on the level of variability of T₁ values measured in a standard quantitative system phantom across site-specific protocol implementations of a gold standard quantitative technique. With the goal of promoting reproducible research, all collected data, processing pipeline code, computational environment files, and analysis scripts were shared. Future work could compare these results with other multi-center studies that used this phantom to quantify T₁ mapping stability (Bane et al., 2018; Keenan et al., 2021).

Acknowledgements

We thank the Reproducible Research and Quantitative MR ISMRM study group committees for their roles in organizing this initiative. Thank you to Joëlle Barral, Paul Tofts, Ilana Leppert, Tommy Boshkovski, Kiril Zelenkovski, and Filip Velkov for their valuable insights and discussions during the planning stages of this challenge. Special thanks to Kathryn Keenan, Zydrunas Gimbutas, and Andrew Dienstfrey from the National Institute of Standards and Technology for their valuable insights and for sharing their numerical ISMRM/NIST phantom ROI template.