Synopsis

Keywords: Diffusion/other diffusion imaging techniques, Data Processing

The lack of a standardized preprocessing pipeline is a significant source of variability that might lower the reproducibility of studies, especially across sites and with incomplete description of the preprocessing workflows. We evaluate the downstream impact of variability in preprocessing workflow by quantifying the reproducibility and variability of region-of-interest (ROI) analyses. While many pipelines achieve excellent reproducibility in most ROI, we observed a large variability in performance of preprocessing workflows to the extent that some pipelines are detrimental to the data quality and reproducibility.

Introduction

The visual and quantitative interpretation of diffusion MR images and derived biomarkers is challenged by various imaging artifacts and thermal noise [1]. Research looking to maximize the sensitivity, specificity, and robustness of their diffusion MR analysis are bound to a multi-step workflow that include image preprocessing. The implementation and integration of methodological advances in publicly available software packages and pipelines keeps improving the accessibility and scalability of complex dMRI preprocessing workflows. However, in the absence of field-accepted standards, the preprocessing pipelines vary substantially across gresearchers [2].

The lack of a standardized preprocessing pipeline is a significant source of variability that might lower the reproducibility of studies, especially across sites and with incomplete description of the preprocessing workflows [2,3]. We further hypothesize that the downstream impact of variability in preprocessing workflow depends on the actual post-processing analysis. In this work, we will evaluate the inter-pipeline variability on region-of-interest (ROI) analyses.

A ROI analysis is a widely-adopted strategy for the quantitative and statistical analysis of diffusion metrics. Typically, in an automated workflow, anatomically defined regions have been delineated in a brain atlas or template and image registration is used to align subject-specific dMRI data with such template. Any lack of anatomical fidelity of the dMR images due to unresolved imaging artifacts is likely to lower the accuracy and precision of such ROI analysis.

Methods:

Data acquisition:
Multi-shell SE-EPI dMRI data (b=0, 1000, and 2000 s/mm2) was acquired on multiple sites using hardware systems from different vendors (3T GE SIGNA Premier, 3T Connectome Siemens, and 3T Achieva Philips). The data include unprocessed magnitude MRI data sets of 13 subjects. Defacing was applied to a subset of data to comply with institutional regulations. For each subject, three scan sessions (“A”, “B”, and “C”) with distinct settings and instructions related to the motion were acquired to introduce artifacts of varying shapes and sizes while preserving the encoded diffusion information. In addition, the dataset included non-diffusion-weighted images with reversed phase encoding, a T1-weighted MPRAGE, and a field map.

Image preprocessing:
54 research teams processed the individual datasets with their routinely-used preprocessing pipeline. All further processing was performed with identical software and algorithms.

Region-of-interest (ROI) analysis:
The widely-used ICBM DTI-81 JHU atlas was chosen as it provides 48 anatomical regions within the cerebral white matter. Each subject’s FA map was rigidly coregistered to its corresponding MPRAGE scan using mutual information. All subject-specific MPRAGE scans were spatially normalized to the T2-weighted template of the JHU atlas using an affine and deformable transformation, with mutual information as an optimization metric and elastic regularization [5]. The deformable and rigid transformations were combined prior to applying them inversely to the JHU region atlas with nearest neighbor interpolation to project the template ROIs to the native diffusion space to avoid interpolation or manipulation of the diffusion maps; see Figure 1. Finally, for all 48 ROIs, the mean FA was calculated, but results from both hemispheres were averaged, resulting in effectively 27 ROI values per subject and per image preprocessing pipeline.

Statistical analysis:
We evaluated the inter-subject, intra-subject, and inter-pipeline variability and inter-session agreement using the coefficient of variation (CoV) and the intraclass correlation coefficients (ICC), respectively. The ICC(2,1) was calculated for two-way mixed effects, single measurement, with an absolute agreement [6].

Results:

In Figure 2, we show the absolute agreement of ROI-averaged FA across scan sessions per preprocessing pipeline for the Posterior limb of the internal capsule and the genu of corpus callosum. For comparison, the absolute agreement of the unprocessed data is shown as well.

In Figure 3, we show the agreement for each ROI with and without preprocessing. The preprocessed value is represented by the 95th percentile of all pipelines. In addition, we show the spatial distribution by color-encoding the atlas label by absolute agreement.

In Figure 4, we visualize the absolute agreement spatially over the ROIs in the standard MNI space.

In Figure 5, we show the intra-subject, inter-subject, and inter-pipeline variability for each ROI. The intra-subject and inter-subject variability were derived from preprocessed data. We report the median of the three pipelines that were most often ranked in the top 5% pipelines when comparing absolute agreement per ROI. For the inter-pipeline variability, we used robust proxies of the mean and standard deviation to compute the CoV to avoid biasing the results by outliers that could easily have been detected using quality assessment of the data.

Discussion

Our results demonstrate that image preprocessing is a critical step in the diffusion MRI workflow that typically improves data reliability and reproducibility of ROI analyses. The impact of the image preprocessing pipeline strongly depends on the spatial location of the ROI, but we observe a significant improvement in all white matter ROIs, including the genu and splenium of the Corpus Callosum, for most preprocessing pipelines.

While many pipelines achieve excellent performance in terms of absolute agreement, despite the introduced artifacts, we observed a large variability in performance of preprocessing workflows to the extent that some pipelines are detrimental to the data quality and reproducibility. We believe that automated quality control (QC) can detect such cases and that the use of such QC tools is warranted.