Introduction

Robust quality assessment and quality control (QA/QC) protocols are now widely considered key to ensuring the reliability of neuroimaging analyses^1,2. Data of insufficient quality (e.g., showcasing acute and widespread head motion) introduce biases sometimes comparable in size to the investigated effects across modalities, as reported for anatomical, functional, and diffusion MRI^3–5.
While it has long been acknowledged that comprehensive QA/QC protocols are crucial⁶, their practical implementation has proven challenging. Despite efforts such as MRIQC^6–8, automated decision-making has not achieved levels of sensitivity and specificity to be reliably adopted. Consequently, QA/QC remains fundamentally a manual task in which one or more experts screen images individually. Even with the adoption of visual assessment aids such as MRIQC’s visual reports^9,10, the endeavor is time-consuming, with the burden exploding with the scale of the dataset. Moreover, the visual QA/QC is prone to inter- and intra-rater variabilities¹¹ that can only be reduced with stringent protocols⁹, bookkeeping errors addressed with managers¹², and setting aside other intrinsic sources of variability such as fundamental preprocessing steps before assessment¹³. In our previous work⁹, we approach QA/QC from a Swiss cheese security model, in which several layers (QC checkpoints) are established along the neuroimaging pipeline with pre-defined exclusion criteria to ensure no subpar images reach the analysis step. Here, we demonstrate the first layer of such an approach (i.e., unprocessed data) on a mid-sized, single-subject dataset leveraging MRIQC. By streamlining this QA/QC checkpoint with the acquisition, the QA/QC protocol allowed the collection of additional data to replace excluded sessions that met QC rejection criteria.

Methods

Data. We employed 40 sessions of the Human Connectome PHantom (HCPh) project¹⁴, a single-individual, dense-sampling study repeating the same comprehensive protocol, including high-resolution, multi-shell diffusion MRI (dMRI), a quality-control task BOLD (blood-oxygen-level-dependent) functional MRI (fMRI), a breath-holding task (BOLD-fMRI), and, finally, resting-state fMRI (RSfMRI, BOLD) while repeatedly watching the same natural-scenes video clip. MRI parameters and further data collection details are described in Tables 2 and 3 of our pre-registered report¹⁴. At the time of submission, 90% of the data had already been collected and used in this communication (20 piloting sessions and 20 planned sessions).
Standard Operating Procedures (SOPs) and QC exclusion criteria. The HCPh pre-registration¹⁴ is accompanied by comprehensive SOPs¹⁵ documenting in detail all aspects of the experiment settings. The SOPs are publicly available online (https://www.axonlab.org/hcph-sops). As we advocated previously⁹, reliable QA/QC requires the definition of exclusion criteria for each QC checkpoint. These exclusion criteria must be tailored to the study's specific application, goals, and data availability and predefined before starting the QA step of the protocol.
Visual QA. We executed MRIQC on all the image modalities supported by the tool (namely, anatomical T1-weighted —T1w—, and T2-weighted —T2w—, dMRI, and BOLD-fMRI). MRIQC generates one visual report per input image, yielding 155 (40/25/30/60; T1w/T2w/dMRI/fMRI) reports in HTML format, which are screened by authors CP and OE.
Image quality metrics (IQMs). MRIQC extracts a vector of features related to quality aspects of the data (e.g., signal-to-noise ratio, which we have demonstrated to reflect head motion on T1w images16). Following MRIQC’s procedures, we retrained the MRIQC-learn classifier to assist in the exclude/include decision of T1w images.

Results

A formalization of exclusion criteria implemented within the HCPh project. We systematically formalized exclusion criteria for all the modalities supported by MRIQC (i.e., T1w, T1w, dMRI, and BOLD-fMRI) tailored to the objectives of the HCPh project toward the reliability characterization of structural and functional brain networks. These exclusion criteria are maintained under version control and rendered for best visualization at https://www.axonlab.org/hcph-sops/data-management/mriqc/.
Quality reports for 155 MRI with QA annotations from two experts. Figure 1 showcases how MRIQC’s visual reports facilitate the QA screening process. Two experts utilized these reports to assign quality ratings using the 'Rating widget,' identifying images that should not be employed in further analyses.
A random-forests classifier was calibrated to flag T1w images of the HCPh for exclusion. We recalibrated MRIQC’s random-forests classifier (MRIQC-learn), fine-tuning it for the T1w images in the HCPh dataset, improving over the original 83% accuracy of the model.

Acknowledgements

OE and CP receive support from the Swiss NSF #185872. The development of MRIQC was supported by the NIMH (RF1MH121867, OE).