INTRODUCTION

Autism Spectrum Disorders (ASD) are a group of neurodevelopmental diseases involving behavioral, social and communicative disabilities. Many studies focus on the analysis of brain MRI images, aiming to detect the anatomical features that characterize ASD. These studies are generally carried out according to the following procedure: collection of brain MRI images of ASD and control subjects from various research centers; image segmentation through specific software tools (e.g. Freesurfer) to extract quantitative measurements of anatomical regions; training classifiers based on Machine Learning (ML), which are able to extract patterns that allow to distinguish ASD from control subjects. The accuracies of these biomarkers in distinguishing ASD from controls are barely reproducible across different studies^1,2,3. The lack of shared results could be attributed to inhomogeneities induced in the images by different MRI scanners or different acquisitions settings that characterize the data of every research center⁴ or sub-optimal performances of segmentation algorithms⁵. Particularly in case of psychiatric diseases, where macroscopic anatomical differences between case and control subjects do not exist, it is of utmost importance to understand the many possible confounding factors that can affect ML analyses.

METHODS & RESULTS

The T1-weighted MRI scans of the about 2000 subjects of the biggest international database on autism (ABIDE I and II collections) have been processed with Freesurfer 6.0 (recon-all workflow) and 420 cerebral features were extracted. Then a multivariate and univariate outlier analysis has been conducted to exclude anomalous data. In order to investigate the effect that different acquisition settings have on a multivariate analysis, different classifiers have been trained on age-matched and sex-matched subjects acquired in different sites. In figure 1 results are reported representing the general performances of classifiers trained to distinguish between all the possible couples of the 39 different acquisition sites that contributed to ABIDE. These results pose new questions: indeed, if the differences due to the acquisition site are so easily detected by a ML approach, there is the risk that, if a group of subjects with the same medical condition comes also from the same site, the classifier may learn a criterion based on the acquisition site rather than the medical condition, giving the false impression of extremely good results. In fact, as the classifiers trained to distinguish cases from controls in subjects coming from the same site show (figure 2), the classification performances are greatly inferior with respect to the ones reported in figure 1. The findings described in this work may seem rather unusual, given that various studies^6,7 affirm that the features extracted with Freesurfer are comparable, even when the acquisition parameters vary. This is due to the fact that a univariate analysis is not able to capture the real proportion of the differences due to the acquisition settings, since they are evenly spread among the various features. This concept is shown in the plots in figure 3. Finally, an analysis was made to investigate the possibility of eliminating or reducing the influence of the acquisition site by removing some features, however, it has been found that there is not a single group of features that varies consistently among the different sites. Therefore, the number of features with which the classifier is trained has been progressively reduced (randomly deleting some features), in order to understand how many features the classifier needs to correctly classify the acquisition site of the subjects. As it can be observed from the plot in figure 4, the performances start to degrade when the feature number goes below 10. Using such a small sample of features would however defeat the purpose of using ML techniques to find patterns in a great amount of data.

CONCLUSION

This study shows that the differences due to the acquisition protocol can have a strong impact on a multivariate analysis. It also emerges that, even if the same protocol is used, small differences (probably intrinsically tied to the specific scanner) make the data not comparable. Preprocessing pipelines devoted to data harmonization⁸, quality check on the extracted features, outlier removal and careful matching for confounding variable are necessary preliminary steps to improve the reliability of the results in multi-center studies. Furthermore, these results underline the necessity of using different acquisition modes, able to detect physical signals more dependent on tissue types rather than acquisition settings, like the recently introduced quantitative Magnetic Resonance Fingerprinting⁹.

Acknowledgements

This work has been supported by the Tuscany Government (PAR-FAS2007-2013, Bando FAS Salute 2014) through the ARIANNA Project(C52I16000020002) https://arianna.pi.infn.it