Resting-state functional connectivity Magnetic Resonance Imaging (Rs-fcMRI) models the interactions between brain regions. These interactions are sensitive to the disease states. Hence, machine learning (ML) algorithms have been used to classify mental/neurological disorders based on rs-fcMRI¹. There has been considerable interest towards using ML to develop diagnostic tools and biomarkers. Unfortunately, due to various factors such as subjective diagnostic criteria, heterogeneity of clinical populations, small sample sizes of neuroimaging data and the complexity of underlying altered brain networks, classification models are less reliable/accurate². Given the large variations in accuracy estimates and difficulty in generalizing the classification results to larger populations, broad conclusions about the validity of ML classifiers for disease classification based on Rs-fcMRI metrics are premature. Given such uncertainties, we used a large array of popular ML classifiers with functional connectivity (FC) features to investigate whether the often used cross-validation accuracy is a measure of the true predictive performance of a classifier by measuring accuracy using a truly independent test dataset that was not used in training or cross-validation.We first used simulated data to validate the classifiers using known ground truth. 1500 normally distributed features with means of 0.2, 0.5 and 0.8 were simulated. The standard deviation (SD) for each feature was incremented from 0.1 to 0.8 in steps of 0.1 to test the classifiers’ robustness to noise. Rs-fcMRI data from 132 subjects were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (http://www.loni.ucla.edu/ADNI). 100 subjects were used for training/validation and 32 subjects were entirely kept aside until used for independent diagnosis/testing. Note that the training/validation and testing data had subjects from different age groups, in order to test the generalizability of classifiers (Fig-1). Standard preprocessing pipeline for Rs-fcMRI data was implemented using Data Processing Assistant for Resting-State fMRI toolbox (DPARSF)³. Mean timeseries were extracted from 200 functionally homogeneous brain regions (CC200 template⁴) and FC was calculated for all region pairs. An initial “feature-filtering” was performed wherein only the connectivity paths that were significantly different between the groups (p<0.05, FDR corrected) in the training/validation data were retained (after controlling for possible confounding factors), thereby reducing the number of features from 19,100 to 655. No statistical tests were performed on the independent test data to avoid introducing any bias. To further reduce the number of features while retaining discriminative information, some of the classifiers were embedded in the recursive cluster elimination (RCE) framework⁵. The classifiers used (Fig-4), were implemented both within (Fig-2) and outside the RCE-framework (Fig-3). Using FC from 100 subjects, classification accuracy was calculated using cross-validation and the classifier models obtained from the different partitionings of the training data (repeats×folds) were saved. Test accuracy was calculated on the 32 independent test subjects using the saved classifier models by a voting procedure. Classifier models obtained in each iteration during training would vote towards a decision on test subjects (accuracy was the percentage of correct votes). The classification procedure was identical for simulated and experimental data.Results for simulated and experimental fMRI data are summarized in Fig-4 and plotted in Fig-5. Accuracy obtained from simulated data was close to 100% for both training/validation and test data for both smaller and larger SDs (except few classifiers whose performance degraded with larger SDs), thus validating the classifiers used. Results for fMRI data indicate that in disease classification tasks, the actual predictive performance obtained from independent test data is significantly lower than the cross-validation accuracy obtained from training data. Since we used a voting procedure on testing data from the saved models, if the majority of the classifier models predict accurately then the test data would be assigned to the correct class. Despite this liberal strategy, the performance was only slightly above chance (>25%) for most classifiers using fMRI data. The performance measures from a small sample could not be generalizable to populations with the same symptoms but in a different age group. Also, using several cross-validation levels reduces the amount of data available for each level, thus making feature selection or performance estimation unreliable in smaller datasets. Given the variability in fMRI data and the possibility of selection bias, it is better to use a completely independent test set rather than cross-validation to infer the predictive power of the classifier for smaller datasets⁷. Given that cross-validation is widely used in neuroimaging to assess classification performance, our results demonstrate the perils of using such a strategy. Validation using independent test data is the gold standard in many other scientific fields and we urge the neuroimaging community to adopt the same standard.