Introduction:

Deep learning based neural networks have been successfully applied to various prediction tasks in resting-state fMRI (rs-fMRI) brain functional connectivity (FC) and outperform conventional machine learning models¹. However, in many circumstances knowing which features matter most in a prediction is as important as reaching a high accuracy, but the feature selection in deep neural network (DNN) often remains a black box. In this study, we developed understanding of the DNN prediction mechanism and proposed a feature selection and ranking method based on each feature’s contribution to the prediction. Experiments were done on the rs-fMRI FC gender prediction to extract gender related brain FC patterns², which can be used as a testbed to study the method’s performance and characteristics for it being applied to other feature selection applications.

Methods:

In Fig. 1, the weight vectors between the input layer and the 1st hidden layer of DNN can be considered as convolutional kernels. These kernels are trained to extract the FC patterns which are most helpful on classifying different classes from the input connectivity matrices. Between the 1st and 2nd hidden layers the weights are used to look for a certain combination of the patterns for the later activations. The absolute value of the weight determines the importance of the corresponding pattern, and the sign of the weight means whether the pattern is preferred to appear in the combination or not. When it comes to the higher hidden layers, the higher level of combinations of the patterns are evaluated for the classification task. The key part of the DNN structure for extracting and ranking the features for each class in the proposed method is between the last hidden layer and the softmax layer. In the application of gender classification, there are two classes, male and female. Each activation, in the last hidden layer will be multiplied by the weights w_M and w_F respectively towards the male and female neurons. The two scores compete with each other and the input will be classified into the class with the larger score. The sign of the difference w_M - w_F determines the class the high-level FC pattern belongs to, while the absolute value of the difference determines how important it is for the DNN classification. Therefore, the absolute value of the difference can be sorted to rank the contribution of the high-level features for both classes. Since a high-level feature can be treated as the combination of lower-level features, they can be mapped back to the FC patterns at the original input level¹. Experiments were done on the rs-fMRI data of 1003 healthy adults in the Human Connectome Project S1200 release³ in the framework of extracting features from predictive models with high accuracies. To test the robustness of the models, all the experiments were done with 50 randomly permuted 2-fold cross validations.

Results:

Fig. 2 shows that DNN reaches 83.0%, 87.6%, 92.0%, 93.5% and 94.1% accuracies respectively with the FC input derived from 25, 50, 100, 200, 300 independent component analysis (ICA) components. DNN outperforms the linear support vector machine (SVM) (Bonferroni corrected p-values<0.001) when smaller ICA component numbers are used. The proposed feature ranking method was validated with the cross-entropy loss on the training dataset. Fig. 3 shows that the more highly ranked feature pair has a lower cross-entropy loss, which means it is better at making the predicted label distribution similar to the ground truth distribution. The robustness and repeatability of the features extracted was also studied. Fig. 4 shows as the neural network becomes deeper, the repeatability of the most important high-level feature becomes higher. The most important male and female features extracted by DNN are shown in Fig. 5. For both 25 and 50 ICA components, females have stronger DM-DM, CC-VIS connections, relatively weaker CC-DM connections, and both stronger and weaker connections in CC-CC (as indicated by the black boxes in Fig. 5). These findings are consistent with previous studies^2,4.

Conclusion:

In this study, we proposed a feature selection method for DNN according to each feature’s contribution to the final prediction, which provides a new understanding of the DNN model and makes DNN no longer a black box. Experiments on FC gender feature extraction serve as the basic framework to verify and study the performance and characteristics of DNN feature selection for its further applications in other fields of MRI/fMRI based predictions.

Acknowledgements

We gratefully acknowledge the Human Connectome Project for the data collection, data processing and data sharing. We also acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. This work was supported by grants from the National Institutes of Health: U01-NS093650. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.