Synopsis

Keywords: Machine Learning/Artificial Intelligence, Aging, Ex-Viov Applications, Neuro (Microbleeds)

Detection of cerebral microbleeds (CMBs) on ex-vivo gradient echo MR images is an important task in MRI-pathology studies in older adults. The goal of this study was to develop a confidence-aware ex-vivo CMB detection algorithm that outputs interpretable probabilities and ranking of CMB candidates on brain MRI scans of community-based older adults. The present study demonstrates that training an ex-vivo CMB detection model with confidence-aware deep learning, a technique for improving confidence estimation and ordinal ranking of examples in classification models, improves detection performance and prediction interpretability.

Introduction

Annotation of cerebral microbleeds (CMBs) on ex-vivo brain MRI scans of autopsied community-based older adults is important for MRI-pathology studies of cerebral small vessel disease (SVD)^­1-3. However, training an ex-vivo CMB detection model is difficult due to the low prevalence of CMBs in the brains of community-based older adults, and the fact that CMB mimics (e.g. air bubbles in the sulci) greatly outnumber CMBs on ex-vivo MRI (Fig.1). Deep learning-based approaches often suffer from overconfident predictions⁴, which is greatly exacerbated in ex-vivo CMB detection where noisy ground truth labels and the use of synthetic data can introduce detrimental biases during neural network training. In this study, we demonstrate that confidence-aware deep learning⁵, a technique for enforcing correct ranking of examples during neural network training, significantly improves ex-vivo CMB detection performance and enhances the interpretability of model predictions.

Methods

Data background and preparation
Cerebral hemispheres from 286 participants of the Rush Memory and Aging Project6 and Religious Orders Study⁷, two longitudinal cohort studies of aging, were included in this work. Ex-vivo multi-echo 3D gradient echo MR images of these hemispheres were collected on 3T scanners with a voxel size of 1x1x1 mm³. Bias field correction was applied to the images and CMBs were manually annotated by an experienced rater blinded to all clinical and pathological information.

Confidence-aware learning
The goal of confidence-aware learning is to regularize deep learning model predictions such that they reflect the degree of confidence a model has in a prediction. Ordinal ranking according to confidence is used as an objective to enforce confidence-aware learning by ensuring that the model gives higher probabilities to examples for which it has greater confidence in its prediction. An estimate of model confidence is modeled during training by correctness, the number of correct prediction events across epochs divided by the total number of epochs. The objective loss function is modified to include a correctness ranking loss (CRL) term that regularizes the class probabilities such that they are ordered according to their correctness scores (Fig.2). CRL gradient updates are computed pairwise across mini-batches on random subsets of all possible pairs during training and summed. A penalty is inflicted each time a pair of example probability estimates violate the ranking of their correctness scores proportional to the size of the difference between probabilities and correctness scores. The final loss function used is the sum of a standard cross-entropy loss and CRL. In this work, models are trained with and without the CRL loss term to evaluate the impact of confidence-aware learning on model performance.

Detector training and evaluation
An end-to-end CMB detection framework that combines data synthesis, candidate selection, false positive reduction, and full scan evaluation was used as the backbone for this work (Fig.3)⁸. A 3D convolutional neural network adapted from ResNet18⁹ was trained to detect whether a patch was centered on a CMB. Patches of size 16x16x16x10 made up of 4 signal channels and 6 PCA feature channels served as input. A custom model for synthesizing hypointensities onto healthy background tissue was used to generate CMBs and non-CMB examples in order to increase the size of the dataset8. Patches from 492 real CMBs were used for training and evaluation. A high sensitivity, low precision SVM based candidate selector was trained to discriminate between CMB and non-CMB voxels from pre-generated image features. Both training and evaluation were done using a repeated randomized 5-fold cross-validation technique to avoid training or evaluation biases caused by the small dataset size. Folds were split by participant. Final evaluation probability maps were generated by averaging the probability maps of each individual holdout run for each participant. Non-maximum suppression and probability smoothing were applied to the final probability maps to produce the final detections.

Results

CMB detection models trained using correctness ranking loss achieved a significantly higher ensemble average precision (0.2915) compared with models trained without correctness ranking loss (0.2379) (Fig.4). The CRL trained ensemble model achieved a 25% higher sensitivity at 0.5 false positives per subject and 12% higher sensitivity at 3 false positives per subject when compared to models trained without CRL.

Discussion

Confidence-aware learning with CRL led to significantly improved generalization performance for ex-vivo CMB detection compared to training without CRL. CRL-trained models demonstrated an enhanced ability to discriminate the most well-defined CMBs at low false positive rates. This property is important in cases where a stricter quantification of CMBs may be desirable for analysis, as the inclusion of borderline cases can increase label uncertainty even for manual raters. CRL-trained models also exhibit probabilities that can be more easily interpreted as confidence measures, a key property for computer assisted ex-vivo CMB annotation. The present work is the first to consider confidence-aware learning to improve CMB detection and establishes a new state-of-the-art for ex-vivo CMB detection in community-based older adults.

Conclusion

This work demonstrates that training with confidence-aware learning can improve the performance and interpretability of ex-vivo CMB detection algorithms in community-based older adults where CMB prevalence is low and mimics are abundant. The use of a confidence-aware CMB detection algorithm for assisted CMB annotation in ex-vivo MRI will aid MRI-pathology studies of cerebral small vessel disease in community-based older adults.

Acknowledgements

National Institute of Neurological Disorders and Stroke (NINDS) UF1NS100599National Institute on Aging (NIA) R01AG064233, R01AG067482, R01AG017917, R01AG015819, P30AG010161, P30AG072975