2507

Interpretable and Intuitive Machine Learning Approaches for Predicting Disability Progression in Relapsing-Remitting Multiple Sclerosis
Yongmei Li1 and Zichun Yan1
1Department of Radiology,, the First Affiliated Hospital of Chognqing Medical University, Chongqing, China

Synopsis

Keywords: Multiple Sclerosis, Multiple Sclerosis

Motivation: Improving the interpretability and intuitiveness of the machine learning models can help physicians in clinical decision-making.

Goal(s): To investigate whether clinical and grey matter atrophy indicators can predict disability in relapsing-remitting multiple sclerosis (RRMS) and to enhance the interpretability and intuitiveness of a predictive model.

Approach: Six machine learning classifiers were trained and tested to predict disability progression. Partial dependence plot (PDP) analysis and a Shiny web application were conducted.

Results: The logistic regression model performed best, with an AUC of 0.950. PDP analysis showed which indicators had increased probabilities of disease progression. Finally, a Shiny web application was developed.

Impact: The PDP analysis and Shiny web application can improve the interpretability and intuitiveness of the machine learning models to help physicians predict disability progression in RRMS.

Introduction

RRMS is a major cause of nontraumatically severe and irreversible disability progression in young adults and has demonstrated an annually increasing morbidity. Therefore, it is necessary to predict the risk of disability progression among RRMS patients in a timely manner and understand its pathophysiological basis. The limited interpretability and intuitiveness of these machine learning algorithms leads to a poor clinical translation. As such, we aimed to 1) compare the prediction performance between different machine learning classifiers and validate them in an external test set; 2) explore the important clinical and GM atrophy indicators associated with disability progression in RRMS and how they affect the results, thus helping enhance the interpretability of the prediction machine learning model; and 3) develop an open web application to provide decision support for predicting disability progression in RRMS.

Methods

This retrospective study was approved by the Institutional Review Board. Consecutively enrolled patients provided written informed consent before the MRI scans.
145 RRMS patients in the remitting phase were retrospectively enrolled from our institution (centre A) as the discovery cohort. 50 patients for an external test set from three other centres (centre B, C, D) were enrolled. A total of 10 baseline clinical variables including sex, age, disease duration, education level, disease modifying therapy (DMT) administration, time to DMT, time on DMT, relapse time, symbol digit modalities test (SDMT) score and EDSS score were collected.
In the discovery cohort, all MRI scans were performed on a 3.0 T MR scanner (Magnetom Skyra, Siemens Healthcare GmbH, Erlangen, Germany) using a 32-channel head coil. The MRI protocol comprised 3D T1 MPRAGE and 3D FLAIR sequence. In the external test set, 3D T1 and FLAIR sequence were performed on GE, Philips and Siemens scanner.
FreeSurfer toolbox 6.0.0 (http://surfer.nmr.mgh.harvard.edu/) was applied to extracted the grey matter atrophy indicators. Six machine learning classifiers, including logistic regression (LR), decision tree (DT), support vector machine (SVM), random forest (RF), eXtreme Gradient Boosting (XGBoost) and K-Nearest Neighbours (KNN), with 10-fold cross-validation, PDP analysis, and Shiny web application were implemented in R software (version 4.3.0; The R Foundation for Statistical Computing). Statistical analyses were executed with SPSS software (version 25.0; SPSS, Chicago, IL, United States).

Results

In the discovery cohort, 98 patients had disability stability, and 47 patients were classified as having disability progression. In the external test set, 35 patients had disability stability, and 15 patients had disability progression. Models trained with both clinical and radiomics features (area under the curve (AUC), 0.725-0.950) outperformed those trained with clinical (AUC, 0.600-0.740) or radiomics features only (AUC, 0.615-0.945). Among clinical + radiomics feature models, the LR classifier-based model performed best, with an AUC of 0.950. The radiomics feature-only models showed fair performance, with AUCs ranging from 0.617 to 0.753 in the external test set. PDP analysis showed that female patients and those with lower volume, surface area, and SDMT scores; greater mean curvature and age; and no DMT had increased probabilities of disease progression. Finally, a Shiny web application was developed to calculate the risk of disability progression.

Discussion

In recent years, several studies have applied structural MRI to predict disease progression but have solely focused on images or a few GM indicators. For example, a deep learning method based on T1- and T2-weighted brain MRI for predicting the EDSS and SDMT information was developed, achieving an accuracy of 0.857. In another study, classifiers built only on radiological features, including T2-weighted lesion load , thalamic and cerebellar GM volumes, and fractional anisotropy of the normal-appearing white matter, had the highest accuracy of 0.79. In this study, more detailed GM atrophy features combined with clinical data, coupled with the unique advantages of the LR classifier, reasonably led to a slightly better predictive performance for disability progression than previous literature.
In terms of the selected clinical features, older age, no DMT, female sex and lower SDMT scores were positively correlated with higher possibility of disability progression. First, ageing of the immune system and central nervous system might be associated with adverse reactions to DMT, thereby also increasing the likelihood of disability progression. Second, DMT reduces disease activity, delays disease progression, reduces recurrence rates, and reduces inflammation and immune system activity. Additionally, previous studies have shown that female patients have an earlier age of onset of symptoms and a faster disease progression. Last, when MS patients have poorer cognitive function by SDMT at the time of diagnosis, they typically show faster cognitive decline and disability progression during follow-up.

Conclusion

Interpretable and intuitive machine learning approaches might can better predict disability progression in RRMS patients.

Acknowledgements

We thank all the subjects who participated in this study.

References

1. Filippi Met.alBar-Or Aet.alPiehl F, et al. (2018) Multiple sclerosis. Nat Rev Dis Primers. DOI: 10.1038/s41572-018-0041-4

2. Jacques F H (2015) Defining the clinical course of multiple sclerosis: the 2013 revisions. Neurology. DOI: 10.1212/01.wnl.0000462309.76486.c5

3. Confavreux Cet.alVukusic Set.alMoreau T, et al. (2000) Relapses and progression of disability in multiple sclerosis. N Engl J Med. DOI: 10.1056/nejm200011163432001

4. Stenager E (2019) A global perspective on the burden of multiple sclerosis. Lancet Neurol. DOI: 10.1016/s1474-4422(18)30498-8

5. Cagol Aet.alSchaedelin Set.alBarakovic M, et al. (2022) Association of Brain Atrophy With Disease Progression Independent of Relapse Activity in Patients With Relapsing Multiple Sclerosis. JAMA Neurol. DOI: 10.1001/jamaneurol.2022.1025

6. Colato Eet.alStutters Jet.alTur C, et al. (2021) Predicting disability progression and cognitive worsening in multiple sclerosis using patterns of grey matter volumes. J Neurol Neurosurg Psychiatry. DOI: 10.1136/jnnp-2020-325610

7. Cortese Ret.alBattaglini Met.alSormani M P, et al. (2023) Reduction in grey matter atrophy in patients with relapsing multiple sclerosis following treatment with cladribine tablets. Eur J Neurol. DOI: 10.1111/ene.15579

8. de Ruiter L R Jet.alLoonstra F Cet.alJelgerhuis J R, et al. (2023) Association of volumetric MRI measures and disability in MS patients of the same age: Descriptions from a birth year cohort. Mult Scler Relat Disord. DOI: 10.1016/j.msard.2023.104568

9. Filippi Met.alPreziosa Pet.alCopetti M, et al. (2013) Gray matter damage predicts the accumulation of disability 13 years later in MS. Neurology. DOI: 10.1212/01.wnl.0000435551.90824.d0

10. Labiano-Fontcuberta Aet.alCosta-Frossard Let.alSainz de la Maza S, et al. (2023) Predictive models of multiple sclerosis-related cognitive performance using routine clinical practice predictors. Mult Scler Relat Disord. DOI: 10.1016/j.msard.2023.104849

11. Marzi Cet.ald'Ambrosio Aet.alDiciotti S, et al. (2023) Prediction of the information processing speed performance in multiple sclerosis using a machine learning approach in a large multicenter magnetic resonance imaging data set. Hum Brain Mapp. DOI: 10.1002/hbm.26106

12. Shin N Yet.alBang Met.alYoo S W, et al. (2021) Cortical Thickness from MRI to Predict Conversion from Mild Cognitive Impairment to Dementia in Parkinson Disease: A Machine Learning-based Model. Radiology. DOI: 10.1148/radiol.2021203383

13. Storelli Let.alAzzimonti Met.alGueye M, et al. (2022) A Deep Learning Approach to Predicting Disease Progression in Multiple Sclerosis Using Magnetic Resonance Imaging. Invest Radiol. DOI: 10.1097/rli.0000000000000854

14. Tommasin Set.alCocozza Set.alTaloni A, et al. (2021) Machine learning classifier to identify clinical and radiological features relevant to disability progression in multiple sclerosis. J Neurol. DOI: 10.1007/s00415-021-10605-7

15. Spitzer Het.alRipart Met.alWhitaker K, et al. (2022) Interpretable surface-based detection of focal cortical dysplasias: a Multi-centre Epilepsy Lesion Detection study. Brain. DOI: 10.1093/brain/awac224

16. Brieva Let.alEstruch B Cet.alMerino J A G, et al. (2022) Disease modifying therapy switching in relapsing multiple sclerosis: A Delphi consensus of the demyelinating expert group of the Spanish society of neurology. Mult Scler Relat Disord. DOI: 10.1016/j.msard.2022.10380

17. Zivadinov Ret.alBergsland Net.alJakimovski D, et al. (2022) Thalamic atrophy measured by artificial intelligence in a multicentre clinical routine real-word study is associated with disability progression. J Neurol Neurosurg Psychiatry. DOI: 10.1136/jnnp-2022-329333

18. Benedict R H Bet.alAmato M Pet.alDeLuca J, et al. (2020) Cognitive impairment in multiple sclerosis: clinical management, MRI, and therapeutic avenues. Lancet Neurol. DOI: 10.1016/s1474-4422(20)30277-5

19. Wattjes M Pet.alCiccarelli Oet.alReich D S, et al. (2021) 2021 MAGNIMS-CMSC-NAIMS consensus recommendations on the use of MRI in patients with multiple sclerosis. Lancet Neurol. DOI: 10.1016/s1474-4422(21)00095-8

20. Raz Eet.alCercignani Met.alSbardella E, et al. (2010) Clinically isolated syndrome suggestive of multiple sclerosis: voxelwise regional investigation of white and gray matter. Radiology. DOI: 10.1148/radiol.2541090817

Figures

Figure 1. Flowchart of patient inclusion. RRMS, relapsing-remitting multiple sclerosis; EDSS, Expanded Disability Status Scale.

Figure 2. Working flow chart of this study. Data preprocessing included feature extraction, feature selection and the LASSO screening process. Model training included six machine learning classifiers built with clinical and radiomics features. Finally, model evaluation included selection of the optimal prediction model, model validation, interpretability analysis and Shiny web application.

Figure 3. ROC curve analysis. Each model was constructed with the data from 145 patients in the discovery cohort by randomly dividing them into training sets and testing sets at a ratio of 8:2. (A) ROC curves of models constructed with clinical features. (B) ROC curves of models constructed with radiomics features. (C) ROC curves of models constructed with clinical+radiomics features.

Figure 4. Interpretability analysis. (A) Importance ranking of a total of 17 indicators (4 clinical features and 13 radiomics features). PDP analysis with numerical (B) and categorical (C) indicators in the top 10 after importance ranking.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
2507
DOI: https://doi.org/10.58530/2024/2507