Introduction

Quantitative Susceptibility Mapping (QSM) is an MRI phase post-processing technique that derives the magnetic susceptibility distribution of tissues and is mostly applied in human brain imaging applications¹. QSM involves a complex post-processing pipeline that involves solving multiple ill-posed inverse problems. This ill-posedness makes QSM evaluation challenging, with the first QSM challenge² using a silver-standard COSMOS³ acquisition as the ground truth and the second⁴ using a realistic in-silico head phantom⁵. Public challenges like these are great opportunities to involve the QSM community and report on the current landscape of QSM algorithms under a common evaluation framework. However, running these challenges is a lot of effort, and there is an opportunity to build an open-source platform for continuous QSM evaluation that is maintained by the community and is always available to submit algorithms, metrics or updated test data and remain relevant into the future. This work presents a continuous QSM challenge platform called QSM-CI, with a prototype implementation published using Back4app and automated using GitHub Actions (https://github.com/QSMxT/QSM-CI). Users can submit algorithms to automatically evaluate against a range of simulated datasets using quantitative metrics and a qualitative Elo rating system.

Methods

The QSM-CI GitHub project includes instructions on how to generate and download test datasets, run QSM algorithms and compute metrics.
The datasets include simulations for gradient-echo (GRE) magnitude and phase images and other necessary data required for QSM reconstruction using the Brain Imaging Data Structure⁶ (BIDS). These simulations include a susceptibility phantom consisting of cylinders with constant susceptibility values and data derived from a realistic in-silico head phantom⁵. The forward simulation was implemented using the qsm-forward pip package⁷.
The algorithms include user-submitted instructions to execute QSM reconstruction pipelines against the BIDS dataset. The current submissions include algorithms available in QSMxT^8,9.
The metrics include the quantitative metrics from the second QSM challenge, including RMSE, NRMSE, HFEN, XSIM¹⁰, MAD, CC and GXE, and a qualitative Elo metric. After a user submits or updates one of their algorithms via a pull request, a GitHub Action will automatically run their pipeline against the datasets and evaluate it using quantitative metrics, publishing them to a Parse backend. Qualitative metrics remain blank until the community contributes to the anonymised evaluation of the final images in a frontend interface using Niivue¹¹ (see Figure 1).

Results

Multiple algorithms have been submitted to QSM-CI with metrics automatically computed and uploaded to the database (see Figure 2).

Discussion and conclusion

A platform for automatically evaluating QSM algorithms was developed and published using GitHub and automated using GitHub Actions. A frontend interface was developed to display the computed metrics for each algorithm against the test datasets, along with an interactive frontend that uses Niivue¹¹ to allow users to interactively rate image quality and determine a qualitative Elo ranking. The QSM-CI project provides a proof-of-concept for an always-online QSM challenge platform to streamline the evaluation of QSM algorithms.

Acknowledgements

No acknowledgement found.

References

[1] Deistung A. et al., NMR Biomed., 2017; [2] Langkammer C. et al., MRM, 2018; [3] Liu T. et al., MRM, 2009; [4] Bilgic B. et al., MRM, 2021; [5] Marques J. P. et al., MRM, 2021; [6] Gorgolewski K. J. et al., Sci Data, 2016; [7] Stewart A. W. et al., GitHub, 2023, https://github.com/astewartau/qsm-forward [8] Stewart A. W. et al., GitHub. 2020. https://qsmxt.github.io/; [9] Stewart A. W. et al., MRM, 2022; [10] Milovic C. et al., Proc Intl Soc Mag Reson Med, 2022; [11] Hanayik T., Rorden C. et al., GitHub, 2023, https://github.com/niivue/.