Introduction

Multi-centre studies are a crucial step in the translation of quantitative MR imaging biomarkers (qMRIBs) from academic research into clinical practice¹. To ensure that analysis outcomes are robust, it is essential that acquisitions are performed according to the study imaging protocol. Manual monitoring of protocol compliance is time-consuming and error-prone, so this work presents a software tool, Protocol Checker, for automatic compliance checks².
The importance of standardised acquisition is reflected in the Myeloma Response Assessment and Diagnosis System³ (MY-RADS) for whole-body (WB-) MRI in myeloma. MUKnine is a prospective multi-centre study using WB-MRI to monitor treatment response in patients with multiple myeloma⁴.
Sites enrolled in the MUKnine study underwent a qualification process to establish a MY-RADS-compliant imaging protocol on a local scanner, with some protocol deviations agreed in advance for specific sites due to hardware and scan time limitations⁵.
The aims of this work were to demonstrate Protocol Checker, and to evaluate protocol compliance in a multi-centre WB-MRI study.

Methods

Protocol Checker was developed in Python, containerised using Docker and installed on the informatics platform XNAT⁶ (v1.8.9) using the container service plugin⁷ (v3.4.1). The pipeline is summarised in Figure 1. For each site, the software generates a protocol file specifying the expected series and acquisition parameter values based on data from a phantom, volunteer or exemplar patient acquired during site qualification. The user can select any DICOM header tag as a parameter to check. All subsequent patient examinations are then compared to the protocol file, generating an output file that summarises missing series and parameter discrepancies.
The software was applied retrospectively to the MUKnine dataset, which contains 174 WB-MRI examinations acquired across ten UK sites using scanners from three manufacturers and at two field strengths.
Two tiers of protocol compliance were defined, as summarised in Figure 2. The first evaluated only whether the scan was conducted in accordance with MY-RADS, while the second included more parameters to compare acquired data to the protocol established during site qualification.

Results

Protocol Checker ran successfully for all examinations in the MUKnine dataset. Extracts from example protocol and results files are shown in Figure 3. Protocol deviations are summarised in Figures 4 and 5 for the two tiers of compliance checks.
88% of examinations were completely compliant with the MY-RADS guidelines, with the most common deviations being incorrect slice thickness for diffusion-weighted imaging (DWI) (93% compliance) and Dixon (92%). Five exams were missing DW and Dixon imaging, and one DWI exam included only two b-values, instead of three. Deviations from the site-specific protocols were more common, although compliance was greater than 75% for all parameters and two sites did not have any deviations. Common deviations included number of averages for DWI (85% compliance) and TR for DWI (82%) and Dixon (75%). 96% of exams were acquired using the scanner that was qualified for this study.

Discussion and Conclusions

For quantitative imaging, consistent acquisition protocols are essential to ensure the repeatability of measurements. In clinical reality protocol modification may be justified, for example to reduce scan time for a patient in severe pain, but these deviations should be recorded so they can be considered at the time of analysis.
Despite the challenges of conducting a multi-centre study during the covid-19 pandemic, the MUKnine dataset has previously been shown to have good overall image quality⁸. This is complemented by the finding here that most sites were able to conduct most examinations according to the MY-RADS guidelines. Protocol deviations did still occur however, underlining the challenge of multi-centre imaging studies. For example, all the exams from Site 4 were conducted with a slice thickness of 7mm (rather than 5mm) and insufficient DWI averaging.
The use of Protocol Checker in MUKnine has shown that MY-RADS-compliant protocols can be established and successfully used at a range of sites with good overall protocol compliance. Real-time use of Protocol Checker would have allowed better compliance to be achieved, with quick, automatic identification of deviations enabling mistakes to be rectified before the next patient was scanned. This requires data to be uploaded to the repository soon after acquisition to allow issues to be addressed quickly and may also depend on standardised anonymisation procedures to ensure relevant DICOM tags are not removed.
The National Cancer Imaging Translational Accelerator (NCITA) is developing the infrastructure to support greater standardisation of image acquisition, analysis, and quality control in UK-based multi-centre studies⁹. NCITA proposes that Protocol Checker will form part of an automated repository-integrated pipeline of publicly available data quality control and analysis tools to aid the clinical translation of qMRIBs.

Acknowledgements

We would like to acknowledge Janssen and Celgene (for supporting the MUKnine OPTIMUM study). This study represents independent research funded by the National Institute for Health and Care Research (NIHR) Biomedical Research Centre at The Royal Marsden NHS Foundation Trust and The Institute of Cancer Research, London, and by the Royal Marsden Cancer Charity and the Cancer Research UK National Cancer Imaging Translational Accelerator (NCITA). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

References

1. O'Connor, J.P., Aboagye, E.O., Adams, J.E., Aerts, H.J., et al., Imaging biomarker roadmap for cancer studies. Nature reviews Clinical oncology, 2017. 14(3): p. 169-186.
2. Sinha, H. and Raamana, P.R. Solving the pervasive problem of protocol non-compliance in MR imaging using an open-source tool mrQA. in Proceedings of the Annual Meeting of the ISMRM. 2023. Toronto, Canada.
3. Messiou, C., Hillengass, J., Delorme, S., Lecouvet, F.E., et al., Guidelines for acquisition, interpretation, and reporting of whole-body MRI in myeloma: myeloma response assessment and diagnosis system (MY-RADS). Radiology, 2019. 291(1): p. 5-13.
4. Brown, S., Sherratt, D., Hinsley, S., Flanagan, L., et al., MUKnine OPTIMUM protocol: a screening study to identify high-risk patients with multiple myeloma suitable for novel treatment approaches combined with a phase II study evaluating optimised combination of biological therapy in newly diagnosed high-risk multiple myeloma and plasma cell leukaemia. BMJ open, 2021. 11(3): p. e046225.
5. Rata, M., Blackledge, M., Scurr, E., Winfield, J., et al., Implementation of Whole-Body MRI (MY-RADS) within the OPTIMUM/MUKnine multi-centre clinical trial for patients with myeloma. Insights into Imaging, 2022. 13(1): p. 1-16.
6. Marcus, D.S., Olsen, T.R., Ramaratnam, M., and Buckner, R.L., The extensible neuroimaging archive toolkit. Neuroinformatics, 2007. 5(1): p. 11-33.
7. Washington University School of Medicine. XNAT Container Service. 2023 06/10/2023]; Available from: https://bitbucket.org/xnatdev/container-service/src/master/.
8. Keaveney, S., Dragan, A., Rata, M., Blackledge, M., et al., Image quality in whole-body MRI using the MY-RADS protocol in a prospective multi-centre multiple myeloma study. Insights into Imaging, 2023. 14(1): p. 1-14.
9. McAteer, M., O’Connor, J.P., Koh, D., Leung, H., et al., Introduction to the National Cancer Imaging Translational Accelerator (NCITA): a UK-wide infrastructure for multicentre clinical translation of cancer imaging biomarkers. British journal of cancer, 2021. 125(11): p. 1462-1465.