4112
Evaluation of the inter-reader reproducibility of the PI-QUAL scoring system for prostate MRI quality1University College London, London, United Kingdom, 2University College London Hospital, London, United Kingdom
Synopsis
We assessed the interobserver reproducibility of the Prostate Imaging Quality (PI-QUAL) score for prostate MR quality between two expert radiologists who independently scored a total of 41 multiparametric prostate MR scans from different vendors and scanners. All men included in this study had biopsy-confirmed prostate cancer and received robotic-assisted laparoscopic prostatectomy after imaging. Agreement was substantial (κ = 0.77; percent agreement = 80%) when assessing each single PI-QUAL score (1 to 5). Two expert radiologists achieved substantial reproducibility for the PI-QUAL score, but the composition of the scoring system will need to undergo further refinements.
Introduction
The PI-RADS guidelines set out the minimal technical requirements for the acquisition of multiparametric magnetic MRI (mpMRI) of the prostate. However, the rapid diffusion of this technique has inevitably led to variability in vendor and scan quality among centres across the world. Suboptimal image acquisition reduces the diagnostic accuracy for the detection of clinically significant prostate cancer and can result in clinicians losing confidence in this technique. Two panels of experts 1,2 have stressed the importance to establish quality criteria for the technical acquisition of mpMRI of the prostate.A first attempt to address this issue has been the publication of the PI-QUAL scoring system 3 from the multi-centre PRECISION trial 4. The PI-QUAL score assesses the quality of prostate mpMRI against both a set of objective technical criteria (as per PI-RADS guidelines) together with a set of subjective criteria from the MR images. PI-QUAL is based on a 1-to-5 scale that indicates the adequacy of the diagnostic quality of a scan, where 1 means all sequences are below the minimum standard of diagnostic quality, 3 means that the scan is of sufficient diagnostic quality, and 5 means that all three sequences are of optimal diagnostic quality. A PI-QUAL score ≥ 4 means that the quality of the MR is high, and all significant lesions can be ruled in and ruled out (Table 1).
In this study we evaluated the inter-reader reproducibility of the PI-QUAL score for assessing the quality of prostate MRI in patients enrolled in the multicentre NeuroSAFE PROOF trial 5 in which all men received mpMRI before receiving radical prostatectomy.
Methods
Two experienced radiologists (with 20 and 7 years of experience in prostate MRI, respectively) evaluated the quality of prostate MRI independently using the dedicated PI-QUAL scoring sheet 3 in 41 scans obtained from different magnets (1.5T and 3T) and vendors (Philips, GE and Siemens).Inter-reader agreement was evaluated by using two methods: the percent agreement and Cohen’s kappa with standard quadratic weighting (κw). The percent agreement was defined as the total number of concordant readings divided by the total number of readings made. Cohen’s kappa coefficients were interpreted as follows: 0.01 - 020, slight agreement; 0.21 - 0.40, minimal agreement; 0.41 - 0.60, moderate agreement; 0.61 - 0.80, substantial agreement; 0.81- 0.90, strong agreement and > 0.90, almost perfect agreement. Statistical analyses were performed using SPSS (IBM, version 25).
Results
Thirty-two out of 41 scans (78%) were conducted on a 1.5T and 9/41 (22%) on a 3T system.Fourteen patients (34%) were examined on a Philips, 10/41 (24%) on a GE and 17/41(42%) on a Siemens scanner.
Reader 1 scored one scan (2%) as PI-QUAL 1, 7 scans (17%) as PI-QUAL 2, 13 (32%) as PI-QUAL 3, 19 (47%) as PI-QUAL 4 and 1 (2%) as PI-QUAL 5.
Reader 2 scored one scan (2%) as PI-QUAL 1, 9 scans (22%) as PI-QUAL 2, 16 (39%) as PI-QUAL 3, 14 (35%) as PI-QUAL 4 and 1 (2%) as PI-QUAL 5.
Fig. 1 shows three different cases scored with PI-QUAL.
Overall, the agreement for each single PI-QUAL score was substantial (weighted κ = 0.77 and percent agreement = 80%) (Table 2). We found that the percent agreement was even higher (90%) when the scans were grouped according to their ability to rule in prostate cancer (i.e. PI-QUAL 1-2 vs PI-QUAL ≥ 3) (Table 2).
The agreement between readers in terms of diagnostic quality for each single sequence was highest for T2-weighted imaging (37/41 scans; 90%), followed by dynamic contrast enhanced (DCE) sequences (36/41; 88%) and diffusion-weighted imaging (DWI) (34/41; 83%).
Discussion
High-quality prostate MRI is a hot topic at the moment, especially when it comes to MRI-derived targeted biopsies that are used to detect clinically significant prostate cancer, since a negative scan should be used to avoid unnecessary immediate biopsy.However, in order to be able to safely rule in and rule out prostate cancer, images with good spatial resolution and high signal-to-noise ratio for each MR sequence are needed. There can be significant variability in the acquisition of prostate mpMRI due to the number of different MR systems available and to the different levels of compliance to the PI-RADS v. 2.1 technical requirements across centres.
PI-QUAL represents a first step towards the standardization of a scoring system to assess the quality of prostate mpMRI prior to radiologist’s interpretation and allows clinicians to have more confidence in using the scan to determine patient care. We observed substantial reproducibility in the assessment of PI-QUAL between two experts. However, our study showed that some of the current technical standards set out in PI-RADS v. 2.1 guidelines could warrant reconsideration. For example, there is no mention of contrast resolution as far as DCE is concerned, and visual assessment is often used as the primary method to assess the quality on DWI.
These and other aspects should be addressed in the future versions of the PI-RADS guidelines.
Conclusion
For the time being, PI-QUAL offers clinicians the only available scoring system for evaluating and reporting the quality of their prostate mpMRI scans but we anticipate that the composition of the scoring system will need to undergo further refinements.Acknowledgements
Francesco Giganti is funded by the UCL Graduate Research Scholarship and the Brahm PhD scholarship in memory of Chris Adams.
NeuroSAFE PROOF has ethical approval (Regional Ethics Committee reference 17/LO/1978). NeuroSAFE PROOF is supported by National Institute for Healthcare Research Research for Patient Benefit funding (NIHR reference PB-PG-1216-20013). Trial registration number NCT03317990.
References
1. Brizmohun Appayya M, Adshead J, Ahmed H, et al. National imple- mentation of multi-parametric MRI for prostate cancer detection— recommendations from a UK consensus meeting. BJU Int 2018;122:13–25.
2. De Rooij M, Israël B, Tummers M, et al. ESUR/ESUI consensus statements on multi-parametric MRI for the detection of clinically significant prostate cancer: quality requirements for image acquisition, interpretation and radiologists' training. Eur Radiol 2020; 30(10):5404-5416.
3. Giganti F, Allen C, Emberton M, Moore CM, Kasivisvanathan V. Prostate Imaging Quality (PI-QUAL): A New Quality Control Scoring System for Multiparametric Magnetic Resonance Imaging of the Prostate from the PRECISION trial. Eur Urol Oncol 2020; 3(5):615-619.
4. Kasivisvanathan V, Rannikko AS, Borghi M, et al. MRI-Targeted or Standard Biopsy for Prostate-Cancer Diagnosis. N Engl J Med. 2018; 378(19):1767-1777.
5. Dinneen E, Haider A, Allen C, et al. NeuroSAFE robot-assisted laparoscopic prostatectomy versus standard robot-assisted laparoscopic prostatectomy for men with localised prostate cancer (NeuroSAFE PROOF): protocol for a randomised controlled feasibility study. BMJ Open 2019;9(6):e028132.
Figures
Fig. 1: Three cases of PI-QUAL score.
From left to right for each row: axial T2-weighted, diffusion (b-values and ADC map) and dynamic-contrast enhanced imaging.
PI-QUAL 2: only one sequence is of acceptable diagnostic quality (T2-WI). It is not possible to rule in and to rule out all significant lesions.
PI-QUAL 3: two sequences taken together are of acceptable diagnostic quality. It is possible to rule in but not to rule out all significant lesions.
PI-QUAL 4: at least two sequences are independently of diagnostic quality, and it is possible to rule in and rule out all significant lesions.