Synopsis

In this talk, we'll discuss the design of repeatability and reproducibility studies in home-built phantoms, standard phantoms and in vivo on our quest for ground truth.

Target audience

• Anyone who wants to implement a quantitative method, should understand how to evaluate the accuracy & repeatability
• Anyone who wants to see their technique get into the clinic, needs to know how to do the accuracy & repeatability studies

Outcome/objectives

• Understand accuracy, precision, repeatability, reproducibility, test-retest
• Design repeatability and reproducibility studies
• Select a phantom to help seek ground truth

Purpose

Recent technological advances have accelerated the acquisition times and potentially the reconstruction times for quantitative MRI. However, with every new method, we need to understand how it was or how it should be validated, and this is our 'quest for ground truth'. In addition, the repeatability and reproducibility of the method need to be tested. Without those studies, we cannot depend on the method to be the same each time it is used on one scanner nor across scanners.

Similarly, several MR biological markers (or “biomarkers”) have been proposed to improve clinical diagnostics. The pharmaceutical industry is also interested in their use for the development of novel therapeutic agents. Biomarkers [1,2] are objectively measured parameters that indicate biological state, biological/pathobiological processes or pharmacologic responses to treatment.

Methods

While quantitative mapping of biomarkers can greatly increase the amount, reliability, and comparability of the data obtained from medical imaging, it requires careful standardization of protocols and the development of phantoms (standard physical or digital reference objects or calibration structures) [3] to validate the accuracy of these in vivo measurements, as well as to assess the repeatability and reproducibility of the measurements across imaging and analysis platforms and time.

The Radiological Society of North America Quantitative Imaging Biomarkers Alliance (RSNA QIBA) has developed several guidelines for designing and analyzing repeatability and reproducibility studies [4-7]. NIST, in collaboration with ISMRM, RSNA-QIBA and NIH, and other groups have developed phantoms that are useful in the quest for ground truth [8-11]. In other cases, a well characterized home-built phantom is appropriate. The caveat here is the characterization methods.

Discussion

Interest in biomarkers has stimulated a large number of new standards efforts to ensure accuracy and consistency across platforms. ISMRM, RSNA-QIBA [12], DoD, VA, NEMA [13], MITA [13], ACR, AAPM, PhRMA, NIH, FDA and others are interested in actively developing standards to ensure image-based biomarkers can be used reliably.

The potential impact of quantitative MRI should not be understated. Based on some preliminary studies, biomarkers could be used for characterization of cancerous tissue, potentially obviating the need for biopsies. For example, ADC has been demonstrated for staging of tumors in the breast [14,16]; however, we cannot yet give guidance for tumor staging that is applicable at all clinical sites [16].

In this talk, we'll discuss the design of repeatability and reproducibility studies in home-built phantoms, standard phantoms and in vivo on our quest for ground truth.

Acknowledgements

The NIST Magnetic Imaging Group: https://www.nist.gov/pml/applied-physics-division/magnetic-imaging

References

1. Atkinson AJ, Colburn WA, DeGruttola VG, DeMets DL, Downing GJ, Hoth DF, Oates JA, Peck CC, Schooley RT, Spilker BA, Woodcock J, Zeger SL. Biomarkers and surrogate endpoints: Preferred definitions and conceptual framework. Clin Pharmacol Ther. 2001;69(3):89-95.
2. Kessler LG, Barnhart HX, Buckler AJ, Choudhury KR, Kondratovich MV, Toledano A, Guimaraes AR, Filice R, Zhang Z, Sullivan DC, QIBA Terminology Working Group. The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions. Statistical methods in medical research. 2014. doi: 10.1177/0962280214537333. PubMed PMID: 24919826
3. Keenan KE, Ainslie M, Barker AJ, Boss MA, Cecil KM, Charles C, Chenevert TL, Clarke L, Evelhoch JL, Finn P, Gembris D, Gunter JL, Hill DLG, Jack CR Jr, Jackson EF, Liu G, Russek SE, Sharma SD, Steckner M, Stupic KF, Trzasko JD, Yuan C, Zheng J. Quantitative magnetic resonance imaging phantoms: A review and the need for a system phantom. Magn Reson Med 2018;79(1):48-61.
4. Raunig DL, McShane LM, Pennello G, Gatsonis C, Carson PL, Voyvodic JT, Wahl RL, Kurland BF, Schwarz AJ, Gonen M, Zahlmann G, Kondratovich M, O’Donnell K, Petrick N, Cole PE, Garra B, Sullivan DC, QIBA Technical Performance Working Group. Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment. Statistical methods in medical research. 2014. doi: 10.1177/0962280214537344. PubMed PMID: 24919831.
5. Obuchowski et al. Quantitative imaging biomarkers: A review of statistical methods for computer algorithm comparisons. Stat Methods Med Res. 2014. doi: 10.1177/0962280214537390.
6. Sullivan et al. Metrology Standards for Quantitative Imaging Biomarkers. Radiology. 2015; 277(3):813-825
7. Huang et al. Meta-analysis of the technical performance of an imaging procedure: Guidelines and statistical methodology. Stat Methods Med Res. 2014. doi: 10.1177/0962280214537394.
8. Russek SE, Boss MA, Jackson EF, Jennings D, Evelhoch J, Gunter J, Sorensen A. Characterization of NIST/ISMRM MRI System Phantom. International Society for Magnetic Resonsance in Medicine; 2012; Melbourne, Australia.
9. Boss MA, Chenevert T, Waterton J, Morris D, Ragheb H, Jackson A, deSouza N, Collins D, van Beers B, Garteiser P, Doblas S, Persigehl T, Hedderich D, Martin A, Mukherjee P, Keenan KE, Russek SE, Jackson EF, Zahlmann G. Thermally-Stabilized Isotropic Diffusion Phantom for Multisite Assessment of Apparent Diffusion Coefficient Reproducibility. Medical Physics. 2014;41(6):464.
10. Pierpaoli C, et al. Polyvinylpyrrolidone (PVP) water solutions as isotropic phantoms for diffusion MRI studies. International Society for Magnetic Resonance in Medicine; 2009.
11. Keenan KE, WIlmes LJ, Aliu SO, Newitt DC, Jones EF, Boss MA, Stupic KF, Russek SE, Hylton NM. Design of a breast phantom for quantitative MRI. JMRI. 2016;44(3):610-9.
12. RSNA-QIBA Profiles http://qibawiki.rsna.org/index.php/Profiles
13. NEMA/MITA/DICOM http://dicom.nema.org/
14. Partridge SC, et al. doi: 10.2214/AJR.09.3534.
15. Partridge SC, et al. doi: 10.1002/jmri.22078.
16. DeSouza. doi: 10.1148/radiol.2018181717.