Synopsis

Keywords: Transferable skills: Reproducible research

In my role as ISMRM President, I have formed a new ad-hoc committee on Standardized Measures and Benchmarks, led by Jim Pipe, PhD. Dr. Pipe has since put together a blue-ribbon panel of experts to define the scope, purview, and governance structure of a new permanent ISMRM committee that will enable members and study groups to develop technical performance metrics and standardized procedures that will serve as the basis for improved communication to enhance scientific process and translation.

ISMRM Annual Meeting Toronto 2023 Session: Standardized Measures & Benchmarks
7 June 2023, 3:45pm, Room 718A
Talk Title: Ad-Hoc ISMRM Committee for Benchmarks
Scott B. Reeder, MD, PhDsreeder@wisc.edu
Highlights:
* Standardized metrics and benchmarks, as well as standardized procedures to define those benchmarks, are critical to facilitate effective scientific and engineering communication, necessary to further scientific progress and reproducible research.
*Most industries rely on non-profit organization to provide guidance on such benchmarks and standardized procedures. Unfortunately, such benchmarks and standards are generally lacking in the field of magnetic resonance.
*An ad-hoc committee on standardized measures and benchmarks has been formed to create the governance structure for a permanent ISMRM committee. The purpose of this committee is to enable members of the ISMRM to create the standards and benchmarks necessary to enhance communication and scientific progress in the field of magnetic resonance.

Have you ever bought a new automobile or new refrigerator? Did you perform a head-to-head comparison of technical specifications such as gas mileage, electricity consumption, horsepower, or cooling power? We take it for granted that such comparisons are possible using standardized metrics and benchmarks provided by independent organizations such as the International Organization for Standardization “ISO” (iso.org). Behind these measures are also standardized procedures that manufacturers agree to follow to create uniform benchmarks in their industry.

Unfortunately, given the continued rapid evolution of MR technology, standardized measures and benchmarks are generally lacking in our field. It becomes difficult for to compare the technical performance of MR systems, such as gradient performance and magnet homogeneity. When scientific papers report spatial resolution of a new imaging method, do we know what spatial resolution really means?
This lack of a common understanding of performance metrics is detrimental to scientific progress and reproducible research, because a lack of common measures, standards and benchmarks leads to poor communication and an inability to compare “apples to apples”. This also has enormous implications for the effective translation of innovation into clinical care, especially quantitative imaging.
As we move into an era of machine learning-based image reconstruction and increasingly non-linear systems, how will we objectively assess fundamental concepts such as image quality and diagnostic confidence? How do we incorporate what a radiologist might see in an image into a standardized metric that facilitates comparison of new technologies or provides robust quality assurance of clinical MR systems?
These are difficult questions to answer, but our society with its remarkable depth of talent, expertise, and decades of experience, is better suited than any other organization in the world to address these challenges. What we need now are governance mechanisms and processes that harness the creativity, energy, and innovation of our members to develop and codify technical performance standards, metrics, and procedures. Only in this way will we be able to speak the same language and work together to further the advancement of science and clinical dissemination of magnetic resonance.
For these reasons, in my role as ISMRM President, I have formed a new ad-hoc committee on Standardized Measures and Benchmarks, led by Jim Pipe, PhD. Dr. Pipe has since put together a blue-ribbon panel of experts to define the scope, purview, and governance structure of a new permanent ISMRM committee that will enable members and study groups to develop technical performance metrics and standardized procedures that will serve as the basis for improved communication to enhance scientific process and translation.
I look forward to our discussions in the session “Standardized Measures & Benchmarks” to kickstart this important initiative aimed at furthering our mission of bringing MR to the world.

Acknowledgements

No acknowledgement found.

References

No reference found.