3028
VISTA: Visualization of Image Segmentation by Transformation and Analysis1Department of Bioengineering, University of Illinois Urbana-Champaign, Urbana, IL, United States, 2Beckman Institute, University of Illinois Urbana-Champaign, Urbana, IL, United States, 3OSF Healthcare, Peoria, IL, United States, 4Department of Pediatrics, University of Illinois College of Medicine, Peoria, IL, United States
Synopsis
Keywords: Visualization, Visualization, Virtual Reality, VR, Software
Motivation: Complex medical procedures often require clinicians to construct a (3D) three-dimensional mental model of a patient's anatomy from 2D medical imaging data.
Goal(s): Our goal was to develop a set of tools which convert 2D imaging data into 3D objects to view in virtual reality (VR).
Approach: Two pipelines were created, one for brain imaging data and another for label mask images, which automatically segment the images, convert them to objects, and merge them into a VR viewable model.
Results: Our software has been successfully used to transform a variety of medical imaging data into 3D files which are viewable on VR platforms.
Impact: Some of the challenges with mentally visualizing two-dimensional medical imaging data should be alleviated by using our software to automatically make the data viewable in a three-dimensional format.
Introduction
For complex medical procedures, constructing a three-dimensional (3D) mental model of the patient from two-dimensional views of medical imaging data is challenging, even for highly trained clinicians. Virtual reality (VR) is a tool that can significantly ease the mental load of constructing a better mental model of complicated patient anatomy1-4. However, making a VR model requires linking several tools together along with extracting 3D objects from imaging data. We have developed two separate pipelines: one which takes unsegmented brain magnetic resonance imaging (MRI) data and the other which takes pre-segmented MRI data to automatically generate VR ready models of patient specific data5. We present these tools in an open-source form to enable easy adoption of VR in medical image viewing.Methods
We developed two containerized pipelines, as seen in Figure 1, to take MRI data and produce VR ready models. The first pipeline takes a brain MRI and automatically segments out the grey matter, white matter, and cerebrospinal fluid. Then, these segmented images are converted into isotropic space by scaling to the highest resolution dimension. Imaging data is converted from a data array to a set of vertices, faces, and indices using the Lewiner marching cubes algorithm which then constructs object files6. The produced object files are automatically loaded into Blender and assigned unique colors. For viewing in the VR environment, the normal vectors are recalculated towards the outside of the model so that the models have texture. Recalculating normal vectors in this manner is critical for Unity based VR environments. Then, the Blender model is saved as an FBX file which can be uploaded to VR environments, such as Enduvo (https://enduvo.com/) or Sketchfab (https://sketchfab.com/).Pre-segmented images for the second pipeline need to be provided where each voxel in each distinct structure in the image has the same integer value (a label mask). For example, for a segmented MRI with two muscles, each muscle would be comprised of voxels such that the first muscle contains only voxels of value 1 and the second muscle contains only voxels of value 2. These segmented images splice out into their own files, are converted into isotropic space, and are converted to object files in the same manner as the first pipeline. These separate files are loaded into Blender, color coded, had normal vectors recalculated, and saved out as an FBX as before.
We have made the tool available on GitHub at: https://github.com/mrfil/VISTA
Results
We demonstrate these pipelines with two different imaging targets. A full brain was segmented from a T1 MPRAGE acquired at 3T using FAST in FSL7 and saved out as a VR ready model, shown in Figure 2, shows the different tissue types. We also took a segmented MRI of the structures involved in speech production from a dynamic MRI scan8, as shown in Figure 3.Discussion
Our pipeline has been deployed in our clinical partners’ environment and has been used to segment, load, and view images. Viewing anatomical information obtained from MRI in VR provides an additional degree of freedom and perspective on patient data. One of the challenges present with VR visualization of medical imaging data is the great difficulty in conveying stereoscopic 3D images in any 2D format. Without being able to view and interact with the 3D model, a sense of depth and texture is lost.Conclusion
We created a set of tools to automatically load in complex patient data and generate 3D models of 2D images which could better represent the underlying imaging data. We use these tools in our research on presurgical planning for epilepsy resection surgeries4. These open-source tools can be modified to enable animation of data; however certain kinds of animation are challenging to automatically generate and load into VR platforms.Acknowledgements
This project has been funded by the Jump ARCHES endowment through the Health Care Engineering Systems Center at the University of Illinois Urbana Champaign.References
1. Quero, G., et al. (2019). "Virtual and Augmented Reality in Oncologic Liver Surgery." Surg Oncol Clin N Am 28(1): 31-44.
2. Mattus, M. S., et al. (2022). "Creation of Patient-Specific Silicone Cardiac Models with Applications in Pre-surgical Plans and Hands-on Training." J Vis Exp(180).
3. Lyuksemburg, V., et al. (2023). "Virtual Reality for Preoperative Planning in Complex Surgical Oncology: A Single-Center Experience." J Surg Res 291: 546-556.
4. Wu, B., et al. (2010). "Visualizing 3D objects from 2D cross sectional images displayed in-situ versus ex-situ." J Exp Psychol Appl 16(1): 45-59.
5. Evans, J., et al. (2022). “Automated Visualization of Ictal and Interictal SEEG Activity to Improve Surgical Planning in Drug Resistant Epilepsy”. AES 2022 Annual Meeting Abstract Database. AESnet.org
6. Lewiner, T., et al. (2003). "Efficient Implementation of Marching Cubes' Cases with Topological Guarantees." Journal of Graphics Tools 8(2): 1-15.
7. Zhang, Y., et al. (2001). "Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm." IEEE Trans Med Imaging 20(1): 45-57.
8. Jin, R., et al. (2023). "Enhancing linguistic research through 2-mm isotropic 3D dynamic speech MRI optimized by sparse temporal sampling and low-rank reconstruction." Magn Reson Med 89(2): 652-664.
Figures
