5545

MR-Guided Mixed-Reality For Surgical Planning: Set-Up and Perceptual Accuracy
Subashini Srinivasan1, Amanda Wheeler2, Brian Hargreaves1, and Bruce Daniel1

1Department of Radiological Sciences, Stanford University, Palo Alto, CA, United States, 2Department of Surgery, Stanford University, Palo Alto, CA, United States

Synopsis

Microsoft HoloLens provides the ability to visualize 3D holograms of preoperative MRI in addition to the physical environment. In this work we have developed a HoloLens application that aligns these preoperative holograms to the patient. The accuracy of perceiving these holograms was evaluated by presenting different shapes of holograms in random locations and comparing their positions to ground truth. The current set-up enables visualization and perception of holograms with a margin tolerance of < 6 mm.

Introduction

Three-dimensional high resolution MRI enables visualization of patient anatomy with sufficient tissue contrast. However, conventional preoperative MRI and even 3D rendered MRI are presented to the surgeon on a stand-alone monitor that is separate from the patient, and requires the surgeon to mentally translate findings to the patient when planning surgery. The ability to visualize 3D MRI overlaid onto the patient would enable surgeons to better visualize the anatomy and potentially improve surgical planning. In this work, we have: (i) developed a mixed-reality HoloLens application to visualize the ‘hologram’ of MRI registered to the patient; and (ii) evaluated the perceptual accuracy of the ‘holograms’ rendered by the HoloLens.

Methods

Microsoft HoloLens1 enables mixed-reality by projecting ‘holograms’ into the visual field. This will enable a surgeon to view the patient as well as ‘holograms’ of pre-operative or diagnostic MRI projected onto the patient. The ‘holograms’ can be registered and locked to the patient using fiducials that are both MR and optically visible, enabling the surgeon to move around the patient for 3D visualization of the MRI from different angles.

HoloLens Application Functionalities: An initial application was developed in HoloLens and tested on a torso phantom. Three-dimensional, spoiled gradient-echo MR images of a torso phantom were acquired with 5 gadolinium filled optical fiducials as shown in Figure 2. The center positions of the markers in the 3D MRI were measured using Osirix2. A contrast-enhanced 3D breast MRI dataset3 was warped to fit the torso phantom4. Skin, chest and a breast tumor were segmented semi-automatically5 and the corresponding surface meshes were created. The ‘holograms’ of the MRI slices can be viewed along with the meshes in different viewing planes (sagittal, coronal, transverse and cut-away) using voice commands (Figure 2). Different slices can be scrolled over using the gesture controls of HoloLens. OpenCV6,7 was integrated in HoloLens and computer vision algorithms were used for automatic detection of the fiducials and registration of the holograms to the phantom. The ‘holograms’ can also be manually registered by either: (1) translating and/or rotating about a plane of the one or more fiducials; or (2) using the user’s head position to move the holograms.

Perceptual Accuracy: A graph paper sheet with five fiducials at the corners (Fig. 3a) was used to evaluate the 2D perceptual accuracy of holograms by 6 subjects. The HoloLens was initially set up to the measured inter-pupillary distance (IPD) of the subject. Later, each subject calibrated the holograms of the fiducials to the automatically detected fiducials in the graph paper. This user-specific calibration was essential to know the position of the HoloLens with respect to the eyes of the subject. The subjects were then presented with 1mm spherical holograms and were asked to draw each location in the graph paper. The error between the marked positions of 20 such spheres located in random positions, and the ground truth was measured. The subjects were also presented with 8 holographic shapes with longest dimension varying from 2 to 4.5 cm and their perceptual accuracy (Fig. 3b) was also analyzed.

Results and Discussion

The HoloLens application is able to register and lock the MRI to the rigid torso phantom using the MR-visible optical fiducials as shown in Figure 2. In the future, 3D perceptual accuracy of palpable tumors will be evaluated with contrast-enhanced supine MRI. Deformable registration of acquired supine MRI to the patient in surgical position will also be considered.

The errors between the marked spheres or shapes, and the ground truth are tabulated in Figure 3. The margin tolerance, the number of additional dilated pixels required on the drawn shapes to completely cover the ground truth shapes, ranged between 0.7 to 5.7 mm. The range of errors along the depth (horizontal) direction, was larger compared to the vertical direction. This could be because the perceptual accuracy of the holograms, in addition to the accurate alignment of the holograms to the object, depends on various factors including IPD, position of the HoloLens in the head. These additional parameters were adjusted during user-specific calibration. Further improvements in calibration are being considered to further improve perceptual accuracy.

Conclusion

A preliminary application in HoloLens has been created to visualize the holograms of three-dimensional MR slices aligned to a torso phantom. The application enables visualization and perception of holograms with a margin tolerance of < 6 mm, which should be improved further with more advanced user-specific calibration methods.

Acknowledgements

Stanford Cancer Imaging Training Program T32 CA 009695

Microsoft Corporation

California Breast Cancer Research Program IDEA award 22IB-0006

References

1. www.microsoft.com/microsoft-hololens/

2. http://www.osirix-viewer.com/

3. https://wiki.cancerimagingarchive.net/display/Public/QIN+Breast+DCE-MRI

4. www.slicer.org

5. Yushkevich PA et al., Neuroimage 2006;31(3):1116-28 (www.itksnap.org/)

6. http://opencv.org/

7. Garrido-Jurado et al., Pattern Recogn. 2014; 47(6):2280-2292

Figures

Figure 1: Three-dimensional colored dicom images with segmented skin (shown as mesh), chest and tumor (green mesh) are aligned to the torso phantom using the five optical fiducials. The cyan dots represent the centers of the Gadolinium filled fiducials placed at the identical locations as optical fiducials. The left image shows a single slice in coronal view and the right image shows a cut-away view of orthogonal MRI slices.

Figure 2: The graph paper set-up with example holograms of shapes and a 1 mm sphere (arrow), as seen through the HoloLens (a). Note that the holograms will be better aligned for a subject with correct IPD, than this photo captured with a HoloLens camera located at a different position than the eye due to parallax errors. An example overlay of ground truth shapes in magenta and white with the drawn shapes shown in green and white (b) show the misalignment perceived by the subject, which was measured to characterize the average error.

Figure 3: The perceived error between the holograms drawn on the graph paper, and the ground truth shapes and dots.

Proc. Intl. Soc. Mag. Reson. Med. 25 (2017)
5545