Synopsis

In this work we use the Microsoft Hololens for holographic visualization of brain MR imaging data aligned to the real world human body. This provides a way to directly “look inside” the subject’s head instead of treating image and subject as two separate entities.

Purpose: While the MR volumes and images show the inside of the human body, visualization and evaluation of these images is done separately on flat displays. This separation between the display and the human body can make it difficult for students to learn human anatomy and for medical professionals to simultaneously evaluate interior and exterior features and how they relate to each other. The Microsoft HoloLens (Microsoft, Redmond WA, USA) is an augmented reality headset that renders arbitrary objects within the user’s field of vision as “holograms”. The holograms are normally fixed to real-world coordinates by a combination of room-tracking cameras and “holographic processing” built into the device. Here, we use the Microsoft HoloLens for holographic visualization of brain MR imaging data aligned to a real world human body to provide a way to directly look inside the subject’s head instead of treating image and subject as two separate entities.

Methods: Magnetic resonance images of a human head were acquired using a spoiled gradient echo brain volume imaging (SPGR BRAVO) sequence with TE/TR 3.5/8.8 ms and 500um isotropic resolution. The data was visualized and registered to the head using in-house software developed using Unity 3D (unity3D.com). The dataset was downsampled to a 256x256x256 matrix size at 1mm isotropic resolution to speed up the holographic rendering. The shader for the hologram adapted the opacity of the voxels based on their signal intensity such that dark voxels (eg. background) appear transparent in the holographic display. Figures 1 and 2 show a schematic of the experimental setup and a photograph of the same scene. Informed consent was given by the subject for presentation of the acquired data. The subject has an image texture-based fiducial marker attached to his head that is used to align the MRI hologram that can be seen by the HoloLens to the real world subject. Alignment is done using a feature detection and tracking technique that detects feature points on the marker and compares these to features received by the camera feed.

Results: Figures 3-5 are animated GIF images that show the hologram registered in real time to the head of the subject. Movement of the HoloLens allows viewing of the hologram from different sides as long as the marker attached to the subject’s forehead is still visible to the HoloLens camera. Movement of the HoloLens with respect to the subject allows the user to slice through the MRI data and get a view inside the brain.

Discussion & Conclusion: The Microsoft HoloLens was successfully used for holographic visualization of brain MR imaging data aligned to a real world human body. Future work will aim to increase the frame rate and improve the alignment for more accurate visualizations from all angles from the marker. The HoloLens itself also still has some issues that complicate MRI alignment to real world subjects. It has a very limited field of view and the camera clipping plane has to be placed more than 0.5m away from the camera to avoid image artifacts, making the hologram disappear if moving too close to the subject. Nevertheless, with further advancements of holographic hardware and software, the holographic visualization of medical imaging data directly on top of the subject can be a natural way to better understand human anatomy and possibly help with the planning and execution of certain medical interventions.

Acknowledgements

References