0972

Real-Time Needle Detection and Segmentation using Mask R-CNN for MRI-Guided Interventions.
Xinzhou Li1,2, Steven S. Raman1, David Lu1, Yu-Hsiu Lee3, Tsu-Chin Tsao3, and Holden H. Wu1

1Radiological Sciences, University of California, Los Angeles, Los Angeles, CA, United States, 2Bioengineering, University of California, Los Angeles, Los Angeles, CA, United States, 3Mechanical and Aerospace Engineering, University of California, Los Angeles, Los Angeles, CA, United States

Synopsis

Real-time needle tracking for MRI-guided interventions is challenging due to variations in the needle features and the contrast between the needle and surrounding tissue. Mask region-based convolutional neural network (R-CNN) is a powerful deep-learning technique for object detection and segmentation in natural images, which has the potential to overcome these challenges. In this study, we train the Mask R-CNN model using annotated intra-procedural images from MRI-guided prostate biopsy cases and real-time images from MRI-guided needle insertion in a phantom. Mask R-CNN achieved accurate needle detection and segmentation in real time (~80 ms/image), which has the potential to improve MRI-guided interventions.

Introduction

Image-guided needle-based interventions play key roles in cancer diagnosis and treatment1,2. Accurate knowledge of the needle position is crucial for guidance, such as for in-bore MRI-guided prostate biopsy3-5. New methods for closed-loop needle manipulation also require real-time needle tracking6. Conventional passive needle tracking using the MRI signal void feature is challenged by feature variations due to different sequence types and parameters, image quality, and needle orientation relative to the B0 field. In addition, the contrast between the needle feature and surrounding tissue may vary. In natural images, real-time object detection and segmentation have been greatly improved by the Mask Region-based Convolutional Neural Network (R-CNN)7,8. Therefore, we propose to implement and evaluate Mask R-CNN for needle detection and segmentation and adapt the results for real-time needle tracking in MRI-guided interventions.

Methods

Mask R-CNN extracts features from the input image in backbone architecture and proceeds to region of interest (ROI) classification, bounding box localization and mask prediction in a head architecture (Fig. 1). We implemented Mask R-CNN using Keras and Tensorflow9. The needle was defined as the only non-background class. The ROI localization corresponds to needle detection, while the mask result corresponds to needle feature segmentation. We trained Mask R-CNN and investigated its performance in intra-procedural prostate biopsy and real-time phantom MR images.

The network was first trained with T2-weighted turbo spin-echo (TSE) MRI from prostate biopsy (Table 1a). 85 clinical cases with 250 images in the transversal and sagittal planes formed a training/validation (200/50) dataset. The needle features in the images were manually annotated by a researcher. Data augmentation including rotation, flipping, translation and rescaling enlarged the dataset to 3200/800 images. The equally weighted loss for the validation dataset were evaluated to tune hyperparameters (Table 1b). Both training and detection were implemented using 2 NVIDIA GTX 1080Ti (11G-P4-6696-KR) cards. The needle detection and segmentation accuracy were assessed using a separate testing set of 15 clinical cases with 80 images. Intersection over Union (IoU) and center of mass (CoM) shift (mm) were calculated with respect to manually annotated ground truth. Average precision (AP) metrics10 were calculated to evaluate the performance.

Next, Mask R-CNN was fine-tuned with real-time golden-angle (GA) radial MRI phantom data (Table 1a). The same hyperparameters were applied (Table 1b). Dynamic needle insertion and retraction was performed using manually-controlled actuators11. 145 images with 11 needle orientations (0º/±5º/±10º/±15º/±20º/±25º) were augmented as previously described to form the training dataset (2320 images). Mask R-CNN was implemented in an online pipeline and evaluated for two real-time MRI datasets with the needle continuously moving along 1 degree of freedom (1-DOF) (62 sec, 646 images) and the needle moving along multiple-DOF with different orientations (78.4 sec, 815 images). 48 images from each dataset were manually annotated to form 2 testing sets for segmentation evaluation. In the 1-DOF dataset, the needle-tip displacements were also calculated based on the mask boundary and compared with a previously calibrated template-matching method12.

Results

Needle detection was 100% successful in the prostate MRI testing set and 1-DOF phantom MRI dataset. In the multiple-DOF dataset, needle detection was successful in 809/815 (99.3%) images. This included the successful identification of frames with no needle present. Example needle segmentation results are shown in Figure 2. Every median IoU was >0.7 and median CoM shift was 0.59-1.29mm (around 1-pixel). Processing time was ~80ms/image. The complete set of performance metrics are reported in Table 2. Real-time tracking of a needle during continuous 1-DOF and multiple-DOF motion are shown in Figure 3. The mean absolute difference between Mask R-CNN results and the template-matching reference was <1mm for 1-DOF needle motion.

Discussion

The needle features were detected and segmented accurately in real time using Mask R-CNN. After training, the entire process using Mask R-CNN was automated and only requires an input image. Needle detection in different backgrounds (e.g., prostate, phantom) for different sequences (e.g., TSE, GRE) required a small fine-tuning dataset. Mask R-CNN can accommodate different needle orientations and identify presence/absence of the needle (Fig. 3b). Mask R-CNN tracking results may provide intra-procedural information to improve MRI-guided prostate biopsy. Real-time needle tracking in the phantom had comparable accuracy with reference methods. In addition, Mask R-CNN has potential to localize multiple-DOF needle motion by handling geometric feature changes. The performance of Mask R-CNN may also be improved by incorporating active device features13 or positive-contrast passive features14.

Conclusion

In this study, Mask R-CNN was implemented and trained for real-time needle detection and segmentation on both intra-procedural and real-time interventional MRI. Mask R-CNN was incorporated into an online software pipeline and achieved accurate real-time needle tracking.

Acknowledgements

We acknowledge the funding from the Department of Radiological Sciences at UCLA and Siemens Healthineers.

References

1. Gupta S, Madoff DC. Image-Guided Percutaneous Needle Biopsy in Cancer Diagnosis and Staging. Tech. Vasc. Interv. Radiol. 2007; 10:88–101.

2. Ahmed M, Solbiati L, Brace CL, et al. Image-guided Tumor Ablation: Standardization of Terminology and Reporting Criteria—A 10-Year Update. Radiology 2014; 273:241–260.

3. Verma S, Choyke PL, Eberhardt SC, Oto A, Tempany CM, Turkbey B, Rosenkrantz AB. The Current State of MR Imaging–targeted Biopsy Techniques for Detection of Prostate Cancer. Radiology 2017; 285:343–356.

4. Moore CM, Robertson NL, Arsanious N, Middleton T, Villers A, Klotz L, Taneja SS, Emberton M. Image-Guided Prostate Biopsy Using Magnetic Resonance Imaging-Derived Targets: A Systematic Review. Eur. Urol. 2013; 63:125–140.

5. Tan N, Lin W-C, Khoshnoodi P, et al. In-Bore 3-T MR-guided Transrectal Targeted Prostate Biopsy: Prostate Imaging Reporting and Data System Version 2–based Diagnostic Performance for Detection of Prostate Cancer. Radiology 2016; 283:130–139.

6. Mikaiel S, Simonelli J, Li X, Lee Y, Lee YS, Sung K, Lu D, Tsao TC, Wu HH. Accuracy and Time Efficiency of Real-Time MRI-Guided Remote-Controlled Targeted Needle Placement During Motion Using Hydrostatic Actuators. Proceedings of the ISMRM 26th Annual Meeting, Paris, France, 2018, p4163.

7. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1. NIPS’15. Cambridge, MA, USA: MIT Press; 2015. pp. 91–99.

8. He K, Gkioxari G, Dollár P, Girshick RB. Mask R-CNN. 2017 IEEE Int. Conf. Comput. Vis. 2017:2980–2988.

9. Abdulla W. Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. GitHub Repos. 2017.

10. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL. Microsoft COCO: Common Objects in Context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, editors. Computer Vision -- ECCV 2014. Cham: Springer International Publishing; 2014. pp. 740–755.

11. Mikaiel S, Simonelli J, Lee Y, Li X, Lee YS, Lu D, Sung K, Tsao TC, and Wu HH. Real-Time MRI-Guided Targeted Needle Placement During Motion using Rolling Diaphragm Hydrostatic Actuators. Proceedings of the ISMRM 25th Annual Meeting, Honolulu, 2017, p736.

12. Li X, Mikaiel S, Simonelli J, Lee Y, Tsao TC, and Wu HH. Real-Time Motion Prediction for Feedback Control of MRI-Guided Interventions. Proceedings of the ISMRM 25th Annual Meeting, Honolulu, 2017, p5540.

13. Hillenbrand CM, Elgort DR, Wong EY, Reykowski A, Wacker FK, Lewin JS, Duerk JL. Active device tracking and high-resolution intravascular MRI using a novel catheter-based, opposed-solenoid phased array coil. Magn. Reson. Med. 2004; 51:668–675.

14. Seppenwoolde JH, Viergever MA, Bakker CJG. Passive tracking exploiting local signal conservation: The white marker phenomenon. Magn. Reson. Med. 2003; 50:784–790.

15. Sørensen TS, Atkinson D, Schaeffter T, Hansen MS. Real-time reconstruction of sensitivity encoded radial magnetic resonance imaging using a graphics processing unit. IEEE Trans. Med. Imaging 2009; 28:1974–1985.

Figures

Figure 1: Structure of Mask R-CNN for needle detection and segmentation. In the backbone, ResNet101 and Feature Pyramid Networks (FPN) extract features from the entire image. The feature maps are shared with a region proposal network (RPN) to generate region of interest (ROI) bounding boxes. In the head architecture, a single pyramidal feature map is processed by the ROIAlign layer to generate features corresponding to each ROI. The input gray-scale image is stacked into 3 color channels. For training using stochastic gradient descent, the loss of the RPN and Mask-RCNN results are computed using binary cross entropy.

Table 1: (a) MR imaging parameters. Golden-angle radial images were reconstructed using non-Cartesian conjugate gradient SENSE15. TSE: turbo spin-echo. GRE: gradient echo. (b) Mask R-CNN training hyperparameters. The network training was initialized from pre-trained weights of the Common Objects in Context (COCO) dataset10, since the MRI training dataset is relatively small.

Figure 2: Example needle detection and segmentation results using Mask R-CNN in both intra-procedural prostate MRI and real-time phantom MRI datasets. The needle detection scores are reported in the middle and right columns. The processing overhead time is also reported. IoU: intersection over union. CoM: center of mass.

Table 2: Needle detection and segmentation performance using Mask R-CNN in intra-procedural prostate MRI testing set (80 images), phantom testing set 1 with a continuous needle insertion and retraction in 1 degree of freedom (DOF) (48 images) and phantom testing set 2 with multiple-DOF needle motion (48 images). Evaluation metrics included: median, 30th percentile (Q30), 60th percentile (Q60) of Intersection over Union (IoU); center of mass (CoM) shift of the segmentation mask; average precision (AP) at different IoU levels (APIoU) and mean AP (mAP).

Figure 3: Example online Mask R-CNN needle segmentation results. (a) Real-time MRI of 1 degree of freedom (DOF) needle motion (104 images). The original image is shown on the left and the Mask R-CNN results are shown on the right. (b) Real-time MRI of multiple-DOF needle motion (208 images). Video is played back with 2x speed.


Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)
0972