0455

Transformer-Based Automatic Pipeline for 3D Needle Localization on Intra-Procedural 3D MRI

Wenqi Zhou¹, Xinzhou Li¹, Fatemeh Zabihollahy¹, David S. Lu¹, and Holden H. Wu¹
¹Department of Radiological Sciences, UCLA, Los Angeles, CA, United States

Synopsis

Keywords: MR-Guided Interventions, Segmentation

Motivation: Needle localization on 3D magnetic resonance imaging (MRI) is critical for MRI-guided percutaneous interventions, but current manual methods are time-consuming.

Goal(s): To develop a transformer-based pipeline for accurate and rapid automatic 3D needle localization on intra-procedural 3D MRI.

Approach: The proposed pipeline adopted a coarse-to-fine segmentation strategy by combining 3D and 2D shifted window (Swin) Transformer networks. The performance was evaluated in pre-clinical pig datasets and compared with human-annotated references.

Results: Computation time was 6 sec/volume. The median 3D needle tip and axis localization errors were 1.48 mm (1.09 pixels) and 0.98°, which were comparable to human-level accuracy.

Impact: The automatic transformer-based pipeline developed in this work achieved rapid (~6s) and accurate pixel-level 3D needle localization on intra-procedural 3D MRI. This new pipeline has the potential to improve MRI-guided percutaneous interventions.

Introduction

Magnetic resonance imaging (MRI) has the advantage of providing excellent soft tissue contrast for guiding interventions. However, accurate and rapid 3D needle localization in intra-procedural MR images remains a major challenge^1,2. In current workflows, manual needle localization is performed, which is time-consuming and cumbersome^3,4. 2D deep learning networks for needle segmentation have achieved promising results^5–7, but require manual prescription of a 2D image plane containing the needle feature and do not directly provide 3D position information⁸. On the other hand, 3D deep learning networks^9,10 require large training datasets of intra-procedural 3D MR images, which may not be available for specific applications.

For the task of needle segmentation, which requires delineating a relatively small object in a large field-of-view (FOV), convolutional neural network (CNN)-based neural networks (e.g., UNets¹¹) may be suboptimal since the convolution operations lack the ability to efficiently capture global information. To better model the long-range information in large FOVs, researchers have developed transformer-based networks that adopt the self-attention mechanism to capture global interactions between contexts¹².

In this work, we developed an automatic pipeline for rapid 3D needle localization by taking advantage of transformer networks. To overcome the restriction of limited 3D datasets for training, we combined the 3D Swin UNEt Transformer (UNETR) and 2D Swin Transformer for coarse-to-fine segmentation, and adopted pre-training and data augmentation strategies. We evaluated the proposed pipeline using pre-clinical intra-procedural 3D MRI datasets.

Methods

Datasets: In an animal research committee-approved study, we performed MRI-guided needle interventions in the livers of seven healthy female pigs at 3T (MAGNETOM Prisma, Siemens). 3D T1-weighted (T1w) GRE Dixon MR images containing the needle feature and 2D real-time radial GRE images aligned with the needle axis were collected (parameters in Figure 1). The segmentation mask and needle tip and entry point were annotated by a trained researcher supervised by a radiologist, with a wash out period of two weeks to assess intra-reader variation.

Automatic 3D Needle Localization Pipeline: The proposed pipeline (Figure 2) had three main steps: (1) initial 3D needle feature segmentation with 3D Swin UNETR; (2) generation of a 2D reformatted image containing the needle feature; (3) fine 2D needle feature segmentation with 2D Swin Transformer and 3D needle localization based on the 3D location of the 2D slice.

Transformer Network Training: For the 3D Swin UNETR, we adopted model weights pre-trained on publicly available CT images¹³ and fine-tuned it with 3D T1w GRE images. The 2D Swin Transformer was first pre-trained using the 2D radial GRE images and then fine-tuned using the 2D reformatted image generated in Step 2 of the pipeline. Data augmentation was performed.

Evaluation Metrics: Seven-fold cross-validation was performed with seven 3D images for validation in each fold. The needle segmentation performance was evaluated by Dice score and compared with 2D and 3D UNets. The needle localization performance was evaluated in terms of needle tip and axis localization error. The needle localization results were compared with a pipeline without the 2D Swin Transformer network and human intra-reader variation.

Results

The average inference time on one NVIDIA RTX A6000 GPU was 2.14 sec/volume for 3D Swin UNETR and 0.011 sec/slice for 2D Swin Transformer. The transformer-based networks achieved higher Dice scores compared with 2D and 3D UNets (Figure 3). The end-to-end computational time of the pipeline was ~6 seconds (examples in Figure 4). Figure 5 shows the 3D needle tip localization error (median of 1.48 mm, 1.09 pixels) and axis localization error (median of 0.98°) . The needle tip localization error of the pipeline was significantly smaller than that of the pipeline without the 2D network and human intra-reader variation.

Discussion

By applying pre-training and data augmentation, we were able to train the 3D and 2D transformer networks with a limited dataset. Both transformer networks outperformed the UNets for needle segmentation with their global information-capturing ability. After initial needle segmentation using 3D Swin UNETR, the pipeline automatically performed image plane realignment and compensated for the over- or under-segmentation in initial 3D segmentations by fine needle segmentation using 2D Swin Transformer. Our proposed pipeline achieved rapid (~6s) 3D needle localization with accuracy sufficient for liver interventions where relevant lesions have a diameter of at least 5-10 mm¹⁴.

Conclusion

We developed an automatic 3D needle localization pipeline that combined 3D and 2D transformer-based networks. The pipeline achieved rapid and accurate 3D needle localization within the range of expert human performance, and thus has potential to improve MRI-guided percutaneous interventions.

Acknowledgements

This work was supported in part by the NIH/NIBIB (R01 EB031934), the Department of Radiological Sciences at UCLA, and Siemens Medical Solutions USA.

References

1. Raj SD, Agrons MM, Woodtichartpreecha P, et al. MRI-guided needle localization: Indications, tips, tricks, and review of the literature. The Breast Journal. 2019;25(3):479-483. doi:10.1111/tbj.13246

2. DeAngelis GA, Moran RE, Fajardo LL, Mugler JP, Christopher JM, Harvey JA. MRI-guided needle localization: Technique. Seminars in Ultrasound, CT and MRI. 2000;21(5):337-350. doi:10.1016/S0887-2171(00)90028-3

3. Meinhold W, Martinez DE, Oshinski J, Hu AP, Ueda J. A direct drive parallel plane piezoelectric needle positioning robot for MRI guided intraspinal injection. IEEE Transactions on Biomedical Engineering. 2020;68(3):807-814.

4. Morris EA, Liberman L, Dershaw DD, et al. Preoperative MR Imaging—Guided Needle Localization of Breast Lesions. American Journal of Roentgenology. 2002;178(5):1211-1220. doi:10.2214/ajr.178.5.1781211

5. Li X, Young AS, Raman SS, et al. Automatic needle tracking using Mask R-CNN for MRI-guided percutaneous interventions. Int J CARS. 2020;15(10):1673-1684. doi:10.1007/s11548-020-02226-8

6. Li X, Lee YH, Lu DS, Tsao TC, Wu HH. Physics-Driven Mask R-CNN for Physical Needle Localization in MRI-Guided Percutaneous Interventions. IEEE Access. 2021;9:161055-161068. doi:10.1109/ACCESS.2021.3128163

7. Weine J, Rothgang E, Wacker F, Weiss CR, Maier F. Passive Needle Tracking with Deep Convolutional Neural Nets for MR-Guided Percutaneous Interventions. In: Proceedings of 12th Interventional MRI Symposium Oct. Vol 12. ; 2018:53.

8. Lee EJ, Farzinfard S, Yarmolenko P, Cleary K, Monfaredi R. Toward Robust Partial-Image Based Template Matching Techniques for MRI-Guided Interventions. J Digit Imaging. 2023;36(1):153-163. doi:10.1007/s10278-022-00716-6

9. Mehrtash A, Ghafoorian M, Pernelle G, et al. Automatic Needle Segmentation and Localization in MRI With 3-D Convolutional Neural Networks: Application to MRI-Targeted Prostate Biopsy. IEEE Transactions on Medical Imaging. 2019;38(4):1026-1036. doi:10.1109/TMI.2018.2876796

10. Weine J, Breton E, Garnon J, Gangi A, Maier F. Deep learning based needle localization on real-time MR images of patients acquired during MR-guided percutaneous interventions. In: Proceedings of the 27th Annual Meeting of ISMRM, Montreal, Canada. ; 2019:973.

11. Krithika alias AnbuDevi M, Suganthi K. Review of Semantic Segmentation of Medical Images Using Modified Architectures of UNET. Diagnostics. 2022;12(12):3064. doi:10.3390/diagnostics12123064

12. Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Published online June 3, 2021. Accessed July 10, 2023. http://arxiv.org/abs/2010.11929

13. Tang Y, Yang D, Li W, et al. Self-supervised pre-training of swin transformers for 3d medical image analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. ; 2022:20730-20740.

14. Moche M, Heinig S, Garnov N, et al. Navigated MRI-guided liver biopsies in a closed-bore scanner: experience in 52 patients. Eur Radiol. 2016;26(8):2462-2470. doi:10.1007/s00330-015-4097-1

Figures

Figure 1. MRI datasets and imaging parameters. TR: repetition time. TE: echo time. OP: out of phase. IP: in phase. FOV: field-of-view. N/A: not applicable. 15-fold data augmentation was performed to improve network training. The 3D needle tip and axis references were annotated on the 3D T1w GRE images by marking the 3D coordinates of the needle tip and needle entry point. The 3D needle tip and axis annotation process were performed twice with a washout period of two weeks in between to assess the human intra-reader variation.

Figure 2. Diagram of the proposed automatic pipeline for 3D needle localization. The 3D MR images were used for training and testing 3D Swin UNETR and evaluating the needle localization performance of the pipeline. The 2D GRE images were used for pre-training the 2D Swin Transformer. The 2D Swin Transformer network was further fine-tuned using the 2D reformatted image generated in Step 2 of the pipeline.

Figure 3. Needle feature segmentation Dice scores from cross-validation (49 sets of 3D MRI). (a) Violin plots of the Dice scores for 3D needle feature segmentation using 3D UNet and 3D Swin UNETR. (b) Violin plots of the Dice scores for 2D needle feature segmentation using 2D UNet and 2D Swin Transformer. The numbers shown on the violin plots are the medians of the Dice scores. In the pair-wise comparisons, p-values of the Wilcoxon signed rank test are shown on the connecting lines. * indicates p<0.05.

Figure 4. Example outputs from the pipeline. (a) Shallow insertion depth around 20 mm. (b) Moderate insertion depth around 60 mm. (c) Deeper insertion depth around 90 mm. 3D needle feature segmentation: 3D needle feature segmentation shown with the 2D reformatted image plane in 3D Slicer. 2D needle feature segmentation: 2D needle feature segmentation shown on the 2D reformatted image. 3D needle localization results: Predicted (blue) and reference (yellow) needle tip and axis in 3D space. The needle tip error and needle axis error are reported for each example.

Figure 5. 3D needle tip (a) and axis (b) localization results from cross-validation (49 sets of 3D MRI). The numbers shown on the violin plots are the medians of the results. The Kruskal-Wallis test was performed across groups first, and if showed a significant difference (p<0.05), then pair-wise comparisons between groups were performed using the Wilcoxon signed rank test. In the pair-wise comparisons, multiple comparisons were accounted for using Bonferroni correction. The p-values of the pair-wise comparisons are reported on the connecting lines. * indicates adjusted p<0.05.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)

0455

DOI: https://doi.org/10.58530/2024/0455