2715
Multi-Class 3D Transformer-Based Segmentation of the Liver, Liver Vessels, and Interventional Needle on Intra-Procedural 3D MRI1Department of Radiological Sciences, UCLA, Los Angeles, CA, United States
Synopsis
Keywords: MR-Guided Interventions, Segmentation
Motivation: Localization of liver, liver vessels, and interventional needle on 3D magnetic resonance imaging (MRI) provides essential information for MR-guided interventions.
Goal(s): To develop a multi-class network for segmenting the three classes on intra-procedural 3D MRI.
Approach: 3D Swin UNEt Transformer (UNETR) with pre-trained model weights was trained with data augmentation. Needle localization was performed based on the predicted needle segmentation.
Results: In six-fold cross validation of 42 3D images, the multi-class model achieved median Dice scores of 0.87, 0.64, 0.76 for liver, liver vessels and needle. The needle tip localization showed improvements compared to a single-class 3D Swin UNETR model.
Impact: We trained the 3D Swin UNETR for 3D liver, liver vessel, and interventional needle segmentation on intra-procedural 3D MRI and showed that the needle localization performance can be improved using multi-class model compared to single-class model for needle localization.
Introduction
Magnetic resonance imaging (MRI) provides excellent soft-tissue contrast and is an emerging modality for guiding percutaneous interventions in abdominal organs such as the liver1. Identification of the liver, liver vessels, and needle in 3D is required for planning the needle trajectory with respect to the anatomy and target lesion, and for adjusting the needle to avoid vessels during insertion2,3. However, manual segmentation of the liver, vessels, and needle is time-consuming and prolongs the procedure time4.Automatic methods such as applying 3D neural networks have been studied for liver and liver vessel segmentation5–7, but previous models lacked the ability to simultaneously provide the 3D needle location information for guiding interventional procedures8. For 3D needle segmentation, single-class 3D models have been developed6. Due to the similarity between the appearance of interventional needle and liver vessels (both are often hypointense linear structures), providing multi-class segmentation labels of both the needle and vessels is expected to improve the needle localization performance compared with a single-class model. Moreover, the convolutional neural network (CNN)-based neural networks used in previous works may be suboptimal for vessel and needle segmentation due to the lack of global information capturing ability.
There are two objectives to this work: First, to take advantage of long-range information modeling ability of the 3D Swin UNEt Transformer (UNETR)9 and achieve 3D multi-class segmentation of the liver, liver vessels, and needle on MRI. Second, to assess the needle localization performance of the multi-class model comparing with a single-class model.
Methods
Dataset: In an animal research committee-approved study, we performed MRI-guided needle interventions in the liver in six healthy pigs at 3T (Prisma, Siemens). Forty-two 3D T1-weighted (T1w) GRE Dixon MR images (TR/TE1/TE2: 3.91/1.23/2.46 ms, field-of-view [FOV]: 346x237 mm2, in-plane resolution: 1.35x1.35 mm2, flip angle [FA]: 9°, axial slab, slice thickness: 1.5 mm, 120 slices, parallel imaging factor: 4, 13-sec breath-held scan) were used for training the models. The segmentation references were annotated in 3D Slicer10 by a trained researcher under the guidance of an interventional radiologist and clinical imaging analysts. The needle tip and axis references were annotated by marking the 3D coordinates on the 3D T1w-GRE images.3D Swin UNETR: We trained the 3D Swin UNETR9 (Figure 1) for multi-class segmentation. We adopted pre-trained weights from publicly available CT images11 and performed 15-fold data augmentation. For comparison, we trained a single-class 3D Swin UNETR model specifically for needle segmentation.
Evaluation Metrics: We calculated the 3D Dice scores of the three classes. Needle localization was performed by automatically identifying the main axis of the 3D segmentation as the needle axis and the intersection of the main axis and the surface of the 3D needle segmentation as the needle feature tip. We performed six-fold cross-validation (seven T1w-GRE images in each fold).
Results
The average inference time for 3D Swin UNETR was 2.56 s/volume. Example results are shown in Figures 2 and 3. The multi-class segmentaion Dice scores were shown in Figure 4. The median needle tip localization error was 1.65 mm (single-class: 1.94 mm), and median needle axis localization error was 1.05° (single-class: 0.98°). There was a significant difference in the needle tip localization error between the multi-class and single-class models (p=0.045) and no significant difference in needle axis localization error (p=0.23).Discussion
By applying pre-training weights and data augmentation, we trained the 3D Swin UNETR using a limited dataset and achieved multi-class segmentation of liver, liver vessel, and needle on intra-procedural MRI. The liver vessel segmentation Dice score was not ideal due to the complex structure especially in smaller branches. In future work, the performance of vessel segmentation can be analyzed in more detail by calculating the Dice score of liver vessels with a diameter >3 mm, as those are the most relevant vessels for interventional procedures12,13. The needle localization performance was improved by using the multi-class model compared to the single-class model. Although the two models had similar needle segmentation Dice scores, the single-class model often had over- or under segmentations around the needle tip. The multi-class segmentation labels provided more information about surrounding tissue and vessels around the needle tip, and therefore improved the needle tip localization performance.Conclusion
We trained the 3D Swin UNETR for 3D liver, liver vessel, and interventional needle segmentation on intra-procedural 3D MRI for interventional guidance. The needle localization performance using the multi-class model outperformed a single-class model.Acknowledgements
This work was supported in part by the NIH/NIBIB (R01 EB031934), the Department of Radiological Sciences at UCLA, and Siemens Medical Solutions USA.References
1. Kaye EA, Granlund KL, Morris EA, Maybody M, Solomon SB. Closed-Bore Interventional MRI: Percutaneous Biopsies and Ablations. American Journal of Roentgenology. 2015;205(4):W400-W410. doi:10.2214/AJR.15.14732
2. DeAngelis GA, Moran RE, Fajardo LL, Mugler JP, Christopher JM, Harvey JA. MRI-guided needle localization: Technique. Seminars in Ultrasound, CT and MRI. 2000;21(5):337-350. doi:10.1016/S0887-2171(00)90028-3
3. Alhonnoro T, Pollari M, Lilja M, et al. Vessel Segmentation for Ablation Treatment Planning and Simulation. In: Jiang T, Navab N, Pluim JPW, Viergever MA, eds. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2010. Lecture Notes in Computer Science. Springer; 2010:45-52. doi:10.1007/978-3-642-15705-9_6
4. Morris EA, Liberman L, Dershaw DD, et al. Preoperative MR Imaging—Guided Needle Localization of Breast Lesions. American Journal of Roentgenology. 2002;178(5):1211-1220. doi:10.2214/ajr.178.5.1781211
5. Ansari MY, Abdalla A, Ansari MY, et al. Practical utility of liver segmentation methods in clinical surgeries and interventions. BMC Medical Imaging. 2022;22(1):97. doi:10.1186/s12880-022-00825-2
6. Mehrtash A, Ghafoorian M, Pernelle G, et al. Automatic Needle Segmentation and Localization in MRI With 3-D Convolutional Neural Networks: Application to MRI-Targeted Prostate Biopsy. IEEE Transactions on Medical Imaging. 2019;38(4):1026-1036. doi:10.1109/TMI.2018.2876796
7. Ciecholewski M, Kassjański M. Computational Methods for Liver Vessel Segmentation in Medical Imaging: A Review. Sensors. 2021;21(6):2027. doi:10.3390/s21062027
8. Orlando N, Gyacskov I, Gillies DJ, et al. Effect of dataset size, image quality, and image type on deep learning-based automatic prostate segmentation in 3D ultrasound. Phys Med Biol. 2022;67(7):074002. doi:10.1088/1361-6560/ac5a93
9. Hatamizadeh A, Nath V, Tang Y, Yang D, Roth HR, Xu D. Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. In: Crimi A, Bakas S, eds. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Lecture Notes in Computer Science. Springer International Publishing; 2022:272-284. doi:10.1007/978-3-031-08999-2_22
10. Pieper S, Halle M, Kikinis R. 3D Slicer. In: 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821). ; 2004:632-635 Vol. 1. doi:10.1109/ISBI.2004.1398617
11. Tang Y, Yang D, Li W, et al. Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis. In: ; 2022:20730-20740. Accessed April 19, 2023. https://openaccess.thecvf.com/content/CVPR2022/html/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.html
12. Kumar RP. Fast Blood Vessel Segmentation for Surgical and Interventional Planning and Navigation. Doctoral thesis. 2014. Accessed November 2, 2023. https://www.duo.uio.no/handle/10852/41751
13. Lu DSK, Raman SS, Vodopich DJ, Wang M, Sayre J, Lassman C. Effect of Vessel Size on Creation of Hepatic Radiofrequency Lesions in Pigs. American Journal of Roentgenology. 2002;178(1):47-51. doi:10.2214/ajr.178.1.1780047
Figures
Figure 1: Overview of the 3D Swin UNETR architecture (W:256, H:256, D:128). Outputs of the network are segmentation labels for liver (red), liver vessels (blue), and needle (yellow). We adopted pre-trained weights from self-supervised learning on publicly available CT images. The 3D MRI datasets were expanded 15-fold for training through data augmentation by random rotation (0°–360°), horizontal flipping, vertical flipping, translation, zooming, and adding Gaussian noise. The model was trained on one NVIDIA RTX A6000 GPU card.
Figure 2: Volume-rendered views of example segmentation reference and model predicted liver (red), liver vessel (blue), and needle feature segmentation (yellow) using 3D MRI.
Figure 3: 2D views of example segmentation reference and model predicted liver (red), liver vessel (blue), and needle feature (yellow) segmentation on axial, sagittal and coronal planes. The shown case has Dice scores of 0.90, 0.72, 0.85 for liver, liver vessel, and needle.
Figure 4: Violin plots of Dice scores of liver, liver vessel and needle from multi-class model and single-class needle segmentation model. The multi-class model achieved median [interquartile range (IQR)] Dice scores of 0.87 [0.05], 0.64 [0.08], 0.76 [0.13] for liver, liver vessels, and needle. The single-class needle segmentation model achieved 0.8 [0.15]. In Mann–Whitney U test, there was no significant difference between the multi-class and single-class needle segmentation Dice score (p=0.09). The numbers shown on the violin plot are the medians of the Dice scores.
Figure 5: Violin plots of 3D needle axis (a) and needle tip (b) localization errors of multi-class and single-class models. For multi-class model, the median [IQR] needle tip localization error was 1.65 [1.39] mm (single-class: 1.94 [1.31] mm) , and median needle axis localization error was 1.05° [1.37°] (single-class: 0.98° [0.79°]). In Mann–Whitney U test, there was significant difference between the needle tip localization error of multi-class and single-class needle segmentation methods (p=0.046), but no significant difference between needle axis localization error.