2091
MR-guided automatic whole-brain segmentation via deep learning technology based on integrated PET/MRI system1Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China., shenzhen, China, 2Department of Medical Imaging, Henan Provincial People's Hospital & People's Hospital of Zhengzhou University, Zhengzhou 450003, China., henan, China, 3Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences., shenzhen, China
Synopsis
Keywords: Analysis/Processing, Brain
Motivation: Segmentation of brain tissues plays a significant role in quantifying and visualizing anatomical structures based on PET/MRI systems.
Goal(s): However, most of the current methods are based on unimodal MRI but rarely combine structural and functional dual-modality information.
Approach: In this paper, we proposed a dual-modality segmentation framework to achieve automatic and accurate segmentation for the whole brain.
Results: The numerical experimental results demonstrate that the proposed method can incorporate multimodal information with the efficient and accurate segmentation performance achieved, allowing for better visualization and quantification results.
Impact: We proposed a novel dual-modality whole-brain segmentation method based on PET and MR images that is beniificial to enrich the network features. Additionally, our method has reduced the segmentation time and could be implemented with other multimodal data.
Introduction
The brain is the command center of the central nervous system [1], and any abnormalities, disorders, or diseases in the human brain will significantly affect the entire body [2]. Currently, real-time PET/MRI imaging offers a promising opportunity for fast and accurate segmentation of the brain based on the fusion of structural and functional information [3]. Among the whole-brain segmentation methods, there are approximately three categories of commonly used algorithms [4]: (a) atlas-based [5]; (b) model-based [6]; and (c) supervised learning [7]. Despite the outstanding performance of these approaches, the majority of them have only been used with MRI images, ignoring the importance of multimodal information fusion on brain images [8]. In addition, different modalities convey different information, which greatly enriches the features during network training. Motivated by these observations, this paper proposes a dual-modality deep learning-based method to segment the whole brain automatically.Materials and methods
Patient studies: We collected 120 brain scans from Henan Provincial Hospital. All images were acquired with a PET/MRI scanner (uPMR 790, produced by UNITED IMAGING HEALTHCARE, UIH). After receiving an intravenous 18 F-FDG injection, all patients underwent head scanning approximately 50 minutes later. A 3D T1-weighted, 10-degree flip-angle magnetization-prepared rapid gradient-echo (MP-RAGE) sequence was used for MRI. FreeSurfer software was utilized to register the PET/MRI image pairs and generate ground truth data. Moreover, a one-hot coding operation was utilized to convert the ground truth's 256×256×256 matrix dimension to 45×256×256×56, where 45 represents the number of distinct classes of labels.Method Implements: Considering that nnU-Net has been successfully applied to medical imaging and can be easily implemented with out-of-the-box system rules without the need for task-specific optimization, we employed nnU-Net for whole-brain segmentation. To leverage the dual-modality information, we concatenated the PET and MRI images and fed them as parallel inputs into the model. Fivefold cross-validation was employed to train and validate the proposed method. In summary, the proposed method is detailed in Figure 1.
Data analysis: The segmentation performance was evaluated with four metrics, namely, the Dice similarity coefficient (DSC), Jaccard index (JAC), recall, and precision. We also compared the proposed segmentation method with a single-modality method, in which PET or MRI images are single-channel inputs to the network. Beyond the effect of input modality, we also explored the effect of different dimensional models on the segmentation performance, specifically, illustrating the difference in segmentation accuracy between 2D and 3D models.
Results
Figure 2 shows the comparative results of predictions based on unimodal PET, unimodal MRI, and bimodal PET/MRI with the 2D nnU-net and 3D nnU-net networks. Generally, the results of the unimodal PET method showed only a rough outline, while the results with PET/MRI dual-modality input were more accurate in shape. We have emphasized in white boxes areas in which the PET/MRI bimodal method is superior to the other methods, which were closer to the ground truth. In addition, we also show the segmentation results for some specific anatomical structures in Figure 3. We can observe that for the larger tissues, the dual-modality method enables us to provide complete details in the segmentation results; for the smaller ones, the proposed method can also perform accurate localization and segmentation. To quantitatively evaluate the performance of our method, the results are listed in TABLE 1. The averages and standard deviations of the DSC, Jaccard index, precision, and sensitivity were 84.24±1.44%, 74.36 ±2.4%, 84.33±1.56%, and 84.73±1.56%, respectively, for all test instances. The dual-modality method described in this study outperforms the unimodal approach in terms of both visual results and quantitative outcomes, with a particular visible improvement in model performance relative to the unimodal PET method.Discussion and conclusion
In this study, we used the deep convolutional network nnU-net to achieve automatic segmentation for whole brain structures with PET/MRI dual-modality data, which improves the accuracy and speed of segmenting brain images. The experimental results demonstrate that the PET/MRI dual-modality approach achieves superior segmentation results across different data dimensionalities. Compared with the unimodal approach, our method has the ability to precisely locate and segment conventional structures, even those with small sizes, by combining structural and functional information. In addition, the proposed method reduces the segmentation time to approximately 2 minutes, an appreciable improvement in the segmentation speed compared to the 7 hours spent by FreeSurfer. In future work, we will attempt to apply our method to other multimodal tasks, such as the analysis of PET/CT data, so that it could play an important role in diagnostic and radiotherapy treatment planning.Acknowledgements
This work was supported by the National Natural Science Foundation of China (32022042 and 62101540), the Shenzhen Excellent Technological Innovation Talent Training Project of China (RCJC20200714114436080), and the Shenzhen Science and Technology Program of China (JCYJ20220818101804009, RCBS20210706092218043).References
[1] S. H. Siddiqi, K. P. Kording, J. Parvizi, and M. D. Fox, "Causal mapping of human brain function," Nat Rev Neurosci, vol. 23, no. 6, pp. 361-375, Jun 2022, doi: 10.1038/s41583-022-00583-8.
[2] E. Candelario-Jalil, R. M. Dijkhuizen, and T. Magnus, "Neuroinflammation, Stroke, Blood‒Brain Barrier Dysfunction, and Imaging Modalities," Stroke, vol. 53, no. 5, pp. 1473-1486, May 2022, doi: 10.1161/STROKEAHA.122.036946.
[3] K. O. Almansory and F. Fraioli, "Combined PET/MRI in brain glioma imaging," Br J Hosp Med (Lond), vol. 80, no. 7, pp. 380-386, Jul 2 2019, doi: 10.12968/hmed.2019.80.7.380.
[4] A. Jog, A. Hoopes, D. N. Greve, K. Van Leemput, and B. Fischl, "PSACNN: Pulse sequence adaptive fast whole brain segmentation," Neuroimage, vol. 199, pp. 553-569, Oct 1 2019, doi: 10.1016/j.neuroimage.2019.05.033.
[5] A. Largent et al., "Automatic brain segmentation in preterm infants with posthemorrhagic hydrocephalus using 3D Bayesian U-Net," Hum Brain Mapp, vol. 43, no. 6, pp. 1895-1916, Apr 15 2022, doi: 10.1002/hbm.25762.
[6] L. Henschel, S. Conjeti, S. Estrada, K. Diers, B. Fischl, and M. Reuter, "FastSurfer - A fast and accurate deep learning based neuroimaging pipeline," Neuroimage, vol. 219, p. 117012, Oct 1 2020, doi: 10.1016/j.neuroimage.2020.117012.
[7] O. Puonti, J. E. Iglesias, and K. Van Leemput, "Fast and sequence-adaptive whole-brain segmentation using parametric Bayesian modeling," Neuroimage, vol. 143, pp. 235-249, Dec 2016, doi: 10.1016/j.neuroimage.2016.09.011.
[8] Y. Zhuge et al., "Brain tumor segmentation using holistically nested neural networks in MRI images," Med Phys, vol. 44, no. 10, pp. 5234-5243, Oct 2017, doi: 10.1002/mp.12481.
Figures
