1001
Deep Learning Combination of FLAIR and T2W for Improved TSC Lesion Detection1Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong, China, 2University of Chinese Academy of Sciences, Beijing, China, 3Department of Neurology, Shenzhen Children’s Hospital, Shenzhen, China
Synopsis
Keywords: Diagnosis/Prediction, Epilepsy
Motivation: This study seeks to address the challenge of limited visibility of periventricular lesions in Tuberous Sclerosis Complex (TSC).
Goal(s): Develop FLAIR3, a deep neural network, for adaptive fusion of T2w and FLAIR images in TSC patients to improve lesion detection.
Approach: The study adopts a dual-stream U-Net network with a pre-fusion module and employs spatial and channel fusion weight for feature fusion. Gradient loss and segmentation annotations are utilized to generate fusion images with clear textures and improved contrast.
Results: The fused image, FLAIR3, demonstrates enhanced lesion contrast and outperforms T2w and FLAIR images in lesion segmentation.
Impact: The enhanced lesion visualization provided by FLAIR3 can aid doctors in accurately identifying and diagnosing cortical tubers, improving the overall epilepsy diagnosis and treatment in TSC patients. This work improves the accuracy of automatic tuber segmentation.
Introduction
Tubers are a major feature for TSC diagnosis, detectable in ~80–90% of patients1,2. In MRI scans, tubers appear as focal regions with distinct changes in T1, T2, and FLAIR, most noticeable when hypointense in T1 and hyperintense on T2 weighted images3. However, T2-weighted (T2w) images often have interference from cerebrospinal fluid (CSF) signal, hindering the visibility of periventricular lesions4. Additionally, the CSF within brain sulci contributes to partial-volume artifacts, thereby limiting the detection of cortical and subcortical lesions5.The FLAIR pulse sequence can improve lesion visualization by suppressing CSF signal. Recent studies have shown that multiplying T2w and FLAIR image volumes (termed FLAIR2) can produce volumes with high multiple sclerosis lesion contrast and CSF suppression6. One of the most crucial issues in image fusion is calculating a weight map which integrates the pixel activity information from different sources. In this paper, this issue is addressed by a deep neural network framework that can adaptively learn fusion weights, enabling the fused image to achieve better contrast.Methods
This paper introduces a novel approach to medical image fusion, employing a dual-stream U-Net network. The method encompasses a dual-stream encoder and decoder to extract features and reconstruct fused images as Figure 1. The initial step involves an affine transformation to precisely align the T2w image with the FLAIR image. Subsequent preprocessing comprises brain extraction and voxel intensity normalization to enhance image quality and consistency as Figure 2. Following preprocessing, the pre-registered T2w and FLAIR images are fed into the dual-stream encoder, which combines features through concatenation or addition. To ensure efficient fusion and minimize information loss, an innovative pre-fusion module is integrated into the encoding phase, facilitating feature fusion across spatial positions and modalities. It calculates custom-designed spatial fusion weight maps using a spatial attention module for each modality. Channel fusion weights are established through channel attention, and weight vectors are computed similarly to spatial weight maps.The derived weight maps and vectors are skillfully employed for feature fusion, combining the encoded features as follows,$$F_{fused} = F_{FLAIR} \odot M_{FLAIR} \otimes W_{FLAIR}+ F_{T2w}\odot M_{T2w} \otimes W_{T2w} $$where $$$ M_{FLAIR} $$$ and $$$M_{T2w} $$$represent the spatial weight maps utilized for fusion, while $$$W_{FLAIR}$$$ and $$$W_{T2w}$$$ represent the spatial weight vectors used in the fusion process.$$$ F_{FLAIR}$$$ and $$$F_{T2w}$$$ symbolize the encoded flair and T2w features, respectively. $$$ \odot $$$denotes element-wise multiplication, and $$$ \otimes $$$ signifies channel-by-channel multiplication.Gradient loss is employed to generate fusion images with rich textures, using the Sobel operator for gradient computation as follows,$$ L_{grad} = \Vert \nabla I_{fused}-max(\nabla I_{FLAIR},\nabla I_{T2w}) \Vert_1$$where $$$ \nabla$$$ represents the gradient calculation, $$$I_{fused}$$$ represents the fused image, $$$I_{FLAIR}$$$ and $$$I_{T2w}$$$ represent the input image.To enhance performance in downstream segmentation tasks, segmentation annotations are incorporated to compute the loss as follows,$$L_{lesion} = \Vert I_{mask} \odot I_{fused}-max(I_{mask} \odot I_{FLAIR},I_{mask} \odot I_{T2w}) \Vert_1 $$$$L_{norm}=\Vert(1-I_{mask}) \odot I_{fused}-min((1-I_{mask})\odot I_{FLAIR},(1-I_{mask}) \odot I_{T2w}) \Vert_1$$By constraining the network to generate high brightness in lesion areas and low brightness in non-lesion areas, fusion images with enhanced contrast and distinct boundaries are created, reducing the difficulty of segmentation tasks. The overall loss of our FLAIR3 is defined as: $$$ L=L_{grad}+L_{lesion}+L_{norm}$$$The study included 300 pediatric TSC patients at Shenzhen Children’s Hospital (Jan 2013-Sep 2018). Parental consent was obtained, and IRB approval was obtained from Shenzhen Institutes of Advanced Technology, Chinese Academy of Science.
Results
In this study, we use the network to fuse T2w and FLAIR images of the TSC patients. The fused image have better lesion contrast and show better performance in the underlying segmentation task. Figure 3 illustrates the comparison of lesion contrast between FLAIR, T2w, FLAIR2 and FLAIR3. Figure 4 displays the segmentation results of FLAIR, T2w, FLAIR2, and FLAIR3 in 3D UX-Net7. When the input image is FLAIR3, 3D UX-Net achieves the highest dice value of 0.668（FLAIR:0.637, T2w:0.596, FLAIR2:0.643).Discussion and Conclusion
A typical symptom of TSC patients is cortical tuber, some of which are related to the occurrence of epilepsy8. Determining the location and shape of lesions is important for the diagnosis and treatment of epilepsy9，10. Therapy is also very important. Due to the large number of TSC lesions, variable locations, shapes, uncertain sizes, and blurred edges, some lesions are difficult for ordinary doctors to distinguish. In this study, the proposed FLAIR3 improves lesion visualization by fusing T2w and FLAIR images of TSC patients, which is beneficial for doctors to identify and diagnose tuber. The automatic segmentation performance of TSC lesion has also been improved.Acknowledgements
This work was partially supported by the National Natural Science Foundation of China (62271474), the National Key R&D Program of China (2023YFB3811400), the High-level Talent Program in Pearl River Talent Plan of Guangdong Province (2019QN01Y986) and the Shenzhen Science and Technology Program (KQTD20180413181834876 and JCYJ20210324115810030), the Guangdong High-level Hospital Construction Fund (ynkt2021-zz11), the National Natural Science Foundation of Shenzhen Children's Hospital (ynkt2021-2z11), the Shenzhen Science and Technology Program (JCYJ20220530160005012), the Shenzhen Key Medical Discipline Construction Fund（No.SZXK033).References
1.Hu Z, Jiang D, Zhao X, et al. Predicting Drug Treatment Outcomes in Childrens with Tuberous Sclerosis Complex–Related Epilepsy: A Clinical Radiomics Study[J]. American Journal of Neuroradiology, 2023, 44(7): 853-860.
2.Zhao X, Jiang D, Hu Z, et al. Machine learning and statistic analysis to predict drug treatment outcome in pediatric epilepsy patients with tuberous sclerosis complex[J]. Epilepsy Research, 2022, 188: 107040.
3.Feliciano D M. The neurodevelopmental pathogenesis of tuberous sclerosis complex (TSC)[J]. Frontiers in Neuroanatomy, 2020, 14: 39.
4.Jiang D, Liao J, Zhao C, et al. Recognizing Pediatric Tuberous Sclerosis Complex Based on Multi-Contrast MRI and Deep Weighted Fusion Network[J]. Bioengineering, 2023, 10(7): 870.
5.Gabr R E, Hasan K M, Haque M E, et al. Optimal combination of FLAIR and T2‐weighted MRI for improved lesion contrast in multiple sclerosis[J]. Journal of Magnetic Resonance Imaging, 2016, 44(5): 1293-1300.
6.Wiggermann V, Hernandez-Torres E, Traboulsee A, et al. FLAIR2: a combination of FLAIR and T2 for improved MS lesion detection[J]. American Journal of Neuroradiology, 2016, 37(2): 259-265.
7.Lee H H, Bao S, Huo Y, et al. 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation[J]. arXiv preprint arXiv:2209.15076, 2022.
8.Yang J, Zhao C, Su S, et al. Importance of Clinical MRI Features in Predicting Epilepsy Drug Treatment Outcome for Pediatric Tuberous Sclerosis Complex[J], ISMRM 2021.
9.Jiang D, Hu Z, Zhao C, et al. Identification of Children's Tuberous Sclerosis Complex with Multiple-contrast MRI and 3D Convolutional Network[C]//2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2022: 2924-2927.
10.Yang J, Zhao C, Su S, et al. Machine Learning in Epilepsy Drug Treatment Outcome Prediction Using Multi-modality Data in Children with Tuberous Sclerosis Complex[C]//2020 6th International Conference on Big Data and Information Analytics (BigDIA). IEEE, 2020: 100-103.
Figures
