3758

Synthesizing Missing MRI Sequences Towards Reliable Brain Tumor Segmentation Using Deep Learning
Abdulkhalek Al-Fakih1,2, Abdullah Shazly1,2, Abbas Mohammed1,2, Mohammed Elbushnaq1, Meena Makary1,3, and Mohammed A. Al-masni2
1Department of Biomedical Engineering and Systems, Cairo University, Cairo, Egypt, 2Department of Artificial Intelligence, Sejong University, Seoul, Korea, Republic of, 3MGH/HST Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Charlestown, MA, United States

Synopsis

Keywords: Synthetic MR, Data Analysis, Brain Tumor Segmentation, MR Sequence Synthesis, nnU-net, GANs, Multi Contrast MR, Deep Learning

Motivation: Automated and robust segmentation of brain tumors requires multiple MRI sequences and can expedite neuro-oncological clinical trials.

Goal(s): Our goal was to develop a deep learning model for brain tumor segmentation, even when some MRI sequences are missing.

Approach: We enhanced a GAN with attention modules to synthesize missing sequences and employed an optimized nnU-Net for segmentation using both real and synthesized sequences.

Results: The proposed AI-based model significantly improved brain tumor segmentation, with overall Dice scores increasing from 0.688% when FLAIR is missing to 0.873% using synthesized FLAIR derived from T2, and achieving 0.901% with real FLAIR.

Impact: The developed two-stage deep learning framework, comprising synthesis and segmentation, enhances segmentation of brain tumors in MRI, especially when real sequences are unavailable. This advancement accelerates clinical trials and reduces manual segmentation time, yielding promising results.

Introduction

Optimal assessment of glioblastoma necessitates the utilization of information derived from all four MRI sequences, namely T1, T2, FLAIR, and T1ce. However, acquiring and incorporating data from all these sequences can be challenging due to the time-consuming and costly nature of the sequence acquisition process.1 In addition, artifacts are commonly present in some sequences due to various reasons such as motion artifacts and magnetic inhomogeneity artifacts. As a result, accessing complete information from all four sequences may not always be feasible. This study aims to address the challenge of acquiring complete MRI sequences, essential for optimal glioblastoma segmentation. We propose a methodology to synthesize missing MRI sequences, thus ensuring all necessary data is available for accurate tumor segmentation.

Methods

This study proposes a deep learning pipeline that encompasses identifying and synthesizing missing MRI sequences and then using these synthesized sequences for brain tumor segmentation, as illustrated in Figure 1. We utilize the BraTS2021 dataset, which includes 1251 subjects, to address the problem of missing MRI sequences.
The proposed AI-based approach initiates by identifying types of brain MRI sequences using an 18-layer residual network.2 Subsequently, we utilize a modified 2D GAN, consisting of a squeeze attention U-Net generator and a PatchGAN3 discriminator, to synthesize absent MRI sequences from the existing ones, completing the set, as depicted in Figure 2. The primary objective of Squeeze-and-Excitation (SE) block4 is to recalibrate channel responses by explicitly capturing the interdependencies between them. For generating missing FLAIR sequences, we trained three distinct GAN models that derived input from T1ce, T1, and T2 images. To obtain a consolidated loss measure during generator training, the total loss is calculated as the average of SSIM and MAE, providing insight into both structural similarity and pixel-wise dependencies.
Next, the four MRI sequence volumes, including any synthesized missing sequences, are fed into a self-configuring 3D optimized nnU-Net segmentation model5 for delineating brain tumors (Edema, Enhancing Tumor, and Necrosis). In this task, a combination of cross-entropy and Dice losses is used as the loss function, and deep supervision is applied to compute the loss at the last three decoder layers.

Results

In this study, we investigated the impact of omitting each MRI sequence individually by replacing it with black images (i.e., zero maps) on the task of brain tumor segmentation. Our findings revealed that both T1ce and FLAIR images play a crucial role in accurately segmenting brain tumors compared to the other T1 and T2 sequences. However, compensating for the contrast agent injected in T1ce poses challenges for deep learning models. Therefore, our study primarily focuses on synthesizing the FLAIR sequence. Figure 3 shows some examples of the synthesized FLAIR sequences generated from T1, T1ce, and T2 sequences.
To demonstrate the effectiveness of our synthesis method, we compared it to scenarios where sequences are missing or substituted (i.e., replaced by a black volume or by a copy of T2). Notably, the absence of the FLAIR sequence significantly affects segmentation compared to other sequences as shown in Table 1. Visual examples in Figure 4 illustrate the significance of the synthesis process in improving segmentation accuracy when compared to alternative methods involving black and copy substitutions.

Discussion

Deep learning techniques for brain tumor segmentation often necessitate the availability of multimodal MRI images, which might not be doable in clinical practice due to various reasons, including differences in acquisition protocols and the presence of image artifacts. This hinders the development and application of such models. This study demonstrates the effectiveness of GANs in generating missing MRI sequences, particularly FLAIR images crucial for tumor segmentation. our proposed GANs using Squeeze Attention U-Net and PatchGAN achieved favorable results, especially when synthesizing FLAIR from T2 sequences.
The synthesized FLAIR sequences were integrated with available data and improved segmentation by 18.5% on average Dice score compared to no FLAIR and by 7% compared to a copy of the T2 sequence. A multi-model approach, combining different inputs, outperformed individual models, offering promising solutions for missing MRI sequences and advancing medical segmentation efforts.

Conclusion

Synthesizing missing MRI sequences can enhance medical segmentation efforts. Promising results were obtained when the synthesized sequence was combined with the three already-existing sequences in the brain tumor segmentation model. The absence of the FLAIR sequence notably impacts segmentation, making its synthesis crucial. Synthesizing the FLAIR sequence from the T2 sequence has exhibited superior performance in terms of SSIM and Dice similarity coefficient compared to synthesizing FLAIR from other sequences.

Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF) funded by the Korean government (MSIT) (No. RS-2023-00243034).

References

[1] Arti Tiwari, Shilpa Srivastava, and Millie Pant, “Brain tumor segmentation and classification from magnetic resonance images: Review of selected methods from 2014 to 2019,” Pattern recognition letters 131 (2020): 244-260.

[2] Jean Pablo Vieira de Mello, Thiago M. Paixão, Rodrigo Berriel, Mauricio Reyes, Claudine Badue, Alberto F. De Souza, and Thiago Oliveira-Santos, “Deep Learning-based Type Identification of Volumetric MRI Sequences,” In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 1-8. IEEE, 2021.

[3] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros, “Image-to-image translation with conditional adversarial networks,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.

[4] Jie Hu, Li Shen, and Gang Sun, “Squeeze-and-excitation networks,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.

[5] Fabian Isensee, Jens Petersen, Andre Klein, David Zimmerer, Paul F. Jaeger, Simon Kohl, Jakob Wasserthal et al., “nnu-net: Self-adapting framework for u-net-based medical image segmentation,” arXiv preprint arXiv:1809.10486 (2018).

Figures

Figure 1. The flowchart of the proposed framework includes sequence identification, missing data synthesis, and brain tumor segmentation.

Figure 2. The architecture of GANs model, involving a squeeze attention generator and a PatchGAN discriminator, used for the synthesizing of missing FLAIR sequences.

Figure 3. Visual examples of the synthesized FLAIR sequences derived from T1, T1c, and T2 data sources. gFLAIR_fT2, gFLAIR_fT1c, and gFLAIR_fT1 represent the generated FLAIR images from T2-weighted, post contrast T1-weighted, and T1-weighted images, respectively. SSIM scores are inscribed on the synthesized images to indicate the degree of resemblance between the generated images and the real FLAIR data.

Table 1. Outcomes of our experiments using the optimized nn-Unet as the segmentation model and squeeze attention GANs as the synthesis model under four conditions: (1) when all sequences are available, (2) when one sequence is missing and replaced with zeros, (3) when the T2 sequence is duplicated to replace the missing FLAIR sequence, and (4) when the FLAIR sequence is synthesized. The arrows in the table indicate the sequences used for synthesizing the FLAIR sequence

Figure 4. Segmentation results of various scenarios when the FLAIR sequence was missing. These scenarios included: the original four MRI sequences, bFLAIR indicates the using of black FLAIR image, and cpT2 refers to a copy of T2-weighted image instead of FLAIR. gFLAIR_fT2, gFLAIR_fT1c, and gFLAIR_fT1 represent the generated FLAIR images from T2-weighted, post contrast T1-weighted, and T1-weighted images, respectively. Dice scores are inscribed on the images.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
3758
DOI: https://doi.org/10.58530/2024/3758