0385
Unified Diffusion model for Multi-contrast Ensembling Synthesis1School of Artificial Intelligence, Yonsei University, Seoul, Korea, Republic of, 2School of Electrical and Electronic Engineering, Yonsei University, Seoul, Korea, Republic of, 3PROBE Medical Inc., Seoul, Korea, Republic of, 4Center for Healthcare Robotics, Korea Institute of Science and Technology, Seoul, Korea, Republic of, 5Department of Oral and Maxillofacial Radiology, Yonsei University College of Dentistry, Seoul, Korea, Republic of, 6Department of Radiology and Center for Clinical Imaging Data Science (CCIDS), Yonsei University College of Medicine, Seoul, Korea, Republic of
Synopsis
Keywords: Acquisition Methods, Brain
Motivation: Scanning for multi-contrast MR images is time-consuming. To reduce scan time, it is beneficial to explore methods for efficiently synthesizing target contrast images from existing contrast scans.
Goal(s): To address the stability issues encountered when dealing with multi-contrast MR image domains individually, we propose a methodology for effectively synthesizing images while incorporating multi-contrast domains.
Approach: Our model is a novel unified diffusion model (UDM) that improves the synthesis of detailed anatomical structures in target contrast images through an ensemble method.
Results: UDM demonstrates effectiveness across multiple domains, outperforming existing methodologies in synthesizing images for each contrast domain.
Impact: By reducing scan times and costs for multi-contrast imaging, UDM facilitates prognosis prediction and treatment planning. This method is not only usable for image synthesis but also extendable to various applications such as reconstruction.
Introduction
Multi-contrast magnetic resonance (MR) images aid in predicting the prognosis and planning the treatment strategy by providing varied information for same anatomy. For the limitations of costs and patient cooperation, it becomes necessary to obtain multi-contrast MRI protocols with short scan times. To reduce scan time, recent studies5,6,7 introduced methods of synthesizing target contrast image using source contrast information, called image synthesis. Effective synthesis of missing or corrupted contrasts from acquired images can be supportive to clinical applications. However, existing methods face challenges with stability when dealing with multi-contrast MR image domains, as they typically require the construction of separate models for each domain.In this paper, we propose a novel unified diffusion model for target contrast image synthesis, called UDM. UDM enhances the quality of synthesized fine anatomical structures in target contrast images by employing an ensemble approach. Experimental results demonstrate that our method outperforms the other existing methods quantitatively and qualitatively on brain MR image synthesis.
Method
We propose a novel MR image synthesis method only using a single model, depicted in Figure 1, for handling more than two domains based on diffusion model. To produce a model for multi-contrast MR images, we train diffusion model with paired multi-contrast MR images that are not constrained by any specific contrast domain. As introduced in DDPM1,8, DDIM3, and MCG4, conditioned diffusion is applied during inference sampling but not during training. The conditional reverse SDE leads to:\begin{equation} \mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}+(1-\alpha_{t}) \triangledown_{\mathbf{x}_t}\log p_{\theta}(\mathbf{x}_t\mid \mathbf{y})\right) + \sqrt{\tilde{\sigma}_t}z, \quad z \sim \mathcal{N}(0,\mathbf{I}),\end{equation}
where $$$\mathbf{x}_t$$$ is the predicted MR image with noise perturbation at step $$$t$$$ and $$$\mathbf{y}$$$ is known multi-contrast MR images.
In the reverse denoising process, the denoising model in the initial steps is specialized to generate low-frequency content for the basic structure of the MR image. The denoising model in the later steps specializes in generating high-frequency content for details. To enhance the quality of the MR image, we introduce an ensemble mechanism for each step, as shown in Figure 2. Our proposed ensemble approach employs progressive attenuation to construct low-frequency information effectively. This indicates that effectively generating the overall structure in the initial steps aids in generating high-frequency details in the later reverse diffusion steps.
Experiments and Results
Dataset and Implementation. We conduct experiments on BraTS 2021 dataset from the RSNA-MICCAI Brain Tumor Radiogenomic Classification challenge2. The dataset comprises four paired multi-contrast MR imagesㅡFLAIR, T1, T1CE, and T2—all of which are aligned to a unified anatomical template. We randomly selected 1900 cases for training and center slices from 80 cases for validation. The base model, DDPM, was trained using the BraTS training data with steps 1000. At testing, the number of reverse denoising steps was 200. For quantitative comparison, we used peak-to-noise-ratio (PSNR)9 and structural-similarity-index (SSIM).Comparison with previous methods. The task involved synthesizing the missing contrast from three known fully-sampled contrasts, with each contrast type being evaluated respectively. We compared our approach with existing methods including: DDPM, DDIM and MCG. Figure 3 shows that brain images reconstructed using previous method do not properly generate important anatomical structures such as tumor. Table 1 also demonstrates our method outperforms the other previous methods. UDM greatly improved the image quality by recovering sharpness and adding more structural details to the brain images.
Ablation Study. As shown in Table 1, we conducted comparative experiments by varying the number and proportion of ensembles applied during the reverse diffusion steps. The ours(a) used 4 ensembles for the first 50% of steps and 2 for the remaining 50%, while ours(b) progressively reduced the ensembles from 4 at the first 50%, to 2 at 25%, and down to 1 for the final 25%. The negligible performance differences suggest that the ensemble technique is robust during the initial steps. Therefore, for enhanced inference speed and efficient use of the ensemble technique, we select the final ours(b).
Conclusions
UDM provides a robust solution for estimating missing MR contrast, which plays a pivotal role in clinical scenarios where a specific image contrast cannot be obtained. The proposed method enables image synthesis across various contrasts using a unified model and enhances the generation of detailed structures by utilizing an ensemble technique. This suggests a significant advance in the practical application of AI for improving medical imaging by leveraging the power of machine learning to increase diagnostic precision and optimize treatment plans.Acknowledgements
This work was supported in part by Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Science and ICT (2021R1C1C2008773, 2021R1A4A1031437, 2022R1A2C2008983). This work was also supported by Artificial Intelligence Graduate School Program at Yonsei University [No. 2020-0-01361], KIST Institutional Program (Project No.2E32271-23-078), and partially supported by the Yonsei Signature Research Cluster Program of 2023 (2023-22-0008).References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
[2] Baid, U.; Ghodasara, S.; Mohan, S.; Bilello, M.; Calabrese, E.; Colak, E.; Farahani, K.; Kalpathy-Cramer, J.; Kitamura, F.C.; Pati, S.; et al. The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification. arXiv 2021, arXiv:2107.02314.
[3] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR, 2021a.
[4] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00941, 2022a.
[5] Dar, Salman UH, et al. "Image synthesis in multi-contrast MRI with conditional generative adversarial networks." IEEE transactions on medical imaging 38.10 (2019): 2375-2388.
[6] Karthik, Enamundram MV Naga, Catherine Laporte, and Farida Cheriet. "Three-dimensional segmentation of the scoliotic spine from MRI using unsupervised volume-based MR-CT synthesis." Medical Imaging 2021: Image Processing. Vol. 11596. SPIE, 2021.
[7] Kim, Sewon, et al. "Fat-saturated image generation from multi-contrast MRIs using generative adversarial networks with Bloch equation-based autoencoder regularization." Medical Image Analysis 73 (2021): 102198.
[8] Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020).
[9] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Imagequality assessment: from error visibility to structural similarity,” IEEETransactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004
Figures
Figure 3. (a) The concept of this study. The task is to synthesize one sequence image using only three sequence images.
(b) Quantitative and qualitative results of our study. This figure presents example MR images for each of the four sequences.