0879

A deep multimodal fusion framework for MRI-based segmentation of intracranial arterial calcification
Xin Wang1,2, Gador Canton2, Yin Guo2,3, Kaiyu Zhang2,3, Thomas S. Hatsukami2,4, Jin Zhang5, Beibei Sun5, Huilin Zhao5, Yan Zhou5, Mahmud Mossa-Basha2, Chun Yuan2,6, and Niranjan Balu2
1Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, United States, 2Vascular Imaging Lab, Department of Radiology, University of Washington, Seattle, WA, United States, 3Department of Bioengineering, University of Washington, Seattle, WA, United States, 4Department of Surgery, University of Washington, Seattle, WA, United States, 5Department of Radiology, Ren Ji Hospital, Shanghai Jiao Tong University, Shanghai, China, 6Department of Radiology and Imaging Sciences, University of Utah, Salt Lake City, UT, United States

Synopsis

Keywords: Analysis/Processing, Segmentation, Vessel wall imaging, intracranial calcification, multimodal fusion

Motivation: Recently, MRI-based intracranial arterial calcification segmentation has got increasing interest due to its clinical value, but current approaches to this challenging problem suffer from poor performance.

Goal(s): To develop a deep learning model for enhancing calcification segmentation on MRI by using CT as additional training resource.

Approach: A dissimilarity loss is proposed to align the latent features learned from MRI and CT of the same subject, thus making MR feature simpler and it easier for segmentation.

Results: Compared with several commonly used segmentation networks, our model demonstrates superior performance in calcification segmentation. The ablation study further shows the effectiveness of the dissimilarity loss.

Impact: The proposed model could be applied in clinical scenarios to automatically segment calcification on cerebral MR scans and it does not require CT imaging. Radiologists could leverage the segmentation result in the analysis of various vessel plaque components.

Introduction

Intracranial arterial calcification is a crucial vessel plaque component in the development of atherosclerosis, stroke, and other vascular diseases1, highlighting its importance in quantification. Previous studies for calcification segmentation were usually conducted on CT, since calcification appears with high signal. However, recently there is a rapidly growing demand for MRI-based calcification assessment, in order to avoid radiation and study the clinically significant interaction between calcification and other plaque components which are only visible on MRI2. It is highly difficult to segment calcification on MRI because it usually appears dark (Figure 1), and existing deep networks could suffer from poor performance. In this work, we hypothesize that since CT presents clearer calcification structures, it can guide the network to better extract features from MRI. Thus, we present a novel model, which learns to extract similar features from MRI and CT during training. Experiments showed such strategy improves the segmentation performance of our model on test MR images even without the presence of CT, compared to several state-of-the-art networks.

Methods

This study utilized 113 subjects from Renji hospital, China. The subjects underwent CT angiography (CTA) and multi-sequence MRI scans including T1-weighted, time-of-flight and SNAP. For each subject, the CTA and three MR sequences were co-registered, and then the vessel centerlines of internal carotid arteries and middle cerebral arteries were tranced by a radiologist3. Curved planar reformation was then performed to generate 2D cross-sectional images along the centerlines4. Therefore, for each location on a centerline of a subject, there are four co-registered 2D slices with vessel centered, corresponding to CTA and three MR sequences, respectively. They are input together to the model to predict one 2D segmentation mask for calcification. The subjects are randomly divided into three groups with a ratio of 10:1:1, for training, validation and test, respectively.

Our network contains an MR encoder, a CT encoder, and a segmentation decoder, each of which is the same as in a U-Net, shown in Figure 2. The MR encoder takes the 2D slices of three MR sequences as input, and produces an MR feature, which actually represents the mean and covariance matrices of a diagonal Gaussian distribution, denoted by $$$q_{MR}:=q(z|x_{MR})$$$, where $$$x_{MR}$$$ is the input MR slices. Based on the MR feature, the decoder predicts calcification segmentation, which is supervised by the Focal loss, a segmentation loss designed for highly unbalanced datasets. Moreover, during training the CT encoder extracts from the input CT slice a CT feature, which parameterizes another Gaussian distribution $$$q_{CT}:=q(z|x_{CT})$$$, similar to the MR encoder. To use CT to guide network training for better MR feature extraction, we impose a dissimilarity loss between the MR and CT features:$$L_{dis}=D_{KL}[(q_{MR}q_{CT})^{\frac{1}{2}}\|\frac{1}{2}(q_{MR}+q_{CT})],$$where $$$D_{KL}$$$ is the Kullback-Leibler (KL) divergence between two distributions. Intuitively, $$$L_{dis}$$$ measures the dissimilarity between the geometric and arithmetic means of the MR and CT features. Thus, its minimization will encourage the network to extract similar features from MRI and CT.

Intuitively, tissue structures in CT are less complex than MRI, therefore, CT feature could potentially be simpler and thus better for the decoder to segment calcification. By forcing the network to extract a similar feature from MRI, we could improve the performance when the decoder takes an MR (rather than CT) feature as input. Note that our goal is to improve MRI-based segmentation, so CT is used for training only, but not for test.

Results

We compared our model with several commonly used networks5,6,7, which segment calcification based on MRI because they cannot leverage CT during training. We also implement a variant of our model by removing the CT encoder and only using the segmentation loss for training on MRI slices. As in Table 1, our model generally shows superior performance compared to all baselines. Besides, our model trained with CT is better than that without CT, even if during test there are only MR images presented. This demonstrates the effectiveness of the strategy of using CT to guide model training via minimizing feature dissimilarity. Results on three example slices are also shown in Figure 3. It is evident that our model achieved relatively accurate calcification boundaries.

Discussion and Conclusion

The proposed method demonstrated promising results for MRI-based calcification segmentation, by fusing information between CT and MRI during training to learn better MR feature extraction. Although MRI-based calcification assessment is considered extremely difficult, this work shows that we can go beyond conventional network design and instead improve the results by rethinking the core challenge. This work could help establish a more comprehensive MRI-based plaque analysis workflow where calcification evaluation is an integral part of the process.

Acknowledgements

This work was partially funded by National Institute of Health (NIH) grants R01NS092207 and R01NS127317.

References

  1. Bugnicourt Jean-Marc, Leclercq Claire, ChillonJean-Marc, et al. Presence of Intracranial ArteryCalcification Is Associated With Mortality and Vas-cular Events in Patients With Ischemic Stroke AfterHospital Discharge. Stroke. 2011;42(12):3447-3453.
  2. Mandell D.M., Mossa-Basha M., Qiao Y., et al. Intracranial Vessel Wall MRI: Principles and Expert Consensus Recommendations of the American Society of Neuroradiology. American Journal of Neuroradiology. 2017;38(2):218–229.
  3. Chen Li, Mossa-Basha Mahmud, Balu Niranjan, et al. Development of a quantitative intracranial vascular features extraction tool on 3D MRA using semi-automated open-curve active contour vessel tracing. Magnetic Resonance in Medicine. 2018;79(6):3229-3238.
  4. Guo Yin, Canton Gador, Chen Li, et al. Multi-Planar, Multi-Contrast and Multi-Time Point Analysis Tool (MOCHA) for Intracranial Vessel Wall Characterization. Journal of Magnetic Resonance Imaging. 2022;56(3):944-955.
  5. Ronneberger Olaf, Fischer Philipp, Brox Thomas. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab Nassir, Hornegger Joachim, Wells William M., Frangi Alejandro F., eds. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, :234–241Springer International Publishing; 2015; Cham.
  6. Khanna Anita, Londhe Narendra D., Gupta S., Semwal Ashish. A deep Residual U-Net convolutional neural network for automated lung segmentation in computed tomography images. Biocybernetics and Biomedical Engineering. 2020;40(3):1314-1327.
  7. Oktay Ozan, Schlemper Jo, Folgoc Loic Le, et al. Attention U-Net: Learning Where to Look for the Pancreas. 2018.

Figures

Figure 1. An example slice showing intracranial arterial calcification segmentation with yellow contours on CT angiography (CTA), T1-weighted and time-of-flight (TOF).

Figure 2. The proposed model architecture.

Table 1. Quantitative segmentation performance metrics from the compared methods.

Figure 3. Visualization of calcification segmentation on three example slices, with yellow and green contours indicating the ground-truth and predictions from our model, respectively. CTA is presented for reference purpose, but was not used for network prediction.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
0879
DOI: https://doi.org/10.58530/2024/0879