2093

Contrastive Learning with Multi-Contrast Constraints for Segmentation in Renal Magnetic Resonance Imaging

Aya Ghoul¹, Lavanya Umapathy², Cecilia Zhang³, Petros Martirosian³, Ferdinand Seith³, Sergios Gatidis^1,4, and Thomas Küstner¹
¹Medical Image And Data Analysis (MIDAS.lab), Department of Diagnostic and Interventional Radiology, University Hospital of Tuebingen, Tuebingen, Germany, ²Department of Radiology, Center for Advanced Imaging Innovation and Research (CAI2R), New York University Grossman School of Medicine, New York, NY, United States, ³Department of Diagnostic and Interventional Radiology, University Hospital of Tuebingen, Tuebingen, Germany, ⁴Department of Radiology, Stanford University, Stanford, CA, United States

Synopsis

Keywords: Analysis/Processing, Segmentation, Reproducibility challenge, multi-parametric renal MRI, AI/ML Image segmentation, Kidney

Motivation: Supervised deep learning provides state-of-the-art medical image segmentation when large labeled images are accessible. However, manual segmentation suffers from prolonged delineation.

Goal(s): In response to the 2024 ISMRM Challenge “Repeat it With Me: Reproducibility Team Challenge”, we aim to show the effectiveness of contrastive learning to find suitable initialization for segmentation with limited annotation.

Approach: We use a multi-contrast contrastive loss guided by representational constraints to learn discriminating features within multi-parametric renal MR images and fine-tune the pretrained model on segmentation tasks.

Results: Our findings validate that pretraining diminishes the needed annotation effort by 60% for different imaging sequences and enhances segmentation performance.

Impact: Multi-contrast contrastive learning reduces annotation effort to train deep-learning segmentation models, confirming prior findings in a new cohort, within the 2024 ISMRM Challenge “Repeat it With Me: Reproducibility Team Challenge” and indicating its potential to improve multi-parametric imaging workflows.

Introduction

Multi-parametric renal MRI is a promising tool for the assessment and monitoring of structural and hemodynamic changes associated with kidney diseases^1,2. Generalizable image segmentation frameworks are required to assist clinical diagnostics and establish standardized data analysis workflows in the clinical routine. Deep learning methods have proved effective in handling multi-parametric datasets^3–5 and have been applied for the segmentation of kidneys^6–8. Nevertheless, collecting large annotated datasets to train such models remains a considerable challenge due to prolonged delineation and reproducibility issues. Self-supervised pretraining has been proposed to overcome the scarcity of labeled data and improve accuracy without direct knowledge of the subsequent downstream task^9,10. In this study, we apply a contrastive loss with multi-contrast constraints for pretraining on multi-parametric renal MRI to reproduce previously published findings¹¹ in a new cohort as part of the 2024 ISMRM Challenge “Repeat it With Me: Reproducibility Team Challenge”. By leveraging representational constraints derived from multi-parametric MR images, the model learns discriminative local representations with similar underlying tissue characteristics. We demonstrate the applicability of the method to reduce the annotation effort and improve segmentation performance. We also assessed the method’s generalizability for different contrast images.

Methods

The multi-contrast contrastive loss: The used contrastive loss^11,12 assumes that local regions within an MR image with similar tissue-specific parameters generate similar features in the learned representational space. First, a pixel-wise principal component (PC) analysis is performed, followed by K-means clustering to generate the constraint maps. These maps identify similar local regions that will be later used as a surrogate for tissue-specific information. During training, each patch within the image is pulled closer with concurrently similar patches in the learned feature space and in the constraint maps based on l₂-normalized cosine similarity measures, whereas dissimilar samples are pushed apart.
Data: We used 79 multi-parametric (Dixon, T1 Mapping, T2 Mapping, BOLD and diffusion) renal MR scans (59 patients with prostate cancer or neuroendocrine tumors and 20 healthy subjects), acquired in-house on a 3T scanner, as stated in Table 1. For training and testing, subjects were split into separate training/validation/test subsets of 58/6/15 subjects. Images were intensity normalized to values between 0 and 1. Clinicians manually annotated all data for quantitative evaluation.
Pretraining: A 2D encoder-decoder architecture is used^11,12. Initially, the encoder undergoes training using a conventional contrastive strategy¹³for 10 epochs. Then, the model is trained using the multi-contrast contrastive loss^11,12. Training is conducted on an NVIDIA V100 GPU with an Adam optimizer (batch size=16, learning rate=1e−4, epochs=150) with the learning rate reducing by half if the contrastive loss on the validation dataset plateaus for 10 epochs. We pretrained the model on all available training subjects (N=58) using the studied contrast images. At this stage, learned representations embed relevant contrast information that can be used as an initialization for a subsequent segmentation task.
Downstream Segmentation Task: We fine-tuned the model on a limited amount of training subjects (N) ranging from 2 to 20 to investigate the performance in case of limited labeled data. As comparison, we fine-tuned on the entire training subset to study the effect of pretraining. We compared two models, trained with weights initialized using a random initialization (RI) and the investigated contrastive loss with multi-contrast constraints (CLC) on the multi-parametric data. Fine-tuning is conducted with the focal Tversky loss¹⁴ with an alpha weighting on each class on an NVIDIA V100 GPU with an Adam optimizer (batch size=20, learning rate=1e−3, epochs=200) with the learning rate reducing by half if the Tversky loss applied on the validation dataset plateaus for 10 epochs.

Results and Discussion

The CLC model yields higher Dice scores compared to random initialization on the segmentation task with limited data (N∈[2, 20]). A CLC-trained model on just N=8 subjects performs similarly to the RI-trained model on a larger dataset of N=20 labeled instances. This underscores a 60% reduction in annotation efforts required on our dataset for all the studied contrast images, as indicated in Table 2. CLC-trained models outperform RI-trained models when fine-tuned on N=58 labeled images, quantitatively, in terms of Dice scores of the left and right kidneys (Fig.1), and kidney volumes (Fig.2) and qualitatively (Fig.3).
The superior performance of CLC-trained models across various contrasts demonstrates the capacity of the studied pretraining strategy for broad applicability. We were able to confirm the findings from the initial publication¹¹ in a new cohort.

Conclusion

Pretraining with the multi-contrast contrastive loss reduced annotation ef-fort compared to random initialization for downstream segmentation on multi-parametric renal MRI. Additional improvements in segmentation performance were also demonstrated when a larger cohort of labeled data was used.

Acknowledgements

No acknowledgement found.

References

1. A. Caroli et al., “Functional magnetic resonance imaging of the kidneys: where do we stand? the perspective of the European cost action parenchima,” Nephrology Dialysis Transplantation, 2018.

2. S. Copur et al., “Future of kidney imaging: functional magnetic resonance imaging and kidney disease progression,” European Journal of Clinical Investigation, vol. 52, no. 5, p. e13765, 2022.

3. F. G. Zöllner et al., “Kidney segmentation in renal magnetic resonance imaging-current status and prospects,” IEEE Access, vol. 9, pp. 71577–71605, 2021.

4. M. Arif et al., “Clinically significant prostate cancer detection and segmentation in low-risk patients using a convolutional neural network on multi-parametric MRI,” Eur. Radiol., vol. 30, pp. 6582–6592, 2020.

5. K. R. Laukamp et al., “Fully automated detection and segmentation of meningiomas using deep learning on routine multipara-metric mri,” Eur. Radiol., vol. 29, pp. 124–132, 2019.

6. M. F. Bobo et al., “Fully convolutional neural networks improve abdominal organ segmentation,” in Medical Imaging 2018: Image Processing, vol. 10574, pp. 750–757, SPIE, 2018.

7. V. Bevilacqua et al., “A comparison between two semantic deep learning frameworks for the autosomal dominant polycystic kidney disease segmentation based on magnetic resonance images,” BMC Medical Informatics and Decision Making, vol. 19, no. 9, pp. 1–12, 2019.

8. M. Haghighi et al., “Automatic renal segmentation in dce-mri using convolutional neural networks,” ISBI 2018, pp. 1534–1537, IEEE, 2018.

9. W. Bai et al., “Self-supervised learning for cardiac MR image segmentation by anatomical position prediction,” MICCAI, Part II 22, pp. 541–549, Springer, 2019.

10. T. Chen et al., “A simple framework for contrastive learning of visual representations,” in ICML, pp. 1597–1607, PMLR, 2020.

11. L. Umapathy et al., “Reducing annotation burden in mr segmentation: A novel contrastive learning loss with multi-contrast constraints on local representations,” ISMRM, 2023.

12. L. Umapathy et al., "Reducing annotation burden in MR: A novel MR-contrast guided contrastive learning approach for image segmentation," Med Phys. 2023;1-14.

13. K. Chaitanya et al., “Contrastive learning of global and local features for medical image segmentation with limited annotations,” Adv. neural inf. process. syst, vol. 33, pp. 12546–12558, 2020.

14. N. Abraham and N. M. Khan, “A novel focal tversky loss function with improved attention u-net for lesion segmentation,” ISBI, pp. 683–687, IEEE, 2019;5

Figures

Table 1: Acquisition parameters of the investigated renal MR protocol.

Table 2: Segmentation performance comparison of CLC-trained and RI-trained models on Dixon, T₁Mapping (T1), T₂Mapping (T2), BOLD and diffusion data. As the number of available training subjects increases, the segmentation performance improves. Significant differences in a paired t-test with Bonferroni correction were denoted with ∗ (p-value≤ 0.05).

Fig.1: Violin plots for Dice scores on test cohort when random initialization(RI) and proposed multi-contrast constraints (CLC) are fine-tuned on N=58 subjects for the segmentation task on Dixon, T1 Mapping, T2 Mapping, BOLD and diffusion data. Dice scores of left (Kidney-L) and right (Kidney-R) kidneys for CLC-trained models were higher compared to RI-trained models and showed fewer outliers. Significant differences in a paired t-test with Bonferroni correction were denoted with ∗ (p-value≤ 0.05).

Fig.2: Computed kidney volumes based on the automated segmentation masks in comparison to manual segmentation in 14 subjects using the CLC-trained model (top) and the RI-trained model for Dixon, T1 Mapping, T2 Mapping, BOLD and diffusion data. For all contrasts, the limits of agreement were wider for the RI-trained model and a larger bias was observed compared to the CLC-trained model.

Fig.3: Comparison of manual (GT) and automated multi-parametric renal MR images segmentation masks for different contrasts of two patients with prostate carcinoma (top) and a neuroendocrine tumor (bottom). N=8 (top) and N=58 (bottom) training subjects were used to fine-tune the CLC-trained and RI-trained models. Labels obtained with CLC models show a closer agreement with the manual masks with sharper and more accurate kidney boundaries compared to RI models, especially in the case of limited data (N=8). Yellow arrows indicate under-segmented areas by the RI-trained model.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)

2093

DOI: https://doi.org/10.58530/2024/2093