0652
Distribution Matching Based Personalized Federated Learning for Multi-Contrast Liver MRI Synthesis and Registration1Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China, 2Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China, 3Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China, 4Department of Radiology, The First affiliated Hospital, School of Medicine, Zhejiang University, Shanghai, China, 5Department of Radiology, Zhongshan Hospital, Fudan University, Shanghai, China, 6Siemens Healthineers, Shanghai, China, 7Philips Healthcare, Shanghai, China, 8Human Phenome Institute, Fudan University, Shanghai, China
Synopsis
Keywords: AI/ML Image Reconstruction, Liver
Motivation: The combined diagnosis of diffusion-weighted imaging (DWI) and dynamic contrast-enhanced (DCE)-MRI is of significant importance for liver diseases, but accurate registration between these two modalities remains a substantial challenge.
Goal(s): Our goal was to design a deep learning model for accurate registration between DCE and DCE-MRI, and conduct multicenter studies based on federated learning.
Approach: We proposed a multi-task synthesis-registration network (SynReg) and a personalized decentralized distribution matching federated framework (PDMa) based on SynReg.
Results: The proposed SynReg and PDMa method increased the registration accuracy in most centers both in liver region and liver tumor region.
Impact: Accurate and rapid registration of DWI and DCE can effectively assist clinicians in leveraging multimodal imaging for efficient diagnosis. Personalized federated learning can effectively aid single-center with limited data to leverage the abundant data from multiple centers for model development.
Introduction
Diffusion-weighted imaging (DWI) and dynamic contrast-enhanced (DCE)-MRI are both of significant value in the clinical diagnosis of liver diseases (1-4). Accurate and robust image registration is crucial to facilitate further multimodal image analysis, such as lesion detection, segmentation and risk stratification (5-6). Recently, owing to the rapid inference speed, high registration accuracy, and diverse network structures, deep learning networks have performed enormous potential in the field of medical image registration. However, two challenges remain when deep learning registration networks are applied in the multi-contrast liver MRI registration between DWI and DCE images. The first challenge is how to address the significant contrast differences between DWI and DCE, and the second challenge is how to trained a satisfactory registration network with insufficient high-quality data.Methods
This retrospective study included 751 consecutive patients with liver cirrhosis (407 HCC patients) who were examined between January 2013 and February 2021. The datasets included patients from five medical centers and three manufacture vendors: a) 110 patients from Site 1 with GE MR scanner; b) 194 patients from Site 2, including 78 patients scanned with Philips MR scanner (Site 2P) and 116 patients with Siemens MR scanner (Site 2S); c) 140 patients from Site 3 with Siemens MR scanner; d) 131 patients from Site 4 with GE MR scanner; e) 176 patients from Site 5 with Philips MR scanner. For each site, the datasets were divided into training, validation, and test sets in a 7:1:2 ratio.The proposed framework primarily consists of two parts. (1) Multi-task synthesis-registration network (SynReg), as demonstrated in Figure 1, SynReg combines the multi-contrast image synthesis task with the mono-contrast registration task. During the training process of the network, the registration network can eliminate the misalignment in paired images of datasets, thereby providing higher quality training data for the synthesis network. Simultaneously, the high-quality synthesized images reduce the difficulty of multi-contrast registration, leading to an improvement in the registration accuracy of the registration network. (2) Personalized decentralized distribution matching framework (PDMa) based on SynReg backbone, as demonstrated in Figure 2, PDMa enables each medical center to leverage similar data from multiple centers, and train high-quality personalized models tailored to their own center. We chose image signal-to-noise ratio (SNR), which significantly impacts registration accuracy, as the matching parameter, and designed a matching mechanism based on SNR distribution information to acquire similar data from multiple centers. Furthermore, we selectively aggregated network parameters based on the multi-task characteristics of synthesis-by-registration paradigm.
In model evaluation, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were utilized to assess the image quality, while target registration error (TRE) and Dice similarity coefficient (DSC) were applied to evaluate registration accuracy.
Results
In the DCE synthesis task, SynReg achieve a PSNR of 20.57±1.70 and a SSIM of 0.769±0.056, which outperformed the GAN model (7) with a PSNR of 19.14±1.74 and a SSIM of 0.744±0.057. Some representative cases of comparative image synthesis results were displayed in Figure 3. In terms of registration, PDMa exhibited superior performance in all centers compared to affine registration, SynReg based on single-center data and conventional personalized federated learning (8), achieving a DSC ranging from 0.770±0.142 to 0.833±0.096 across various centers in the liver region, and a TRE ranging from 2.87±1.55 to 5.67±3.51 across various centers in the tumor region. Specific performance metrics are listed in Tables 1. Moreover, some representative comparative results of liver tumor registration are presented in Figure 4, where the PDMa displays the smallest intra- and inter-slice registration errors.Discussion
The results demonstrated that the proposed SynReg and PDMa frameworks can effectively improve the registration accuracy of the liver region, particularly the tumor areas, between DWI and DCE images. Compared to traditional registration methods, SynReg utilized a multi-task collaborative training mechanism, simultaneously enhanced the performance of both synthesis and registration networks, ultimately yielding more accurate registration results. Based on SynReg backbone, PDMa further utilized the SNR matching mechanism to obtain high-quality multicenter data for personalized federated learning, thereby improving the registration performance of the current center. Future work can focus on finding better registration backbones, improving data matching mechanisms, and extending the proposed framework to other cross-modality MRI registration tasks.Conclusion
This study proposed the synthetic-registration multi-task network, combine with a personalized decentralized federated learning mechanism for accurate liver registration based on multi-center data. The method is expected to be extended to clinical practice for accurate and rapid liver registration between DWI and DCE images, thereby assisting physicians in combining multimodal images for efficient diagnosis of liver diseases.Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 81971583, No. 82271956), Shanghai Municipal Science and Technology Major Project (No. 2018SHZDZX01), National Key R&D Program of China (No. 2018YFC1312900).References
1. Parikh, T., Drew, S. J., Lee, V. S., Wong, S., Hecht, E. M., Babb, J. S. and Taouli, B. Focal liver lesion detection and characterization with diffusion-weighted MR imaging: comparison with standard breath-hold T2-weighted imaging. Radiology, 246, 3 (2008), 812-822.
2. Shankar, S., Kalra, N., Bhatia, A., Srinivasan, R., Singh, P., Dhiman, R. K., Khandelwal, N. and Chawla, Y. Role of Diffusion Weighted Imaging (DWI) for Hepatocellular Carcinoma (HCC) Detection and its Grading on 3T MRI: A Prospective Study. Journal of Clinical and Experimental Hepatology, 6, 4 (2016/12/01/ 2016), 303-310.
3. Digumarthy, S. R., Sahani, D. V. and Saini, S. MRI in detection of hepatocellular carcinoma (HCC). Cancer Imaging, 5, 1 (2005), 20.
4. Chen, B.-B. and Shih, T. T.-F. DCE-MRI in hepatocellular carcinoma-clinical and therapeutic image biomarker. World journal of gastroenterology, 20, 12 (2014), 3125–3134.
5. Rueckert, D. and Aljabar, P. Nonrigid Registration of Medical Images: Theory, Methods, and Applications [Applications Corner]. IEEE Signal Processing Magazine, 27, 4 (2010), 113-119.
6. Nakayama, Y., Li, Q., Katsuragawa, S., Ikeda, R., Hiai, Y., Awai, K., Kusunoki, S., Yamashita, Y., Okajima, H. and Inomata, Y. Automated hepatic volumetry for living related liver transplantation at multisection CT. Radiology, 240, 3 (2006), 743-748.
7. Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: Institute of Electrical and Electronics Engineers, 2017; 5967-5976. doi: 10.1109/CVPR.2017.632.
8. Fallah A, Mokhtari A, Ozdaglar AJae-p. Personalized Federated Learning: A Meta-Learning Approach. arXiv:2002.07948, 2020. doi: 10.48550/arXiv.2002.07948.
Figures
