1475
Multi-task deep learning with triplet uncertainty for segmentation and microvascular invasion prediction of hepatocellular carcinoma1School of Medical Information Engineering, Guangzhou University of Chinese Medicine, Guangzhou, China, 2Department of Medical Imaging Center, Nanfang Hospital, Southern Medical University, Guangzhou, China
Synopsis
Keywords: Liver, Machine Learning/Artificial Intelligence
Multi-task learning has been widely used for jointly tumor segmentation and classification. Uncertainty estimation of the subtask weight coefficient in multi-task learning has been investigated. However, due to the presence of noise in medical image, data uncertainty will affect the performance of multi-task learning. In addition, model uncertainty has not been conducted for multi-task learning. In this work, we propose a triplet-uncertainty in multi-task deep learning network (TU-MTL), simultaneously considering the uncertainty estimation of subtask weight coefficient, data uncertainty estimation and model uncertainty estimation. Experimental results of clinical hepatocellular carcinoma (HCC) demonstrate the effectiveness of the proposed method.Introduction
Hepatocellular carcinoma (HCC) is the third leading cause of cancerous death in the world1, and the microvascular invasion (MVI) has been demonstrated to the key factor for prognosis2. Noninvasive imaging based on Gd-EOB-DTPA in the hepatobiary phase (HBP) has been reported to be effective for MVI prediction3,4. Tumor segmentation, classification and uncertainty estimation are important procedures in machine learning-based diagnosis. Multi-task learning has been widely applied to realize tumor recognition, segmentation and classification, and achieved better performance than single task. Recently, uncertainty estimation of the subtask weight coefficient in multi-task learning has been investigated to balance the contribution of different tasks5,6. However, due to the presence of noise in medical image, data uncertainty7 will inevitably affect the performance of multi-task learning. In addition, model uncertainty8,9 has not been conducted for multi-task learning to demonstrate the reliability of prediction. In this work, we propose a triplet-uncertainty in multi-task deep learning network (TU-MTL), simultaneously considering the uncertainty estimation of subtask weight coefficient, data uncertainty estimation and model uncertainty estimation for segmentation and MVI prediction of HCC.Materials and Methods
137 HCC from 137 patients from January 2017 to September 2021 were included in this study. MRI was performed with 3.0T system (Achieva, Philips Healthcare, Netherlands), and contrast-enhanced MR images include arterial phase (AP), portal vein phase (PVP), delayed phase (DP) and hepatobiliary phase (HBP). The protocol parameters are set as follows: TR/TE = 3.1ms/1.51ms, 304 × 239 matrix, 5-mm slice thickness. The 3D Region of Interest (ROI) of the lesion was outlined by experienced clinicians in the HBP. Among them, 88 cases had MVI-, and 49 cases were pathologically diagnosed as MVI+. The schematic diagram of the proposed triplet-uncertainty based TU-MTL is shown in Figure 1. The orange part in the upper part of Figure 1 is the network backbone of the 3D MVI classification task, and the blue part in the lower part is the network backbone of the 3D tumor segmentation task. The two backbones are mainly based on 3D U-Net and VGG. First, we innovatively introduced data uncertainty estimation in multi-task learning. By learning the mean and variance of features at the same time, we make the features of samples of the same category more compact and further separated samples of different categories. Second, we construct corresponding model uncertainty estimates for each subtask, which improves the performance of multi-task learning, and provides the confidence level of prediction results for clinical reference. Finally, in order to balance the different contributions of the two tasks in the network optimization process, we introduce the task loss function weight based on uncertainty estimation to improve the performance of multi-task learning. For MVI classification task, accuracy (ACC), sensitivity (SEN), specificity (SPE), and area under the receiver operating characteristic curve (AUC) were used to evaluate the performance of the proposed method. We used two standard evaluation metrics, Dice coefficient (Dice) and Jaccard Index (JI). All experimental results were based on 5 times 4-folded cross validation.Results
As tabulated in Table 1, both data uncertainty and model uncertainty can clearly improve the performance of baseline both in MVI prediction and segmentation. By comparison, data uncertainty has more impact for improving the performance of tumor segmentation and classification compared with the model uncertainty. The combination of model uncertainty and data uncertainty can yield the best performance in both the two tasks. From the quantitative results shown in Table 2, the introduction of data uncertainty and model uncertainty can further improve the performance of task weighting uncertainty, classification accuracy and Dice performance by 10.51% and 2.2%, respectively. Furthermore, the performance of MTL shown in Table 2 obtained slightly better than those of the single task learning, indicating that the two tasks may be related and can promote each other. Overall, the proposed TU-MTL model obtained the best performance in the two tasks. In the segmentation task, Figure 2 shows the visualization effect of image segmentation guided by uncertainty under the framework of multi-task learning, which further indicates that the uncertainty constraint can improve the accuracy of the network.Discussion
To our knowledge, we are the first to simultaneously consider subtask weight coefficient via uncertainty estimation, data uncertainty and model uncertainty in MTL for significant performance improvement. Previous work mainly focused on the uncertainty of model weight coefficient5,6 and model uncertainty 8,9, which paid less attention to data noise. Data uncertainty7 is to capture the change of output caused by the noise of input data, that is, the uncertainty of data will follow the input of the model to interfere with the output of the model. Our results (Table 2) also verifies that data uncertainty has more impact for improving the performance of MTL compared with the model uncertainty and the subtask weight coefficient.Conclusion
In this work, we proposed a triplet-uncertainty based multi-task learning network (TU-MTL) model for tumor segmentation and classification. Experimental results demonstrate that the introduction of data uncertainty can significantly improve the performance of single and MTL, while the proposed TU-MTL model can yield the best performance. We hope the proposed method can be integrated into conventional MTL scenarios for performance improvement.Acknowledgements
This study is sponsored by the National Nature Science Foundation of China (No.81771920)References
1. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin.2021 Feb 4.doi:10.3322/caac.21660. (2021)
2. Erstad DJ, Tanabe KK. Prognostic and therapeutic implications of microvascular invasion in hepatocellular carcinoma. Ann Surg Oncol. 2019;26:1474-1493.
3. Lee S, Kim SH, Lee JE, Sinn DH, Park CK. Preoperative gadoxetic acid-enhanced MRI for predicting microvascular invasion in patients with single hepatocellular carcinoma. J Hepatol. 2017;67:526-34.
4. Feng S, Jia Y, Liao B, et al.: Preoperative prediction of microvascular invasion in hepatocellular cancer: a radiomics model using Gd-EOB-DTPA-enhanced MRI. Eur Radiol 2019; 29:4648-4659.
5. Kendall A, Gal Y, and Cipolla R. Multitask learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7482–7491.
6. Aıcha BenTaieb and Ghassan Hamarneh. Uncertainty driven multi-loss fully convolutional networks for histopathology. Intravascular Imaging and Computer Assisted Stenting, and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, 2017: 155–163.
7. Xu Y, Fang X, Li X, et al. Data uncertainty in face recognition. IEEE transactions on cybernetics. 2014; 44(10): 1950–1961.
8. Wang X, Tang F, Chen H, et al. Ud-mil: Uncertainty-driven deep multiple instance learning for OCT image classification. IEEE Journal of Biomedical and Health Informatics 2020;24(12): 3431– 3442.
9. Khanzhina N, Kashirin M, Filchenkov A. Monte Carlo concrete DropPath for epistemic uncertainty estimation in brain tumor segmentation. InUncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis 2021 Oct 1 (pp. 64-74). Springer, Cham.
Figures
