4195
Jointly grading and microvascular invasion predicting of hepatocellular carcinoma based on multi-task deep learning1School of Medical Information Engineering, Guangzhou University of Chinese Medicine, Guangzhou, China, 2Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China, 3Department of Radiology, Guangdong General Hospital, Guangzhou, China
Synopsis
The pathological grade and microvascular invasion (MVI) of hepatocellular carcinoma (HCC) are two key factors related to the patient's prognosis. Previous studies usually predict these two factors separately based on medical images. In this study, we propose an end-to-end multi-task deep learning network to simultaneously predict the MVI and grading information. Specifically, we are the first to demonstrate that these two tasks are related and can promote each other in the framework of multi-task deep learning. Experimental results of HCC in Contrast-enhanced MR demonstrate the effectiveness of the proposed method, outperforming the single task learning.
Introduction
Microvascular invasion (MVI) of hepatocellular carcinoma (HCC) is an important predictor of poor prognosis after curative liver resection or transplantation 1. In addition, differentiation of HCC is also considered to be an important predictor of patient prognosis 2. However, information about MVI status and tumor differentiation is usually not available before surgery, thus limiting their clinical application in decision-making. Many studies focus on imaging findings of preoperative imaging to predict MVI and grading of HCC 3. Recently, 2D and 3D Convolutional neural network (CNN) based on contrast-enhanced MR and diffusion-weighted MR have been used in HCC grading and MVI prediction 4-5. However, previous studies generally predict the two factors separately based on medical images. One disadvantage is that it is cumbersome to conduct the two individual tasks. Furthermore, the relationship between the two factors has not been investigated for better prediction. In this study, we propose a multi-task deep learning network to jointly predict the MVI and grading information of HCC.Materials and Methods
This retrospective study was approved by the local institutional review board and the informed consent of patients was waived. There were one hundred and twelve consecutive patients with 117 hisotologically proven HCCs after surgical resection from October 2012 to October 2018 included in this retrospective study. Gd-DTPA-enhanced MR imaging were performed with a 3.0T MR scanner (Signa Excite HD 3.0T, GE Healthcare, Milwaukee, WI, USA). The ground truth of histological grade and MVI information of HCC were pathological diagnosed based on surgically resected specimens. Of the 117 lesions, 73 were pathologically determined as the absence of MVI, while 44 were pathologically determined as the presence of MVI. The 117 HCCs were classified into four Edmonson grades, four of which was grade I, fifty-one of grade II, fifty-seven of grade III, and four of grade IV. Note that Edmondson grades I and II were low-grade (54 HCCs) and Edmondson grades III and IV were high-grade (63 HCCs). To overcome the lack of large training data sets to apply deep learning algorithms to clinical medical data, image resampling method is adopted for data augmentation to extract numerous 3D patches within limited number of tumors. The proposed multi-task deep learning network for jointly grading and MVI predicting was shown in Figure 1. First, we adopt two CNN networks to conduct the grading and MVI prediction in parallel using the arterial phase images. Then, we design a total loss function to combine the loss functions corresponding to the two tasks with an equal weighting strategy or an uncertainty weighting strategy 6. Finally, we compare the performance of multi-task deep learning and the single task learning. The dataset was subdivided into the training set (77 HCCs) and the test set (40 HCCs). Evaluation metric of accuracy, sensitivity, specificity and AUC values was used to assess the performance of different methods. The proposed multitask learning method was implemented using “Tensorflow”.Results
Table 1 showed the performance comparison of the single task learning and the multi-task learning with two different loss weighting strategies for grading and MVI prediction of HCC using the arterial phase images. It can be found that multi-task learning with two different loss weighting strategies yielded better performance than those of single task learning for both grading and MVI prediction. Further, the weighting strategy of uncertainty obtained better performance than that of the equal weight in the framework of multi-task learning. Figure 2 showed accuracy curves, loss curves and ROC curves of single task learning and multi-task learning with two different loss weighting strategies. It can be observed that with the increase of iteration, the multi-task learning with uncertainty weighting strategy finally yielded the highest accuracy values and the lowest loss values. Figure 3 showed the visualization of saliency maps using Grad-CAM 7, which reflected the dominant features in specific spatial areas. The single task model’s focus on the tumor is relatively scattered, while the multi-task models focus more on the main parts of the tumor. Correspondingly, Figure 4 depicted the output probability values of the three deep learning models for grading and MVI prediction for the HCC. The single task model made incorrect prediction for both grading and MVI, while the multi-task learning models made correct predictions with high output possibility.Discussion
This study might be the first work to jointly predict the MVI and grading information of HCC using the multi-task deep learning network. Our experiment demonstrates that grading and MVI prediction of HCC can be jointly predicted in an end-to-end deep learning network due to the observation that the grading and MVI information have been separately predicted from MR images 3-5. Furthermore, the present study demonstrates that the multi-task learning obtains better performance than the single task learning, indicating that the two tasks are related and can promote each other to yield better prediction performance. Finally, the weighting strategy of uncertainty obtained better performance than that of the equal weight, indicating that adjusting the relative weight by automatically learning the weights of different tasks is significant for multi-task learning 6.Conclusion
The present study demonstrates that the MVI and grading information of HCC can be jointly predicted by the multi-task deep learning network.Acknowledgements
This research is supported by the grant from National Natural Science Foundation of China (NSFC: 81771920).References
1 Erstad DJ, Tanabe KK. Prognostic and therapeutic implications of microvascular invasion in hepatocellular carcinoma. Ann Surg Oncol. 2019;26:1474-1493.
2 Jonas S, Bechstein WO, Steinmuller T et al. Vascular invasion and histopathologic grading determine outcome after liver transplantation for hepatocellular carcinoma in cirrhosis. Hepatology 2001; 33 (5):1080-1086.
3 Witjes CDM, Willemssen EJA, Verheij J. Histological Differentiation Grade and Microvascular Invasion of Hepatocellular Carcinoma Predicted by Dynamic Contrast-enhanced MRI. J Magn Reson Imaging 2012;36:641-647.
4 Zhou W, Wang G, Xie G, Zhang L. Grading of hepatocellular carcinoma based on diffusion weighted images with multiple b-values using convolutional neural networks. Medical Physics 2019;46(9):3951-3960.
5 Men S, Ju H, Zhang L, Zhou W. Prediction of Microvascular invasion of hepatocellular carcinoma with Contrast-enhanced MR using 3D CNN and LSTM. IEEE International Symposium on Biomedical Imaging, 2019:810-813.
6 Kendall A, Gal Y, Cipolla R. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2018:7482-7491.
7 Selvaraju RR, Cogswell M, Das A, et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2016:618-626.
Figures
