Introduction

The preoperative knowledge of the biological aggressiveness or malignancy degree of hepatocellular carcinoma (HCC) is significant for determining treatment decisions and predicting patient prognosis¹. Deep feature based on convolutional neural networks (CNN) has been shown to be promising to represent the biological aggressiveness of neoplasm^2-3. However, the slice thickness of MR imaging may remarkably degrade the clarity of 3D lesion images within coronal or sagittal views so as to influence the performance of lesion characterization. To alleviate the problem, we propose an end-to-end super-resolution and classification framework based on Generative adversarial networks (GAN) for improving the malignancy characterization of HCC.

Methods

The study was approved by the local institutional review board and the patient's informed consentswere waived. From October 2012 to October 2018, there were 115 cases in 112 subjects (101 males and 11 females, aged 51.09±11.53), ranging from 27 to 79 years) underwent HCC resection. Contrast-enhanced MR images were acquired for all subjects using 3.0 Tesla magnetic resonance scanner (signa excite HD 3.0T, GE health care, milwaukee, wi, usa). The image matrix dimension was 512×512× 92 and the voxel spacing was 0.46mm×0.46mm×2.2mm. Histological grading data for these 115 HCCs were obtained from clinical histological reports confirmed by histopathology, including 3 cases of Edmondson I, 51 cases of Edmondson II, 57 cases of Edmondson III and 4 cases of Edmondson IV. In clinical routine, Edmondson I and II correspond to low-grade and Edmondson III and IV correspond to high-grade. The framework of the proposed method is shown in Figure 1. The whole network consists of two components: a generative module G and a discriminative module D. The super-resolution subnetwork and the classification subnetwork are embedded in the generative module and discriminative module, respectively. The generator is trained with the super-resolution subnetwork to synthesize high-resolution image patches based on the low-resolution input patches corresponding to the coronal and sagittal views. Conversely, the discriminator is trained adversarially to identify whether the synthesized patches are high-resolution or low-resolution. Meanwhile, the classification subnetwork is jointly trained with the discriminator and to decide the high-grade or low-grade of the lesion. Furthermore, a self-attention mechanism⁴ is adopted to enable both the generator and the discriminator to efficiently capture relationships between widely separated spatial regions, yielding more discriminative features to further improve the performance of super-resolution and classification. The data set is randomly divided into two parts: training and verification set (75 HCCs) and fixed test set (40 HCCs). Receiver operating characteristic curve (ROC) and area under the curve (AUC) were used to assess the characterization performance.

Results

Table 1 showed the performance comparison of through plane patches (coronal or sagittal) for malignancy characterization using the conventional CNN method and the proposed method. It can be clearly observed that the characterization performance of the through-plane slices is rather low with the conventional CNN method⁵. Comparatively, the proposed method can significantly improve the characterization performance for each view. Table 2 showed the performance comparison of the proposed method and other methods in the conventional framework of multi-view convolutional networks for malignancy characterization of HCC, in which deep features separately extracted from three 2D orthogonal views by CNN were concatenated for classification. It can be found that the GAN+CNN method⁶ is superior to the traditional CNN method due to the adoption of GAN for more sample generation. Subsequently, the proposed method with the super-resolution (SR) module can significantly improve the characterization performance. Furthermore, the proposed method with the adoption of self-attention (SA) also obtain further improved performance. The ROC curves, loss curves, and accuracy curves of the performance comparison corresponding to Table 2 were plotted in Fig. 2.

Discussion

The proposed method can signifificantly improve the characterization performance for each view. There might be several reasons accounting for these improvements. On the one hand, the proposed method can yield more high-resolution samples so as to alleviate the problem of over-fitting due to the limited training data. On the other hand, the proposed method can extract multiple features from widely separated regions in high-resolution images for performance improvement of characterization. Subsequently, the proposed method with the super-resolution module can significantly improve the characterization performance. This is attributed to the adoption of super-resolution, which can make the GAN generate more high-resolution images. Furthermore, the proposed method with the adoption of self-attention also shows that further improved performance can be obtained because of the aggregation of multiple features from widely separated regions. Note that the loss curve of the conventional CNN method has a unwarping shape, and such slight over-fitting might be caused by the limitation of training data. Comparatively, the proposed method can remarkably alleviate the risk of over-fitting due to the adoption of GAN with super-resolution and self-attention to generate more high-quality samples.

Conclusion

An end-to-end super-resolution and classification framework based on conditional adversarial networks (CAN) is proposed to alliviate the problem of slice thickness of MR for improving the malignancy characterization of HCC. Experimental results of clinical HCCs demonstrate the superior performance of the proposed method compared with other methods. Our experiments for MRI super-resolution and classification also give insight to design proper network architectures for lesion characterization in clinical practice.

Acknowledgements

This research is supported by the grant from National Natural Science Foundation of China (NSFC: 81771920).