Introduction

Since the inception of machine learning, especially convolutional neural networks - they have been widely used in various fields of image processing – including in medical imaging. CNNs trained in a supervised manner to classify various diseases have shown their immense potential¹; however, they typically come with the so-called “black-box” problem – as the complex reasoning done by such models are difficult to comprehend by humans. To address this problem, several post-hoc interpretability techniques can be applied^2,3. On the other hand, CNNs can also be trained to segment relevant parts of the image, such as tumours. Typically, they are trained in a supervised manner and warrant a manually annotated dataset⁴. Such annotated datasets need to be prepared by clinicians, which is a time-consuming and daunting task. This work presents a deep learning-based classifier that inherently provides attributions on the input – to be able to understand which parts of the image had the highest impact on the model’s decision. Moreover, the potential of extending such attributions to perform segmentation without explicitly training the model with a manually segmented dataset to perform segmentation has also been shown.

Methods

A typical deep learning-based model classification model, like VGG⁵, ResNet⁶ or InceptionNet⁷, condenses the input image using strided-convolutions or pooling layers repeatedly. Finally, the most condensed form, which might typically be multiple features, but just a single pixel per feature, is then forwarded to a fully-connected layer to obtain the final prediction. Contrastingly, a typical segmentation or even reconstruction model, such as U-Net⁸ or ReconResNet⁹, condenses the input in a similar fashion to obtain a latent representation, commonly known as the contraction operation. Then the latent representation is expanded multiple times using transposed-convolution or interpolation+convolution or even using unpooling layers to finally obtain the original input size back – commonly known as the expansion operation. GP-UNet¹⁰ combined these two techniques into one and used it to learn segmentation with regression labels. The current research extends that for brain tumour classification and segmentation using a different segmentation model, ReconResNet⁹: GP-ReconResNet.
In this method (Fig. 1), initially, the input is given to a ReconResNet⁹ model (Fig. 2). This model provides n output images of the same size as the input, where n is the number of desired segmentation classes. On the contrary, the proposed method first sends the output to a global adaptive pooling operation to convert the full-size output into one pixel for each class and then sends it to the final fully-connected convolution layer, which also provides a single-pixel (neuron) for each class. Consequently, the network can be trained using classification image-level labels instead of segmentation labels. During inference, this adaptive max-pooling operation can be disabled, so the full-sized output of the transposed-convolution block can directly go to the fully-connected convolution layer to generate one output of the size of the input per class: known as the class activation maps or heatmaps, and by looking at them, the model’s focus areas can be understood without applying any post-hoc interpretability method. The adaptive max-pooling operation can be turned on again to obtain the classification result from the same model during inference. To generate the segmentation from this model, the generated heatmap is first suppressed (negative values set to 0) then thresholded with Otsu’s method with an offset of 0.1.
The proposed method has been evaluated on a brain tumour dataset containing 3064 T1-weighted contrast-enhanced MRIs, a mix of axial-coronal-sagittal orientations, from 233 patients with three kinds of brain tumours: meningioma (708 slices), glioma (1426 slices), and pituitary tumour (930 slices). The data was split into training, validation and testing sets, with a ratio of 60:15:25. Mixed-precision training was performed using weighted-cross-entropy loss and was optimised using the Adam optimiser for 150 epochs (until convergence) with a learning rate of 0.001

Results

The proposed GP-ReconResNet achieved an F1-score of 0.9 on the test set. Fig. 3 shows the scores obtained on the evaluation metrics and were compared against an adapted version of the GP-UNet¹⁰ for this task. Fig. 4 shows the explainability results and the corresponding segmentations. To focus on the tumour, suppressed versions of the heatmaps were created - focusing on the positive attributions only, where blue signifies the main focus areas of the network. Finally, the segmentation masks and their comparisons with the ground-truth masks are shown.

Discussion

The heatmaps make the GP-ReconResNet explainable, as they show the network’s focus area while predicting the tumour without applying any post-hoc techniques. Moreover, this research also shows the potential of performing tumour segmentation by post-processing the heatmaps - which does not require any manually annotated training data. Even though the segmentations are not perfect, it shows the potential of being part of a decision support system.

Conclusion

This research proposed the GP-ReconResNet, which uses ReconResNet as the backbone model, combined with global pooling to classify the input image. The network’s focus area (heatmap) can be obtained by removing the pooling, making this network explainable. By thresholding the heatmaps, segmentation results can be obtained without training separately for segmentation. The applicability of this network and its superiority over using UNet as the backbone model have been shown for the task of tumour segmentation.

Acknowledgements

This work was in part conducted within the context of the International Graduate School MEMoRIAL at OvGU (project no. ZS/2016/08/80646) and supported by the Wu Tsai Neurosciences Institute Seed Grant program.