Multi-parametric MRI (mp-MRI) is a promising imaging modality for the detection and grading of prostatic carcinoma (PCa) [1], but current mp-MRI scoring systems, such as PI-RADS v2 [1], are generally subjective and have a limited ability to distinguish between indolent and clinically significant (CS) PCa. Automatic classification algorithms to improve the current scoring systems are an active research area [2] but typically require precise suspicious lesion boundaries, anatomical information, and carefully designed handcrafted features. Deep learning, a novel machine learning method, has recently garnered attention because of its superior performance in image recognition. However, applying deep learning to medical imaging diagnosis is non-trivial due to its requirements for massive clinical datasets for training. In this work, we propose an automatic classification method that can overcome the limitation of small clinical datasets by combining deep features extracted from a pre-trained convolutional neural network (CNN), known as OverFeat [3], and conventional statistical features [4].

With IRB approval, a study cohort of 68 consecutive men who underwent 3.0T mp-MRI (Skyra and Trio, Siemens Healthcare) prior to radical prostatectomy was included (6/2010–9/2014). Each mp-MRI study, including T2-weighted (T2w), DWI and DCE images, was correlated with whole mount histopathology by experienced GU pathologists, and lesions were matched with respect to location, size and Gleason Score (GS). Indolent PCa cases were defined as GS smaller than seven (GS ≤ 6) and CS PCa ones were larger or equal to seven (GS ≥ 7). A total of 102 lesions were identified, including 48 indolent and 54 CS sub-cohorts.

Figure 1 illustrates our proposed method. The middle slice of regions of interest (ROIs) suspicious for PCa (annotated by squares) in T2w, ADC and DCE (K^trans) images are interpolated and rescaled to 512×512 pixels (“Pre-processing”). Two training stages are used to obtain the final decision. In the first stage, OverFeat [3], is used to overcome the small sample size [5]. Deep features from the last convolutional layer (layer 21 in OverFeat) are employed for each T2w (f_T2), ADC (f_ADC) and K^trans (f_K) image separately. Three linear SVM classifiers are then adopted to train f_T2, f_ADC and f_K respectively. In the second stage, the decision values from the three classifiers are combined with six statistical features to train a Gaussian radial basis function (RBF) kernel SVM classifier, which outputs the final decision (indolent vs. CS). Statistical features (f_s) include skewness-of-intensity histogram in T2w images, average ADC value, lowest 10th percentile ADC value, average K^trans, highest 10th percentile K^trans value, and ROI size in T2w images [1].

The training process is designed as follows. First, the whole dataset is randomly divided into five folds of similar size. One fold is then selected as test set IMAGE_test and the other four folds are training set IMAGE_train. After this, IMAGE_train is equally and randomly divided into two phases, IMAGE_train1 and IMAGE_train2. IMAGE_train1 is employed to train the three linear SVMs in stage 1 with leave-one-out cross-validation for selecting the optimal parameters. Once trained, the three trained classifiers are applied to IMAGE_train2, generating prediction score vectors. With the prediction scores and f_s, IMAGE_train2 is used to train the RBF SVM in stage 2 and the performance of the prediction is measured on IMAGE_test. The whole procedure is repeated five times (known as five-fold cross-validation), where each fold is used as a test set once. The final classification results are the average performance of the five-fold cross-validation.

To evaluate the effectiveness of the proposed system, we built four comparison classification models. Four different SVMs are built using only f_s, f_T2, f_ADC or f_K, respectively. The performance of these models are also evaluated using five-fold cross validation using the whole dataset. The results are measured using the mean areas under curve, mean accuracy, mean sensitivity and mean specificity (Table 1). Figure 2 shows the receiver operating characteristic (ROC) curves. The proposed model achieves the highest performance compared to other models. The standard model using six statistical features achieves the lowest performance mainly due to lack of accurate lesion contours and anatomical-location-specific training. The results also suggest that deep features significantly contribute to the improvement of the performance.We present a novel and effective framework for improved mp-MRI-driven classification of indolent vs. clinically significant PCa, combining deep learning and conventional statistical features. The proposed model achieves significantly higher accuracy on distinguishing indolent vs. clinically significant PCa without requiring precise segmentation of lesion boundaries nor location-specific training. Our method has the potential to improve subjective radiologist based performance in the detection and grading of suspicious areas on mp-MRI.