A major hurdle for successful application and robust use of MRS in the clinic is the need for local technical expertise to prevent inappropriate interpretation of bad MR spectra¹. A previous study² suggested an automatic procedure based on support-vector-machine (SVM) classification to rate MRS tumor data as acceptable or unacceptable to eliminate the need for the human specialist and associated subjectivity. We tried to verify its performance on a much larger database and on upon finding poor performance, we aimed to pinpoint the reasons for this and to develop a more robust tool as listed in Table 1.

Spectra were downloaded from eTUMOUR³ and INTERPRET⁴ databases (Table 2). These spectra were acquired from 1.5T MR scanners (GE, Siemens, Philips) using standard PRESS and STEAM sequences at either short (20-32ms) or long echo times (135-144ms). Judgment of quality was originally done by three spectroscopists. If two of them had agreed, it was classified as acceptable.

All spectra were preprocessed using DMS software⁵. Briefly, it included phase-correction using the water reference, residual water removal, spectral alignment, SNR calculation and unit length normalization. Training and test sets for developing new classifiers included 1916 and 241 ¹H single-voxel spectra, respectively. The latter were also judged by 4 local spectroscopists (twice, each).

The method of Wright et al.² consists of extracting 5 independent components (ICs) and training a SVM classifier. All classifiers, like RUSBoost (random undersampling and boosting)⁶, and feature extraction methods have been implemented in MATLAB.

Wright’s method² performed well when applied to the same dataset as in Ref. 2, but when applied to the whole dataset (1916 spectra), the highest specificity (i.e. the proportion of bad spectra that are correctly identified) was around 45%. We tested potential reasons for this failure and potential modifications and new tools (see Table 1 for overview and Table 3 for the results). Neither separation of the data along echo time (TE) or according to database improved the classification performance considerably; neither did an increase of the number of ICs. Changing to more features, using sequential forward feature selection and principal component analysis (PCA), in combination with replacement of SVM by LDA (Linear discriminant analysis)⁷ clearly improved specificity but still not sufficiently.

To test whether the huge class imbalance in our dataset (87% acceptable) was the reason for bad performance, we used RUSBoost⁶ as classifier, which is known for handling class imbalance. Its specificity was highest for 9 ICs. Since ICs may be too simplistic, we added other features (see Table 1) and used treebagger (Bootstrap aggregation for ensemble of decision trees) in MATLAB to estimate the feature importance and arrived at 24 features, where SNR and skewness of the spectrum were ranked highly. With these features, the specificity improved to 83%. Another extension was to move to a three class grading (“poor” if just one spectroscopist accepted the spectrum). The specificity increased substantially to 92%.

To judge potential effects of inconsistent human labeling, 241 spectra were relabeled locally and intra- and inter-rater reliability, as well as consistency with the earlier raters was tested. Intra-rater variance was low (< 1 point mean difference in a 10 point quality score), as was inter-rater variability. The average agreement on classifications between the new consensus ratings and the individual spectroscopists was 92% (binary labels). The local spectroscopists agreed in 84% with the earlier consortium experts. However, they were systematically more critical in accepting spectra as reflected in the lower sensitivity (Table 4).

Finally, all local human reclassification cases and RUSBoost binary and multi-class classifiers were combined. Table 5 shows that the RUSBoost classifiers had a good performance, though not quite reaching human raters.

Classification appears to be easier when the classes are nearly balanced, especially when the class you are targeting is the rarer one. It is common to use classification accuracy as a first measure to determine the classifier performance but when the classes are imbalanced (such as 87% in one class), accuracy measures are misleading since they may just reflect the underlying class distribution even if the true accuracy is higher. It is better to look at sensitivity and specificity which give more insight into the classifier performance. Here, we trained the RUSBoost classifier that combats the imbalanced training data which showed improved specificity with an independent test-set of data compared to previously published methods. In addition, our results suggest that multiclass labels (3 classes) may be beneficial for improved classification performance. The final classifiers had a comparable performance to the panel of human expert spectroscopists in rejecting spectra.