Introduction

Quantifying uncertainty in machine learning (ML) models for diagnostic imaging is critical for both human and machine decision making. Quantitative MRI (qMRI) aims to estimate biophysical tissue properties while reducing subjectivity and improving reproducibility in the decision-making process. In addition, qMRI utilizing ML enables rapid and simultaneous multi-parametric mapping with accelerated phase-cycled balanced steady-state free precession (pc-bSSFP) data¹. However, continuous efforts are needed to test algorithms for evaluating model uncertainty. Quantile regression (QR) offers an effective and easily interpretable method for quantifying uncertainty in regression models². Unlike conventional regression, QR provides upper and lower quantile estimates, creating prediction intervals that do not rely on any distributional assumptions about the target data, such as Gaussianity. In addition, conformal prediction can be applied on pre-trained regression or classifier models to provide finite sample coverage guarantees for the estimated prediction intervals^3–5. In this work we propose to use conformalized quantile regression (CQR) deep neural networks (DNNs)⁵ for in silico and in vivo quantile mapping of T1 and T2 relaxometry parameters, along with B1⁺ and ∆B0 field estimates from pc-bSSFP data. We demonstrate that predicted confidence intervals of CQR models reflect model uncertainty depending on the training data and explore them as an additional information source.

Methods

3D pc-bSSFP whole-brain data (TR/TE/α_nom=4.8ms/2.4ms/15°) were acquired from six healthy subjects, using 12 phase-cycles, linearly distributed in a 2π range. For accelerated imaging, those data were retrospectively undersampled along the phase-cycle dimension by a factor of 2 and 3, retaining 6 and 4 phase-cycles, respectively. Target T1, T2, B1⁺, and ∆B0 data were derived from inversion recovery turbo-spin-echo, spin-echo, TurboFLASH with preconditioning RF pulse, and dual-echo gradient-echo scans, respectively¹.
In vivo training data consisted of ~550,000 brain voxels from four subjects. Whole-brain data from two additional subjects were used for testing. In silico pc-bSSFP signals were generated by uniformly sampling the target parameters in specified ranges across 400,000 samples: T1 (ms) [400, 3000], T2 (ms) [15, 500], B1⁺=α_act/α_nom (actual/nominal flip angle) [0.7, 1.3], ∆B0 (rad) [0, 2π]. TR, TE, flip angle, and number of phase-cycles were identical to those used for the in vivo scans.
For DNN training, both QR and mean regression utilized the magnitude and phase of the pc-bSSFP data voxelwise as input, with T1, T2, B1⁺, and ∆B0 as targets. The trainings were conducted in silico (12 phase-cycles) and in vivo (12, 6, and 4 phase-cycles). QR DNNs estimated an upper and lower quantile function, effectively doubling the number of target parameters (Figure 1). A multilayer perceptron with three hidden layers (128, 64, 32 neurons), RELU activation function, initial learning rate of 1e-3, batch size of 128, 300 epochs, and ADAM optimization was used for DNN training. The learning rate was halved when the validation loss did not improve over 10 epochs. This work opted for 90% coverage (miscoverage rate α=10%), resulting in the fitting of 5% and 95% quantiles. This was achieved using the $$$\textit{pinball loss}$$$ for QR⁶

$$Loss(y,\widehat{y})=\begin{cases}\alpha(y-\widehat{y})&\text{if}\hspace{0.2cm}y-\widehat{y}>0,\\(1-\alpha)(\widehat{y}-y)&\text{otherwise}\end{cases}$$

with target ($$$y$$$) and prediction ($$$\widehat{y}$$$). In comparison to the mean regression training data (train/validation/test: 60/20/20%), a calibration set was preserved for the conformalization step subsequent to QR (train/validation/calibration/test: 60/20/10/10%)^4,5. Performance was validated by the parameters coverage rate $$$C=\frac{\sum_{i=1}^{n}\mathbb{I}\left(y_i\geq\widehat{y}_{lo,i}\hspace{0.1cm}\text{and}\hspace{0.1cm}y_{i}\leq\widehat{y}_{hi,i}\right)}{n}\times100$$$, where $$$\widehat{y}_{lo}$$$ and $$$\widehat{y}_{hi}$$$ represent the lower and upper quantile predictions, respectively.

Results

Figure 2 shows the mean (blue) as well as the lower (green) and upper (red) quantile predictions for in silico and in vivo DNNs trained with 12 bSSFP phase-cycles as input. The mean predictions fall within the lower and upper quantiles and predictions exhibit increased confidence intervals for target parameters with a low number of training samples. Furthermore, coverage rates in Table 1 demonstrate that the quantiles were successfully trained in silico and in vivo given 90% coverage rate. Deviations in the coverage rate among whole-brain test subjects may reflect discrepancies in the distribution of training and testing data or overfitting. The in vivo mean parameter predictions from 12 pc-bSSFP data displayed in Figure 3 are in high agreement with the measured ground truth and spatial uncertainty estimates can be represented as confidence interval maps ($$$\widehat{y}_{hi}$$$-$$$\widehat{y}_{lo}$$$). In addition, Figure 4 demonstrates the use of such confidence interval maps for quantifying uncertainty using accelerated input data.

Discussion and Conclusion

Distribution-free uncertainty estimation in quantile regression was achieved by changing the loss function, thereby negating the need for costly post-training uncertainty quantification. The confidence interval defined by upper and lower quantiles reflects model uncertainty. Future work will focus on improving generalization performance on unseen subject data and investigate use cases for CQR in qMRI, e.g. abnormality detection in patients.

Acknowledgements

This work is supported by DFG HE9297/1-1 (German Research Foundation)

References

1. Heule R, Bause J, Pusterla O, Scheffler K. Multi‐parametric artificial neural network fitting of phase‐cycled balanced steady‐state free precession data. Magn Reson Med. 2020;84(6):2981-2993. doi:10.1002/mrm.28325

2. Koenker R, Bassett G. Regression Quantiles. Econometrica. 1978;46(1):33. doi:10.2307/1913643

3. Vovk V, Gammerman A, Shafer G. Algorithmic Learning in a Random World. Cham: Springer International Publishing; 2022. doi:10.1007/978-3-031-06649-8

4. Angelopoulos AN, Bates S. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. December 2022. http://arxiv.org/abs/2107.07511.

5. Romano Y, Patterson E, Candes E. Conformalized Quantile Regression. In: Wallach H, Larochelle H, Beygelzimer A, Alché-Buc F d’, Fox E, Garnett R, eds. Advances in Neural Information Processing Systems. Vol 32. Curran Associates, Inc.; 2019

6. Steinwart I, Christmann A. Estimating conditional quantiles with the help of the pinball loss. Bernoulli. 2011;17(1). doi:10.3150/10-BEJ267