4213

State of the ART (Adversarial Robust Training) to Reconstruct Clinically Relevant Features in Accelerated Knee MRI

Francesco Caliva¹, Victor Kaiyang Cheng², Rutwik Shah¹, Misung Han¹, Sharmila Majumdar¹, and Valentina Pedoia¹
¹University of California San Francisco, San Francisco, CA, United States, ²University of California Berkeley, Berkeley, CA, United States

Synopsis

We propose an Adversarial Robust Training (ART) strategy to overcome the problem with accelerated MRI models, which are prone to missing small yet clinically relevant features. We introduced small, difficult to reconstruct synthetic features to undersampled MRIs and encouraged their reconstruction through robust training. To assess generalizability of our technique to real world applications, we annotated morphological features relevant to musculoskeletal disease diagnosis on images in the FastMRI dataset and tested ART. Overall, the approach has potential to reduce network instability and improve reliability and fidelity in image reconstruction.

Introduction

Deep learning (DL) has demonstrated ability to preserve imaging quality and contrast in accelerated MRI;^1-8 nevertheless, the reconstruction of clinically relevant features is a source of concern regarding the applicability of DL models to image reconstruction. Studies have shown that DL models can introduce or omit abnormalities during reconstruction, based on the presence or absence of abnormal features in the training dataset.⁹ DL algorithms may fail in reconstructing relatively small, yet relevant, abnormalities such as subchondral osteophytes or meniscal tears, which are of outmost importance for clinical evaluation.¹⁰ They are not only required for grading of severity of lesions, but also decide further course of treatment.^11,12 Reconstruction artefacts could be the result of imperceptible perturbations in the sampling and/or image domain.^13,14 Current research has improved our understanding of DL reconstruction pitfalls; however, there is still a dearth in tailored methods that guarantee image reconstruction with preserved diagnostic features. In this study we leveraged adversarial robust training (ART) to develop DL reconstruction models that focus on these difficult to reconstruct features, and obtain reliable and faster imaging.

Method

The knee FastMRI dataset was used, pertaining to the original FastMRI challenge splits.^10,15 In the proposed image reconstruction framework, a DL network was first pre-trained to reconstruct accelerated single-coil MRIs; next, an adversarial attacker network was trained to identify small features, which if introduced to a to-be-reconstructed image, the pre-trained reconstruction network would have difficulty reconstructing these small features; last, difficult to reconstruct features were utilized in a robust training step, where the reconstruction network was retrained to focus on the region with these features and thereby improve sensitivity to small imaging features (Figure 1). A Unet¹⁶ architecture was employed as the baseline reconstruction model. To assess the effectiveness of ART on the reconstruction of real abnormalities, a musculoskeletal radiologist annotated clinically relevant features on 418 coronal proton density with fat suppression knee MR exams. The annotation dataset is the result of careful inspection of each MRI volume for cartilage, meniscus, bone marrow lesions (BML), and cysts based on the Whole Organ Magnetic Resonance Scoring scale.¹¹ Bounding boxes were introduced to define cartilage lesions and BML in 4 sub-regions of the tibio-femoral joint: medial and lateral femur and medial and lateral tibia. Additionally, bounding boxes were placed on lesions in the medial and lateral sub-regions of the meniscus and on cysts. Figure 2 is an example of the annotations depicting the presence of bone marrow lesion in lateral femur subjacent to articular cartilage. The manual annotation was used to measure reconstruction error in areas covered by abnormalities, and train a classifier to discriminate the presence or absence of the abnormalities in the reconstructed accelerated MRI. For this, we fine-tuned a SqueezeNet¹⁷, previously pre-trained on Imagenet, on 64x64 patches extracted from the fully-sampled images. 7644/2110 patches were randomly extracted from those samples in the training and validation sets, ensuring balance between patches including abnormalities and not. The fine-tuned model was utilized to assess the presence of abnormalities in patches extracted from the MRIs reconstructed with/without ART. Abnormality reconstruction quality was assessed by computing mean-squared-error (MSE) inside the annotated areas, between fully-sampled and reconstructed images; the SqueezeNet’s classification performance in terms of Sensitivity (Sn) and F1-score.

Results

Smaller features are more difficult to reconstruct, as demonstrated by the trend in Figure 3, with higher errors associated with the reconstruction of smaller abnormalities. Figure 4 is an example of reconstruction (A, B) in the presence of a bone marrow lesion at the lateral edge of the femur (C). The difference map in (D) shows a finer reconstruction of features using the proposed approach, which is a better representation of the fully-sampled MRI. Higher acceleration factor (AF) (8x) reconstructions benefitted more from ART than lower AF reconstructions (e.g. 2x or 6x), with a notable reduction in MSE for all compartments, except BML in the femoral lateral cartilage and BML in the medial femur (Table 1). For SqueezeNet’s classification performance, our proposed approach using reconstructed images (at 4x: Sn=89.74, F1=79.17; at AF 8x: Sn=85.71, F1=79.21) was more accurate for the identification of abnormality presence than the baseline UNet without ART (at AF 4x: Sn=90.23, F1=79.64; at AF 8x: Sn=85.30, F1=78.89).

Discussion

The application of ART resulted in a better preservation of abnormal signal changes in cartilage, and clearer visibility of cartilage lesions; crucially, these are features necessary for an accurate clinical interpretation of the image. Currently, a significant patient population with severe knee osteoarthritis receives appropriate treatment after having waited too long. In the event of missed clinical feature reconstruction, these patients would wait even longer for intervention, resulting in compromised quality of life.¹⁸ Besides concerns related to missing important features during reconstruction, there are doubts about the choice of appropriate reconstruction evaluation metrics.³ There is, however, a general consensus that evaluation metrics must provide, with high fidelity, a finer and reliable assessment of the reconstruction of relevant features to guarantee diagnostic power.

Conclusions

We improved the reconstruction of clinically relevant features in accelerated MRI using adversarial attacks and robust training. The annotation dataset utilized in this work is available upon contacting the authors.

Acknowledgements

This work was supported by the NIH/NIAMS R00AR070902 grant. We would like to thank Fabio De Sousa Ribeiro (University of Lincoln), Radhika Tibrewala (New York University) Madeline Hess and Koren Roach (UCSF) for the fruitful discussions.

References

¹ Recht, M.P., Zbontar, J., Sodickson, D.K., Knoll, F., Yakubova, N., Sriram, A., Murrell, T., Defazio, A., Rabbat, M., Rybak, L. and Kline, M., 2020. Using Deep Learning to Accelerate Knee MRI at 3 T: Results of an Interchangeability Study. American Journal of Roentgenology, 215(6), pp.1421-1429.

² Subhas, N., Li, H., Yang, M., Winalski, C.S., Polster, J., Obuchowski, N., Mamoto, K., Liu, R., Zhang, C., Huang, P. and Gaire, S.K., 2020. Diagnostic interchangeability of deep convolutional neural networks reconstructed knee MR images: preliminary experience. Quantitative Imaging in Medicine and Surgery, 10(9), p.1748.

³ Hammernik, K. and Knoll, F., 2020. Machine learning for image reconstruction. In Handbook of Medical Image Computing and Computer Assisted Intervention (pp. 25-64). Academic Press.

⁴ Schlemper, J., Qin, C., Duan, J., Summers, R.M. and Hammernik, K., 2019. $\Sigma $-net: Ensembled Iterative Deep Neural Networks for Accelerated Parallel MR Image Reconstruction. arXiv preprint arXiv:1912.05480.

⁵ Putzky, P. and Welling, M., 2019. Invert to learn to invert. In Advances in Neural Information Processing Systems (pp. 446-456).

⁶ Pezzotti, N., de Weerdt, E., Yousefi, S., Elmahdy, M.S., van Gemert, J., Schülke, C., Doneva, M., Nielsen, T., Kastryulin, S., Lelieveldt, B.P. and van Osch, M.J., 2019. Adaptive-CS-Net: fastMRI with adaptive intelligence. arXiv preprint arXiv:1912.12259.

⁷ Pezzotti, N., Yousefi, S., Elmahdy, M.S., Van Gemert, J.H.F., Schuelke, C., Doneva, M., Nielsen, T., Kastryulin, S., Lelieveldt, B.P., Van Osch, M.J. and De Weerdt, E., 2020. An Adaptive Intelligence Algorithm for Undersampled Knee MRI Reconstruction. IEEE Access, 8, pp.204825-204838.

⁸ Wang, P., Chen, E.Z., Chen, T., Patel, V.M. and Sun, S., 2019. Pyramid Convolutional RNN for MRI Reconstruction. arXiv preprint arXiv:1912.00543.

⁹ Cohen, J.P., Luck, M. and Honari, S., 2018, September. Distribution matching losses can hallucinate features in medical image translation. In International conference on medical image computing and computer-assisted intervention (pp. 529-536). Springer, Cham.

¹⁰ http://FastMRI.med.nyu.edu/

¹¹ Peterfy, C.G., Guermazi, A., Zaim, S., Tirman, P.F.J., Miaux, Y., White, D., Kothari, M., Lu, Y., Fye, K., Zhao, S. and Genant, H.K., 2004. Whole-organ magnetic resonance imaging score (WORMS) of the knee in osteoarthritis. Osteoarthritis and cartilage, 12(3), pp.177-190.

¹² McAlindon, T.E., Bannuru, R., Sullivan, M.C., Arden, N.K., Berenbaum, F., Bierma-Zeinstra, S.M., Hawker, G.A., Henrotin, Y., Hunter, D.J., Kawaguchi, H. and Kwoh, K., 2014. OARSI guidelines for the non-surgical management of knee osteoarthritis. Osteoarthritis and cartilage, 22(3), pp.363-388.

¹³ Cheng, K., Calivá, F., Shah, R., Han, M., Majumdar, S. and Pedoia, V., 2020, September. Addressing the false negative problem of deep learning MRI reconstruction models by adversarial attacks and robust training. In Medical Imaging with Deep Learning (pp. 121-135). PMLR.

¹⁴ Antun, V., Renna, F., Poon, C., Adcock, B. and Hansen, A.C., 2020. On instabilities of deep learning in image reconstruction and the potential costs of AI. Proceedings of the National Academy of Sciences.

¹⁵ Zbontar, J., Knoll, F., Sriram, A., Murrell, T., Huang, Z., Muckley, M.J., Defazio, A., Stern, R., Johnson, P., Bruno, M. and Parente, M., 2018. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839.

¹⁶ Ronneberger, O., Fischer, P. and Brox, T., 2015, October. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.

¹⁷ Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J. and Keutzer, K., 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size. arXiv preprint arXiv:1602.07360.

¹⁸ Ghomrawi, H.M.K., Mushlin, A.I., Kang, R., Banerjee, S., Singh, J.A., Sharma, L., Flink, C., Nevitt, M., Neogi, T. and Riddle, D.L., 2020. Examining timeliness of Total knee replacement among patients with knee osteoarthritis in the US: results from the OAI and MOST longitudinal cohorts. JBJS, 102(6), pp.468-476.

Figures

Figure 1: Proposed image reconstruction framework. During training, an attacker introduces difficult to reconstruct features (δ) to under-sampled images (x): it maximizes a reconstruction error between x and a fully-sampled image (y), in the presence of δ, given a network parametrized by θ. Next, network's parameters (θ) are updated to minimize a robust training loss, which includes a reconstruction error and a robust training term that further penalizes reconstruction errors in the regions including abnormalities.

Figure 2: Example of clinically-relevant anomalies, which were annotated, as part of this study, in the knee FastMRI dataset: A) cartilage lesion in lateral femur; B) cartilage lesions in the medial femur and tibia.

Figure 3: Smaller features are more difficult to reconstruct. Correlation between the reconstruction loss and abnormality size, the loss was computed in the region occupied by the abnormality.

Figure 4 Abnormality reconstruction improved using our proposed ART strategy (based on the SqueezeNet classifier). Reconstruction of 4x undersampled MRI obtained using A) baseline Unet, B) proposed approach; C) fully-sampled MRI. D) Difference map between A) and B), shows clearer visibility of cartilage lesions and a better preservation of abnormal signal changes.

Table 1: Reconstruction mean-squared-error (MSE) inside regions containing abnormalities using a baseline UNet and our proposed ART approach, on 8x undersampled MRI.

Proc. Intl. Soc. Mag. Reson. Med. 29 (2021)

4213