Introduction:

Recently, deep unfolding methods^1,2 had been exploited in the field of MRI reconstruction beyond black-box end-to-end networks, due to their better performance and be more explainable. Although these unfolding methods can produce excellent reconstructed MR images, the networks themself are often non-robust³ and lead to severe artifacts despite only some perturbations mixed in the sampled k-space data.
To reduce the artifacts in reconstructed MR images, we propose a deep adversarial-equilibrium learning method. As a consensus, modeling artifacts in MR image is often challenging and inaccurate, therefore, the perturbation which leads to the worst reconstructed result is taken consideration into adversarial learning for improving the robustness to artifacts. Furthermore, the idea of deep equilibrium (DEQ) networks^4,5 is used to ensure the convergence of adversarial learning networks.

Method

Traditional unfolding methods in MRI reconstruction can be modeling as following
$$ \min \|\boldsymbol{M} * f f t(\boldsymbol{c s m} * \boldsymbol{x})-\boldsymbol{x}\|_2^2+\lambda R(\boldsymbol{x}) $$
where $$$\boldsymbol{M}$$$ denotes the binary matrix of under-sampling pattern, $$$ f f t$$$ represents operator of Fast Fourier Transform, $$$\boldsymbol{csm}$$$ is a mapping estimation of different channels and transform the image-domain image into a multi-channel estimated image via Hadamard product. $$$ R(\boldsymbol{x})$$$ represents regularization constraint for describing prior of MR images, $$$\lambda$$$ is the penalty of regularizer. $$$\boldsymbol{x}$$$ and $$$\boldsymbol{b}$$$ represents MR image to be reconstructed and under-sampled multi-channel k-space signal respectively.
To solve this model, iteration process in unfolding methods often can be formulated as
$$\boldsymbol{x}^{k+1}=\operatorname{Net}\left(\boldsymbol{x}^k-\boldsymbol{A}^T(\boldsymbol{A} \boldsymbol{x}-\boldsymbol{b})\right)$$
where $$$\operatorname{Net}$$$ can be different flexible network, $$$\boldsymbol{A}$$$ represents forward operator and $$$\boldsymbol{Ax}$$$ can be described as $$$\boldsymbol{M} * f f t(\boldsymbol{c s m} * \boldsymbol{x})$$$ , correspondingly, $$$\boldsymbol{A}^T$$$ denotes the adjoint operator.
Adversarial learning⁶ plays an essential role in reconstruction against artifacts, specifically, we consider the worst condition in reconstruction, which is formulated as
$$ \min _\theta \max _\varphi\left\|\boldsymbol{A} f_\theta\left(\boldsymbol{A} \boldsymbol{x} ; \boldsymbol{b}+\delta_{\varphi}(\boldsymbol{b})\right)-\boldsymbol{b}\right\|+\left\|f_\theta(\boldsymbol{A} \boldsymbol{x} ; \boldsymbol{y})-\boldsymbol{x}_{g t}\right\| \\ \text { s.t. }\left\|\delta_{\varphi}(\boldsymbol{b})\right\| \leq \varepsilon $$
$$$ f_\theta$$$ represents the reconstruction net, and $$$\delta_{\varphi}$$$ is a neural network to learn the worst perturbation to disrupt the reconstruction result as much as possible.
DEQ can be simply described as $$$\boldsymbol{x}^{\infty} = f\left(\boldsymbol{x}^{\infty}, \boldsymbol{b}\right)$$$, superscript represents the number of iterations, Anderson acceleration is utilized to figure out this the fixed point separately for example in a batch, the mapping relationship from $$$\boldsymbol{x}^{k}$$$ to $$$\boldsymbol{x}^{k+1}$$$ denotes as $$$ f$$$
$$\boldsymbol{x}^{k+1}=\sum_1^m \alpha_i * f\left(\boldsymbol{x}^{k-i+1}, \boldsymbol{b}\right)$$
where $$$\boldsymbol{\alpha}=\min _{\boldsymbol{\alpha}}\|\boldsymbol{G} \boldsymbol{\alpha}\|_2^2$$$, subject to $$$\mathbf{1}^T \boldsymbol{\alpha}=1$$$, and $$$\boldsymbol{G}$$$ is a matrix whose $$$\mathrm{i}$$$-th column is the (vectorized) residual $$$f\left(\boldsymbol{x}^{k-i}, \boldsymbol{b}\right)-\boldsymbol{x}^{k-i}$$$, with $$$i=0, \cdots, m-1$$$. Furthermore, it can also extend the iterations to either a generalized updating form
$$\boldsymbol{x}^{k+1}=(1-\beta) \sum_1^m \alpha_i * \boldsymbol{x}^{k-i+1}+\beta \sum_1^m \alpha_i * f\left(\boldsymbol{x}^{k-i+1}, \boldsymbol{b}\right)$$
according to chain rule, $$$\frac{\partial l}{\partial \theta}=\left(\frac{\partial\boldsymbol{x}^{\infty}}{\partial \theta}\right)^T \frac{\partial l}{\partial\boldsymbol{x}^{\infty}}$$$, and it can be extended as
$$\frac{\partial l}{\partial \theta}=\left(\frac{\partial f_\theta\left(\boldsymbol{x}^{\infty}, \boldsymbol{b}\right)}{\partial \theta}\right)^T\left(\boldsymbol{I}-\left.\frac{\partial f_\theta(\boldsymbol{x}, \boldsymbol{b})}{\partial \boldsymbol{x}}\right|_{x=x^{\infty}}\right)^{-T} \frac{\partial l}{\partial \boldsymbol{x}}$$
We conducted experiments on the public fastMRI dataset⁷, and randomly select 28 individuals for training, 4 individuals for test. The proposed method can be applicable to different unfolding networks, ISTA-net⁸ is taken as a basic method for MRI reconstruction in our experiment, and four fully- connected layers is utilized as the perturbation net. Framework of the proposed model is shown in Figure.1.

Result

To assess the robustness of different methods, we added different levels of simulated artifacts⁹, specifically, from 0 to 15% Frobenius norm of under-sampled k-space signal, to the under-sampled multi-channel MR signals in k-space at 4× acceleration shown in Figure.1. Furthermore, ISTA-net⁸ (baseline), ISTA-net with the worst perturbation adversarial learning, ISTA-net with DEQ architecture, and ISTA-net with both two modules are compared in the experiment, shown in Figure.2 and Figure.3, to display the improvement of quality and robustness of reconstructed MR image brought by adversarial learning and DEQ architecture. Furthermore, we add results of simple IFFT reconstruction by fully-sampled k-space signal with different levels of simulated artifacts to eliminate the influence of the sampling pattern in Figure.2.

Conclusion

The result demonstrates that the proposed method, which considers the worst perturbation in k-space, improve the performance both in quality and robustness of reconstructed MR images.

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (61871373, 62271474, 81830056, 61771463, U1805261, 81729003, 81901736, 12026603, 62206273 and 81971611, 12001180), the National Key R&D Program of China (2023YFB3811400)，the Strategic Priority Research Program of Chinese Academy of Sciences (XDB25000000 and XDC07040000), the High-level Talent Program in Pearl River Talent Plan of Guangdong Province (2019QN01Y986), the Key Laboratory for Magnetic Resonance and Multimodality Imaging of Guangdong Province (2023B1212060052), the Science and Technology Plan Program of Guangzhou (202007030002), the Key Field R&D Program of Guangdong Province (2018B030335001), the Shenzhen Science and Technology Program, Grant Award (JCYJ20210324115810030), and the Shenzhen Science and Technology Program (Grant No. KQTD20180413181834876, and KCXF20211020163408012).

References

[1] Y. Yang, J. Sun, H. Li, and Z. Xu, “ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 3, pp. 521–538, Mar. 2020, doi: 10.1109/TPAMI.2018.2883941.

[2] D. Liang, J. Cheng, Z. Ke, and L. Ying, “Deep Magnetic Resonance Image Reconstruction: Inverse Problems Meet Neural Networks,” IEEE Signal Process. Mag., vol. 37, pp. 141–151, 2020, doi: 10.1109/MSP.2019.2950557.

[3] V. Antun, F. Renna, C. Poon, B. Adcock, and A. C. Hansen, “On instabilities of deep learning in image reconstruction and the potential costs of AI,” Proc. Natl. Acad. Sci. U. S. A., vol. 117, no. 48, pp. 30088–30095, 2020, doi: 10.1073/pnas.1907377117.

[4] A. Raj, Y. Bresler, and B. Li, “Improving robustness of deep-learning-based image reconstruction,” 37th Int. Conf. Mach. Learn. ICML 2020, vol. PartF16814, pp. 7888–7898, 2020.

[5] S. Bai, J. Zico Kolter, and V. Koltun, “Deep equilibrium models,” Adv. Neural Inf. Process. Syst., vol. 32, no. NeurIPS, 2019.

[6] D. Gilton, G. Ongie, and R. Willett, “Deep Equilibrium Architectures for Inverse Problems in Imaging,” IEEE Trans. Comput. Imaging, vol. 7, pp. 1123–113file:///C:/Users/zhoutian/Desktop/ismrm/kn, 2021, doi: 10.1109/TCI.2021.3118944.

[7] F. Knoll et al., “FastMRI: A publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning,” Radiol. Artif. Intell., vol. 2, no. 1, 2020, doi: 10.1148/ryai.2020190007.

[8] J. Zhang and B. Ghanem, “ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 1828–1837, 2018, doi: 10.1109/CVPR.2018.00196.
[9] G. Oh, J. E. Lee, and J. C. Ye, “Unpaired MR Motion Artifact Deep Learning Using Outlier-Rejecting Bootstrap Aggregation,” IEEE Trans. Med. Imaging, vol. 40, no. 11, pp. 3125–3139, 2021, doi: 10.1109/TMI.2021.3089708.