0342
A Novel End-to-end Joint Reconstruction and Segmentation Interaction Network for MRI1Harbin Institute of Technology, Harbin, China
Synopsis
Keywords: AI/ML Image Reconstruction, Data Processing
Motivation: For magnetic resonance imaging (MRI) applications, rapid imaging and automatic segmentation of target tissues are critical. However, most existing methods barely consider MR image segmentation in fast imaging scenarios.
Goal(s): Our goal is to simultaneously achieve high scanning acceleration and accurate multi-class tissue segmentation results under a unified framework.
Approach: We propose a novel multi-task method with a novel interaction module to reconstruct undersampled MR images based on modified ISTA-Net and simultaneously segment tissues based on lightweight U-Net.
Results: Experiments on cardiac and knee datasets demonstrate that our method outperforms existing state-of-the-art multi-task approaches for joint MR image reconstruction and segmentation.
Impact: The proposed multi-task interaction method can effectively achieve high scanning acceleration and accurate segmentation results simultaneously, which can further expand the application of MR in clinical disease diagnosis.
Introduction
Magnetic resonance imaging (MRI) has been a powerful imaging technique that could generate high-resolution images and provide a clear soft tissue depiction[1]. However, most MR reconstruction methods focus on the accelerated imaging while overlook the affect of reconstruction error on subsequent application. On the other hand, the segmentation methods usually take full sampled image as input which require a time consuming imaging. Consequently, the simultaneous achievement of high scan acceleration and precise segmentation results is of utmost importance for the successful implementation of MRI in various clinical scenarios.With the development of deep learning, various neural network-based reconstruction methods have been proposed[2][3]. Dong et al. [4] proposed to utilize the Generative Adversarial Network (GAN) combined with spatial attention and channel attention to alleviate artifacts caused by high undersampling. Yang et al. [5] proposed the ADMM-Net for MRI reconstruction which unrolled the ADMM iterative optimization algorithm into a deep network. However, those reconstruction methods usually neglect the subsequent image applications, which are crucial for medical professionals.
In order to achieve automatic segmentation of MRI, many CNN-based segmentation methods have been proposed [6][7]. For instance, Wang et al.[8] proposed to utilize efficient U-Net model which constructs a symmetrical encoder-decoder structure with skip connections to leverage multi-scales image features for multi-sequence MR image segmentation, achieving satisfactory results. [9] proposed a multi-pathway U-Net combined with residual connection to segment tumors in multimodal brain MR images. However, MR image segmentation methods typically assume high-quality images as input, which unfortunately requires a time-consuming data collection process.
There have been limited attempts to integrate MRI reconstruction and segmentation into an end-to-end training framework[10][11].
Methods
Fig.1 is the diagram of the proposed method. Specifically, we alternate MRI reconstruction and segmentation in a unified framework with a interaction module. For the reconstruction module, we construct an unfold network which is based on the ISTA-Net[12], as shown in Fig.1. In order to process different components of the input image, we utilize a Gaussian kernel to split the image into high frequency components and low frequency components. For different components, we construct a parallel processing structure, using convolutional layers and non-linear layers to learn sparse transformations for each frequency, and learn the threshold of the sparse domain and then use inverse transformation to solve. The final restored image is aggregated from the two parts. For the segmentation module, we use the lightweight U-Net[13] , as shown in Fig.2. The basic block consists of a 3 × 3 conventional layer, a group normalization (GN) layer, and a ReLU layer. In addition, in order to improve the overall effect of the model, we propose to use the interaction module which Introduce semantic information in segmentation results as image structure prior into the reconstruction process. The loss function of the proposed method is the combination of reconstruction loss and segmentation loss.The proposed method is implemented on the Pytorch framework using a Nvidia Quadro GV100 GPU. The learning rate was initially set to 0.0005, the iteration for reconstruction was 8, and the number for segmentation networks was 5. The network was trained using AdamW optimizer with a weight decay of 0.0001.
Results and discussion
We conducted experiments using two publicly available datasets, namely the ACDC dataset [14] and SKM-TEA dataset[15], utilizing various sampling mask. We split the dataset into training data, validation data, and testing data following the official division of each dataset. And we unify images and labels of ACDC to the size of 192*192 and unify images and labels of SKM-TEA to the size of 320*320. The performance of the proposed method is compared with two state-of-the-art joint reconstruction and segmentation methods, namely the SegMRI[16] and SegAware[17], and the reconstruction method DCCNN[18], and our proposed MPISTA. Fig.3 and Fig.4 shown the reconstruction and segmentation results of different methods. Table.1 shown the comparison of task evaluation indices, including PSNR, SSIM, and Dice coefficient. It is obvious that our proposed method could obtain the best performance both on MR image reconstruction and segmentation task.Conclusion
In this paper, we propose a novel multi-task method for MRI joint reconstruction and segmentation. The EJRS-Net constructs an unfolded deep learning network with interleaved connections of the reconstruction iterations and segmentation networks to realize the effective interaction between two tasks, enhancing the overall performance. To validate the efficacy of our proposed method, we conducted experiments on two public datasets, demonstrating that our method consistently outperforms the SOTA methods in terms of reconstruction and segmentation tasks. Overall, our study proposed a promising solution both for joint accelerated MR imaging and segmentation.Acknowledgements
This work is supported by Natural Science Foundation of Heilongjiang YQ2021F005 and China NSFC 62371167.References
1. De Schepper A M, De Beuckeleer L, Vandevenne J, et al. Magnetic resonance imaging of soft tissue tumors[J]. European radiology, 2000, 10: 213-223.
2. Wang S, Su Z, Ying L, et al. Accelerating magnetic resonance imaging via deep learning[C]//2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, 2016: 514-517.
3. Guo P, Patel V M. Reference-based MRI Reconstruction Using Texture Transformer[C]//Medical Imaging with Deep Learning. 2023.
4. Yang G, Yu S, Dong H, et al. DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction[J]. IEEE transactions on medical imaging, 2017, 37(6): 1310-1321.
5. Yang Y, Sun J, Li H, et al. ADMM-CSNet: A deep learning approach for image compressive sensing[J]. IEEE transactions on pattern analysis and machine intelligence, 2018, 42(3): 521-538.
6. Tran P V. A fully convolutional neural network for cardiac segmentation in short-axis MRI[J]. arXiv preprint arXiv:1604.00494, 2016.
7. Huang X, Deng Z, Li D, et al. Missformer: An effective transformer for 2d medical image segmentation[J]. IEEE Transactions on Medical Imaging, 2022.
8. Liu Y, Wang W, Wang K, et al. An automatic cardiac segmentation framework based on multi-sequence MR image[C]//Statistical Atlases and Computational Models of the Heart, 2020: 220-227.
9. Saha A, Zhang Y D, Satapathy S C. Brain tumour segmentation with a muti-pathway ResNet based UNet[J]. Journal of Grid Computing, 2021, 19: 1-10.
10. Huang Q, Yang D, Yi J, et al. FR-Net: Joint reconstruction and segmentation in compressed sensing cardiac MRI[C]//Functional Imaging and Modeling of the Heart: 10th International Conference, FIMH 2019, Bordeaux, France, June 6–8, 2019, Proceedings 10. Springer International Publishing, 2019: 352-360.
11. Calivá F, Leynes A P, Shah R, et al. Breaking speed limits with simultaneous ultra-fast MRI reconstruction and tissue segmentation[C]//Medical Imaging with Deep Learning. PMLR, 2020: 94-110.
12. Zhang J, Ghanem B. ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 1828-1837.
13. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C]//Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015: 234-241.
14. Bernard O, Lalande A, Zotti C, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?[J]. IEEE transactions on medical imaging, 2018, 37(11): 2514-2525.
15. Desai A D, Schmidt A M, Rubin E B, et al. Skm-tea: A dataset for accelerated mri reconstruction with dense image labels for quantitative clinical evaluation[J]. arXiv preprint arXiv:2203.06823, 2022.
16. Sun L, Fan Z, Ding X, et al. Joint CS-MRI reconstruction and segmentation with a unified deep network[C]//Information Processing in Medical Imaging: 26th International Conference, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26. Springer International Publishing, 2019: 492-504.
17. Acar M, Çukur T, Öksüz İ. Segmentation-aware MRI reconstruction[C]//International Workshop on Machine Learning for Medical Image Reconstruction. Cham: Springer International Publishing, 2022: 53-18. Schlemper J, Caballero J, Hajnal J V, et al. A deep cascade of convolutional neural networks for dynamic MR image reconstruction[J]. IEEE transactions on Medical Imaging, 2017, 37(2): 491-503.
Figures
