1079

POCS-Transformer for MR Image Reconstruction
Anam Nazir1, Muhammad Nadeem Cheema1, Yiran Li1, Yulin Chang2, John A Detre3, and Ze Wang1
1Department of Diagnostic Radiology and Nuclear Medicine, School of Medicine, University of Maryland,, Baltiomore, MD, United States, 2Siemens Healthineers, Baltimore, MD, United States, 3Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States

Synopsis

Keywords: Image Reconstruction, Brain, Reconstruction

Motivation: Transformers excel in medical image processing but require many parameters and training data. We mitigated this issue with the POCS-Transformer method.

Goal(s): POCS-Transformer goal was to enhances MR image reconstruction along with preserving image quality with various under-sampling masks.

Approach: The POCS-Transformer, built on Swin-T using FastMRI data, employed binary undersampling, POCS augmentation, data consistency penalties, and was compared to VN and POCS-CycleGAN on test data.

Results: POCS-Transformer outperformed POCS-CycleGAN with superior image quality and less blurring. POCS-Transformer achieved higher mean PSNR and SSIM compared to both VN and POCS-CycleGAN in knee and brain image datasets.

Impact: The POCS-Transformer improves MR image reconstruction in terms of reducing blurring even under diverse under-sampling conditions. Its impact extends to healthcare and research. New questions involve its applications in medical imaging, merging traditional and modern methods to inspire further innovations.

Introduction

Transformers were originally designed for natural language processing and have achieved state-of-art performance in medial image processing including MR image reconstruction 1-3. One limitation is transformers contain more parameters and often need more training samples. We addressed this problem using the Projection-Onto-Convex-Set (POCS) method. We compared the proposed POCS enhanced Transformer (POCS-Transformer) to the Variational Network (VN) 4, which represents the current state-of-art, and the POCS-CycleGAN, a POCS-based DL MR image reconstruction algorithm 5.

Methods

The Swin-T architecture 6 was used to build the POCS-Transformer. Training data were downloaded from FastMRI 7. Undersampled k-space data were synthesized by multiplying the fully sampled k-space data with three different binary undersampling masks. These data were then Fourier transformed into the image domain to get the “zero-filling” images to be used as the input for the Transformer. The fully sampled images were used as the training reference. POCS was applied five times during network training to augment the training data. At each time, the temporary outputs of Transformer were inverse Fourier transformed and replaced with the original k-space data at the undersampled k-space positions. They were then Fourier transformed and included as additional training samples. Finally, the model was retrained using the original and the augmented training data. Data consistency penalty (distance between the acquired k-space data and the reconstructed k-space at the sampling positions) was added as an additional loss function. The final output was further augmented using POCS. POCS-Transformer was then compared to VN and POCS-CycleGAN using the same testing data.

Results and Discussion

The POCS-Transformer, implemented in PyTorch, was trained using the ADAM algorithm with a learning rate of 0.001. Training involved a batch size of 4 for up to 100 epochs on a PC with an Intel Core i7-9800X CPU and an Nvidia GeForce Titan Xp GPU (12GB). The training phase, which included 103,758 brain slices, took approximately 118 hours. During testing, it took 0.041 seconds to process a single slice out of the 13,344 tested.
Fig. 1 and 2 show the reconstruction results of POCS-CycleGAN, VN, and POCS-Transformer and difference image. As compared to POCS-CycleGAN, both VN and POCS-Transformer yielded much better image quality. Their outputs were nearly identical to the references. POCS-Transformer showed much less image blurring than POCS-CycleGAN. Fig 3 shows the reconstruction results of POCS-Transformer for 2 different under sampling masks (spiral 10%, spiral 20%) when tested on the same pre-trained network. The outputs of POCS-Transformer were very similar to the reference. Table 1 lists the quantitative evaluation results knee and brain image slices. POCS-Transformer produced higher mean PSNR (Peak Signal to Noise Ratio, p<0.001) than the POCS-CycleGAN for both the knee and brain datasets. SSIM (structural Similarity index) of POCS-Transformer was also higher than that of VN for both datasets (p<0.001, paired t-test). These differences were statistically tested using paired t-tests and the p values (POCS-CycleGAN Vs. POCS-Transformer)for brain SSIM and PSNR were 2.034x10-285, 1.54x10-126 , for knee SSIM and PSNR were 4.63x10-269, and 3.66x10-303 respectively. P values for VN compared to POCS-Transformer for brain SSIM and PSNR were 2.635x10-142, 2.081x10-51 , for knee SSIM and PSNR were 2.054x10-112, and 1.166x10-206 respectively.

Conclusion

Our results showed that by combining the traditional POCS-based image reconstruction and Transformers, POCS-Transformer produces better MR image reconstruction results than VN by preserving image quality across various under-sampling masks and other POCS based reconstruction method. Compared to the convolutional neural network-based POCS-CycleGAN, POCS-Transformer yielded much less image blurring.

Acknowledgements

This work was supported by NIH grants: R01AG060054, R01AG070227, R01EB031080-01A1, R21AG082345, R21AG080518, P41EB029460-01A1 and by the University of Maryland Baltimore, Institute for Clinical & Translational Research (ICTR) through the NIH grant: 1UL1TR003098.

References

  1. Guo Pengfei, Yiqun Mei, Jinyuan Zhou, Shanshan Jiang, and Vishal M. Patel. "ReconFormer: Accelerated MRI reconstruction using recurrent transformer." IEEE Transactions on Medical Imaging (2023).
  2. Naoto Fujita and Yasuhiko Terada, DC-Swin: Deep Cascade of Swin Transformer with Sensitivity Map for Parallel MRI Reconstruction Proc of ISMRM, 2919, 2023
  3. Zhou B, Dey N, Schlemper J, Salehi SS, Liu C, Duncan JS, Sofka M. DSFormer: A dual-domain self-supervised transformer for accelerated multi-contrast MRI reconstruction. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2023 (pp. 4966-4975).
  4. Sriram, Anuroop, et al. "End-to-end variational networks for accelerated MRI reconstruction." Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23. Springer International Publishing, 2020.
  5. Li,Yiran, et al. "POCS-Augmented CycleGAN for MR Image Reconstruction." Applied Sciences 12.1 (2021): 114.
  6. Liu, Ze,et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
  7. Zbontar, Jure, et al. "fastMRI: An open dataset and benchmarks for accelerated MRI." arXiv preprintarXiv:1811.08839 (2018).

Figures

Fig 1: POCS-CycleGAN, VN and POCS-Transformer reconstruction results for two brain slices. A 10% undersampling spiral trajectory was used to get the undersampled data. Second and fourth rows are the difference between the reconstructed images and the ground truth.

Fig 2: POCS-CycleGAN, VN and POCS-Transformer reconstruction results for two 10% undersampled knee images. A 10% undersampled spiral trajectory was used. Second and fourth rows are the difference between the reconstructed images and the ground truth.

Fig 3: Output of POCS-Transformer results for brain and knee datasets with 10% and 20% masks for spiral under sampling trajectory.

Table1: Performance indices of different reconstruction methods for brain and knee images for a 10% undersampling rate using a spiral trajectory. PSNR and SSIM are listed as mean ± standard deviation.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
1079
DOI: https://doi.org/10.58530/2024/1079