4786

POCS Augmented CycleGAN for MR Image Reconstruction
Hanlu Yang1, Yiran Li1, Danfeng Xie1, and Wang Ze2

1Electrical & Computer Engineering Department, Temple University, Philadelphia, PA, United States, 2Department of Radiology, Temple University, Philadelphia, PA, United States

Synopsis

Traditional MRI reconstruction depends heavily on solving nonlinear optimization problems, which could be highly time-consuming and sensitive to noise. We proposed a hybrid DL-based MR image reconstruction method by combining two state-of-art deep learning networks, U-Net and CycleGAN (Generative adversarial network with cycle loss) and a traditional method: projection onto convex set (POCS). Our result shows a high reconstruction accuracy and this method can be further used to increase the sample size, which may find many applications in situations where the training samples are limited such as medical images.

Introduction

Traditional MRI reconstruction depends heavily on solving nonlinear optimization problems, which could be highly time-consuming and sensitive to noise. By contrast, deep learning (DL)1 doesn’t need an explicit analytical data model and is robust to noise due to the large data-based training, opening a great opportunity for fast and high-fidelity MR image reconstruction2. While DL can be completely independent of the non-DL methods, it certainly can benefit from incorporating the rather more established traditional methods to achieve better results. To test that hypothesis, we proposed a hybrid DL-based MR image reconstruction method by combining two state-of-art deep learning (DL) networks, U-Net3 and CycleGAN (Generative adversarial network with cycle loss)4and a traditional method: Projection Onto Convex Set (POCS)5.

Materials and Methods

Fig.1 shows the method scheme, which includes two iterations as indicated by the blue and red arrows. CycleGAN and U-Net were combined to project the zero-filling reconstructed undersampled images to the fully sampled reference images. U-net was used as the generator to create the intermediate image whose difference to the reference was then estimated by the discriminator and sent back to refine the generator in order to get a better guess to the reference. A POCS process was then applied to the final output of the network trained during the first iteration (based on the original undersampled images only) by replacing the k-space data in the acquired locations with the original sampled k-space data. POCS images were then obtained by Fourier transform and were included as additional training samples to further refine the entire network (the 2nd iteration). We dubbed the afore-mentioned method as POCS-augmented CycleGAN or POCS-CycleGAN, which was compared to Compressed sensing (CS6), U_net, CycleGAN. Tensorflow7 was used to build all DL networks. The training and test images are extracted from T1-weighted images from our local database. Randomly undersampling with a rate of 30% was simulated. Peak-Signal-to-Noise(PSNR), the Structural Similarity Index (SSIM) and Mean-Square-Error (MSE) were collected to quantify the performance.

Results

Fig.2 is the bar graphs of the testing results. As compared to the zero-filling method and CS, U-net, CycleGAN, and POCS-CycleGAN provided better reconstruction quality in terms of higher SSIM and PSNR. U-net and CycleGAN showed no significant performance difference. POCS-CycleGAN yielded the best results. Fig. 3 shows two representative reconstruction cases. The difference maps clearly showed that POCS-CycleGAN produced the best image reconstruction quality.

Discussion

CycleGAN was shown to outperform other state-of-art reconstruction methods by Quan8. Our method differs by a POCS-based data augmentation, which directly doubled the training sample size, which may be the main reason for the performance gain of the method. Additional work (data didn’t show here due to the space limit), simply attaching POCS to CycleGAN can slightly improve CycleGAN but not as good as POCS-CycleGAN. Future work would be required to validate POCS-CycleGAN for high acceleration rates, and for 2D or 3D accelerations. Current POCS-CycleGAN directly learns the fully sampled images. Future work should be performed to test whether learning the artifact-dominated difference images as in9 could further improve performance.

Conclusion

We combined CycleGAN and POCS for sparsely sampled image reconstruction, providing a good strategy to double the training samples. By adding another iteration, it can further increase the sample size by another fold, which may find many applications in situations where the training samples are limited such as medical images.

Acknowledgements

No acknowledgement found.

References

  1. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. nature, 521(7553), 436.
  2. Wang, Dayong, et al. "Deep learning for identifying metastatic breast cancer." arXiv preprint arXiv:1606.05718 (2016).
  3. Hyun, C. M., Kim, H. P., Lee, S. M., Lee, S., & Seo, J. K. (2018). Deep learning for undersampled MRI reconstruction. Physics in medicine and biology
  4. Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint.
  5. Haacke, E. Mark, Zhi-Pei Liang, and Fernando E. Boada. "Image reconstruction using projection onto convex sets, model constraints, and linear prediction theory for the removal of phase, motion, and Gibbs artifacts in magnetic resonance and ultrasound imaging." Optical Engineering 29.5 (1990): 555-567.
  6. Doneva, M., Mutapcic, A., & Lustig, M. (2011). Compressed Sensing (CS) Workshop: Basic Elements of Compressed Sensing.
  7. Abadi, Martín, et al. "Tensorflow: a system for large-scale machine learning." OSDI. Vol. 16. 2016.
  8. Quan, T. M., Nguyen-Duc, T., & Jeong, W. K. (2018). Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE transactions on medical imaging, 37(6), 1488-1497
  9. Wang, Shanshan, et al. "Accelerating magnetic resonance imaging via deep learning." Biomedical Imaging (ISBI), 2016 IEEE 13th International Symposium on. IEEE, 2016.

Figures

POCS augmented CycleGAN structure

SSIM and PSNR evaluation

Image quality comparison with different networks

Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)
4786