1988

Edge Map Reinforced Two Stage Super-Resolution GAN with Attention-Based DenseNet Generator for Knee MR Images
Muhammad Adnan Nasim1, Marva Touheed1, Faisal Najeeb1, and Hammad Omer1
1Medical Image Processing Research Group (MIPRG), Department of Electrical and Computer Engineering, COMSATS University, Islamabad, Pakistan

Synopsis

Keywords: AI/ML Image Reconstruction, Machine Learning/Artificial Intelligence

Motivation: Super-resolution is challenging especially in medical imaging where recovering fine structures from low-resolution images is essential for accurate diagnosis.

Goal(s): Our goal is to achieve super-resolution for the low-resolution knee MR images.

Approach: We propose an edge map reinforced super-resolution GAN with attention-based DenseNet generator to enhance the knee structures in two stages: (1) Generating an enhanced edge map for the low-resolution input image; (2) Generating super-resolution output by using the enhanced edge map generated in the first stage and low-resolution input images.

Results: The results show that the proposed method provides better output than the contemporary super-resolution GAN method.

Impact: This work presents a robust edge map reinforced super-resolution framework for knee MRI by using an attention-based DenseNet generator in super-resolution GAN. The proposed method generates MR images with clear edges and facilitates accurate medical assessment for knee related disorders.

Introduction:

In medical images, spatial resolution is often not sufficient because of certain constraints such as long image acquisition time and hardware limits1. The low-resolution imaging in knee MRI can significantly hamper the diagnosis of knee related medical conditions2.
Generative Adversarial Networks (GANs) have demonstrated their ability to produce high quality output images, enabling a wide range of image-to-image transformations3. One most relevant work in literature for achieving super-resolution through generative adversarial networks is the development of super-resolution GAN (SRGAN)4, which generates high-resolution output from the low-resolution input images. Another effective GAN based architecture is EdgeConnect5, that has been proposed for image inpainting task. This method is composed of edge-map generator and image completion network. Literature review also reveals another method known as attention-based DenseNet with residual deconvolution (ADRD)6; achieving super-resolution by generating attentive maps to enhance high-frequency features in the input image.
Inspired by the SRGAN4, EdgeConnect method5 and ADRD6, we propose a novel edge map reinforced two stage super-resolution network with attention-based DenseNet generator for the enhancement of structures in knee MRI.

Method:

In this work, a two stage GAN based super-resolution approach is proposed for knee MR images using edge maps. For training of the proposed method, the low-resolution images are obtained by applying a window of size 160x160 at the center of k-space of high-resolution images with a size of 320x320. The edge maps for the low-resolution input knee MR images and their ground truths are generated using canny edge detection method5. The generator network at each stage of the proposed method, incorporates DenseNet blocks and spatial attention mechanism (Figure-1). Within each DenseNet block, there are three dense layers designed to capture the important features. The aim of spatial attention module is to improve the representation of high-frequency information by learning a set of attentive maps including: residual feature maps, attentive maps and enhanced feature maps (Figure-1). The spatial attention module incorporates a 1x1 convolutional layer to ensure that the number of channels in input Ein matches the number of channels in output of the dense block Edbk. The residual maps Eresidual are obtained by applying the subtraction operation (Equation-1).


Eresidual = | Ein - Edbk | (Equation-1)

To obtain the attentive feature maps Eatt, an attentive function fatt (consisting of two consecutive 3x3 convolutional layers) and tanh activation function are applied on Eresidual (Equation-2). The generation of residual attentive maps Eresamp is done by taking Hadamard product of Eatt and Eresidual (Equation-3). In the final step, the enhanced feature maps Eenhanced are generated by combining the weighted Eresamp (with parameter λ=0.5) and Edbk (Equation-4).

Eatt = Tanh(fatt (Eresidual)) (Equation-2)

Eresamp = Eatt ⊙ Eresidual (Equation-3)

Eenhanced = λ Eresamp + Edbk (Equation-4)

To generate super-resolution output images, the proposed method relies on using GANs in two stages (Figure-2). In the first stage, GAN-1 generates an enhanced edge map EMemap for the low-resolution input edge maps of knee images LRemap (Equation-5). In the second stage, GAN-2 predicts the super-resolved images by using the enhanced edge map EMemap and the original low-resolution input image LRimg (Equation-6). The discriminator networks for both the GANs are the same as used in SRGAN network4.

EMemap = G1 (LRemap) (Equation-5)

SRpred = G2 (EMemap , LRimg) (Equation-6)


Dataset used in this work has been obtained from NYU fastMRI initiative database (fastmri.med.nyu.edu) 6. We have used randomly selected 5000 knee DICOM images (4000 for training and 1000 for testing).

The proposed framework is implemented on Python 3.11 by Keras using TensorFlow as a backend on AMD-Ryzen7 5800H-CPU, clock frequency 3.19 GHz, 24GB RAM and GPU NVIDIA GeForce GTX3070 for approximately 33-hours. The model is trained for 100 epochs with RMSProp optimizer and the learning rates of 0.0001 and 0.00001.

Results:

The enhanced edge map in Figure-3 and the super-resolution output comparison in Figure-4 show that the proposed method outperforms the SRGAN method4. Table-1 shows a significant improvement in terms of PSNR and SSIM in the images produced by our proposed method.

Discussion and Conclusion:

In this work, we have proposed a novel edge map reinforced two stage super-resolution network that utilizes attention based DenseNet generator network. The enhanced edge maps generated at the first stage of our proposed method actually contribute in the restoration of high-frequency edge details in the super-resolved image. When compared with SRGAN, our proposed method provides better results by finely restoring the structures in knee MR images.

Acknowledgements

We are thankful to NVIDIA for providing Academic Hardware Grant of worth 2400USD. This grant includes 400 compute hours on V-100 GPU instances via NVIDIA's partner, Saturn Cloud.

References

  1. Y. Li, B. Sixou, F. Peyrin, A Review of the Deep Learning Methods for Medical Images Super Resolution Problems, IRBM, Volume 42, Issue 2, 2021, Pages 120-133, ISSN 1959-0318, https://doi.org/10.1016/j.irbm.2020.08.004.
  2. Qiu D, Zhang S, Liu Y, Zhu J, Zheng L. Super-resolution reconstruction of knee magnetic resonance imaging based on deep learning. Comput Methods Programs Biomed. 2020 Apr;187:105059. doi: 10.1016/j.cmpb.2019.105059. Epub 2019 Sep 24. PMID: 31582263.
  3. Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2017.632
  4. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2017.19
  5. Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M. (2019). EdgeConnect: Structure guided image inpainting using edge prediction. 2019 IEEE/ CVF International Conference on Computer Vision Workshop (ICCVW). https://doi.org/10.1109/iccvw.2019.00408
  6. Knoll, F., Zbontar, J., Sriram, A., Muckley, M. J., Bruno, M., Defazio, A., Parente, M., Geras, K. J., Katsnelson, J., Chandarana, H., Zhang, Z., Drozdzalv, M., Romero, A., Rabbat, M., Vincent, P., Pinkerton, J., Wang, D., Yakubova, N., Owens, E., … Lui, Y. W. (2020). FastMRI: A publicly available raw K-space and DICOM dataset of knee images for accelerated Mr Image Reconstruction using machine learning. Radiology: Artificial Intelligence, 2020 Jan 29;2(1): e190007. doi: 10.1148/ryai.2020190007. https://pubs.rsna.org/doi/10.1148/ryai.2020190007 and the arXiv paper, https://arxiv.org/abs/1811.08839.


Figures

Figure-1: (a) Architecture of proposed generator network; (b) Discriminator network; (c) Spatial Attention Mechanism: (1) Generation of Residual feature maps (2) Generation of Attentive maps (3) Generation of enhanced feature maps

Figure-2: Proposed framework for edge map reinforced two stage super-resolution for knee MRI

Figure-3: (1) Low-resolution image; (2) Low-resolution edge map; (3) Enhanced edge map output from GAN-1; (4) Super-resolution output image from GAN-2; (5) Reference image

Figure-4: Comparison of the proposed method and SRGAN output: (1) Low-resolution image; (2) SRGAN Output4; (3) Proposed method output; (4) Reference image

Table-1: Comparison of PSNR and SSIM values of the results obtained from SRGAN4 and the proposed method.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
1988
DOI: https://doi.org/10.58530/2024/1988