Synopsis

For medical imaging applications, it is not straightforward to create a large database due to high costs associated with acquiring the data, patent privacy issues, and challenges in pooling data from multiple medical institutions. Generating high-resolution medical images from the latent noise vector could potentially mitigate training data size issues in applying DNN to medical imaging. This could facilitate objective comparisons between the different machine learning algorithms in medical imaging. In this study, progressive growing strategy is considered to train the GAN stably and generate super resolution brain datasets from noise vector.

Introduction:

Deep neural networks (DNN) have demonstrated potential for various computer vision tasks^1,2. Large scale databases such as ImageNet represent a library of realistic natural images that have been instrumental in accelerating development of DNN techniques. However, for medical imaging applications, it is not straightforward to create such a large database due to high costs associated with acquiring the data, patent privacy issues, and challenges in pooling data from multiple medical institutions. Conventional methods for training data augmentation such as affine or nonlinear deformations of existing data (Figure. 1) can increase the data size. However, these transformations often generate unrealistic data. A possible remedy is to augment existing imaging database with synthetic realistic medical imaging data. Generative Adversarial Networks (GANs) can potentially learn the manifold within the high-dimensional space that medical images resides in; therefore, GAN can synthesize the realistic images^3,4. However, generating large-size of medical images using traditional GANs can be unstable due to the practical difficulties of training the GAN, resulting in non-convergence, mode collapsing and diminished gradient ^5,6 .Recently, progressively growing training strategy has been proposed to improve the stability, quality and variability of the GAN⁷. In this work, we present an effective method to synthesize realistic and high-quality MRI data using the progressively growing training strategy for GAN.

Methods:

Figure.2 (a) shows the overall workflow to generate realistic brain MR images from 512-dimensional Gaussian noise vectors. Noise vectors are fed to the generator and the output of the generator are judged by the discriminator. The Generator and Discriminator consist of the several convolutional layers with up sampling and down sampling operations, respectively. Figure.2 (b) summarizes the training process for the GAN. We start with training the outer layers with very low-resolution images. Subsequently we progressively increased the resolution until the desired resolution are achieved. 2256 complex-valued brain images with a size of 256×256 were used as a bank of real data to train the network. These data were acquired as part of clinically indicated brain scans over a time span of 6 months. The learning rate was set to 0.001 and the lower resolution images were generated by extracting the lower-freuqency component of k-space followed by a Fourier transform. The training time was approximately 8 days on a Windows (64 GB RAM), NVIDIA TITAN Xp (12 GB) by using Tensorflow.

To evaluate the diversity of the generated images, Multi-Scale Statistical Similarity (MSSS) - as a common metric for evaluating GANs - was calculated^7-9. To generate higher resolution images, the network was retrained based on another dataset of 50 brain images (complex valued, 512x512 sized).

Results:

Figure.3 shows the sample of 256x256 brain images generated by the trained network from 512-dimensional noise vectors. These images look highly realistic and are distinct from the bank of our real datasets.

Figure.4 shows a sample of higher resolution 512×512 brain data generated from noise vectors. The MSSS for our network was 0.1988.

Discussion:

Training the GANs to produce the high-resolution images is a challenging task. One of the often encountered problems is that the discriminator can easily detect that the generated high-resolution image is far apart from the training dataset, hence magnifying the gradient problem. Progressively training procedure enables GANs to generate the high-resolution image from latent noise. As can be seen in the Figure.3, not only all the generated data (256×256) look realistic, but also the images cover the wide range of tissue contrast including T1, T2, and FLAIR.

Generating high quality detailed brain structure, as shown in Fig. 4, with no observable collapsing mode confirms that the training process was successful. Generating high-resolution medical images from the latent noise vector could potentially mitigate training data size issues in applying DNN to medical imaging. This could facilitate objective comparisons between the different machine learning algorithms in medical imaging. In this study, we showed the possibility of generating high-resolution images from the noise vector by training the GAN progressively. It is noted that for the tasks such as supervised learning, GAN may also help expand the data size. For instance, one could condition the GAN on the label and train it to produce the corresponding image for the label. Once the GAN is trained, it could produce images that correspond to a given label, expanding the training data for supervised learning algorithms.

Acknowledgements

References

[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, p. 436, 2015.

[2] L. Deng and D. Yu, “Deep learning: methods and applications,” Found. Trends® Signal Process., vol. 7, no. 3–4, pp. 197–387, 2014.

[3] J. T. Guibas, T. S. Virdi, and P. S. Li, “Synthetic Medical Images from Dual Generative Adversarial Networks,” arXiv Paper. arXiv1709.01872, 2017.

[4] I. Goodfellow et al., “Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672–2680.

[5] T. Salimans et al., “Improved Techniques for Training GANs,” in Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V Luxburg, I. Guyon, and R. Garnett, Eds. Curran Associates, Inc., 2016, pp. 2234–2242.

[6] I. Goodfellow, “NIPS 2016 Toturial: Generative Adverserial Networks,” arXiv paper. arXive 1701.00160v4, 2016.

[7] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv Paper. arXiv1710.10196, 2017.

[8] E. L. Denton, S. Chintala, A. Szlam, and R. Fergus, “Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks,” arXiv Paper. arXiv1506.05751, 2015.

[9] W. Gan, “Wasserstein GAN.” arXiv Paper. arXiv1701.07875, 2017.