Introduction

Synthetic magnetic resonance (MR) images are of growing importance in MR research. Such images can be used to expand existing datasets, to address particular data imbalances, and to provide data with rich ground truth information. Previous approaches generally generate short-axis images conditioned on input segmentation masks^1-3. Here we present a method for generating realistic synthetic exams, consisting of both short- and long-axis images of synthetic patients. Additionally, all synthesised images have corresponding segmentation masks consisting of 11 classes.

Methods and Results

We make use of state-of-the-art methods to produce high quality synthetic images from relatively low numbers of real images. Specifically, we use SyleGAN2⁴ with ADA⁵ to train two separate generative models: one for short-axis (SAX) and one for long-axis (LAX) images. We train on 256x256 pixel images resampled to 1mm isotropic resolution and rotated to a standard view. The SAX/LAX models were trained on 1841 SAX images from ACDC⁶ and 4935 LAX images from Kaggle⁷ and the M&M challenge⁸.

After training the StyleGANs we follow the method proposed in DatasetGAN⁹. Specifically, we train a network that classifies pixels in the generated image based on the intermediate features produced in the generator network during image generation. Thus, whenever an image is generated, a corresponding segmentation mask is also produced. To this end, a small number of generated images must be segmented for training. We hand-labelled 20 SAX and 10 LAX images with 11 classes each (9 of which are shared between the two views), plus an “other” class as a catch-all for unlabelled pixels (Figure 1 top).

As in DatasetGAN⁹, we train an ensemble of 10 segmentation networks, enabling the production of a confidence score for every pixel, based on the agreement between the 10 classifiers (we are less “confident” about pixels where the classifiers disagree). This confidence mask can be seen as a measure of the label certainty, and also as a proxy for the synthetic image quality overall, as low quality synthesised images tend to result in more disagreement between classifiers (Figure 1 bottom).

Our generators yield individual labelled SAX and LAX images. We next explore how a coherent set of SAX slices can be generated to emulate a standard multi-slice acquisition. First, we generate 200 random SAX images and heuristically classify each image, using its accompanying segmentation mask, as either basal (large amount of ventricles and lung visible), apical (small amount of ventricles and lung visible) or mid-ventricular (all other images). By choosing a basal and apical image at random, and interpolating between their latent representations, we can use these intermediate latent representations to generate “intermediate” images. These intermediate images in fact provide a good approximation to moving the SAX plane along the long axis of the left ventricle (LV) from the initial basal slice towards the apical slice, see Figure 2.

This technique allows us to generate a “stack” of SAX images, with the image content changing smoothly as we move through the stack, slowly transitioning from a basal to an apical image. However, taking a long-axis slice through the centre of these stacks reveals that a smooth transition between base and apex is not alone sufficient to create a realistic stack of SAX images. In particular, the LV structure in the long-axis direction is often non-physiological, see Figure 3.

To overcome this, we make two adjustments to the way we generate SAX image stacks: Firstly, we shift the generated SAX images in-plane so as to create a straight LV. Secondly, rather than sampling uniformly between the latent representations of the basal and apical images to generate intermediate slices, we instead sample such that the change in LV width as we move down through the SAX image stack matches the change seen in a template LAX image (e.g. from the LAX image generator). In practice, this means that rather than the LV width decreasing approximately linearly as we move down the long-axis, instead the width decreases slowly for the first ~75% of the LV’s length, and decreases rapidly thereafter. Results of these corrections are shown in Figure 4.

Given the resulting SAX stacks with improved physiological plausibility, we lastly show a proof-of-concept for the potential to find a synthetic LAX images that matches the anatomy seen in our synthesised SAX stacks, Figure 5.

Discussion and Conclusion

We have demonstrated generation of realistic labelled SAX image stacks, and corresponding labelled LAX images. These synthetic images could be combined with a bio-mechanical mesh model, as in Joyce et al¹⁰, and then deformed based on simulated mesh deformation to generate multi-slice/multi-view temporal data with known underlying cardiac dynamics. A shape model could also be used to more rigorously correct the long-axis LV shape.

When generating coherent SAX/LAX images, agreement on the heart itself is good. We believe overall agreement could be improved by also matching segmentation masks when generating the LAX view.

Lastly, the synthetic images have many more label classes than existing datasets (such as ACDC⁶), and could be used to train multi-class segmentation networks, allowing the prediction of detailed anatomical labels for real images. These could in turn be used to improve conditional synthesis approaches^1,2, or for quantitate evaluation of MR simulation methodologies.

Acknowledgements

We would like to acknowledge that the first two authors contributed equally to the work presented in this abstract. Funding of the Swiss National Science Foundation, grant 325230-197702, is gratefully acknowledged.