Introduction

3D imaging is commonly used in MRI and CT imaging to assess patient anatomy, however anisotropic imaging with thick slices is often used to reduce scan-times. Super-resolution techniques to achieve high-resolution isotropic volumes have been proposed, however often require paired training data which is not available in many medical applications. We propose that small patches from the high-resolution plane contain similar structures to patches in the low-resolution direction. We use this idea to train a novel unpaired super-resolution approach. Our proposed network consists of a CycleGAN with a cycle-consistent gradient mapping loss and a convolution weight demodulation process. We train and test the resulting network separately using anisotropic MRI and CT data, and compare the results with a standard CycleGAN.

Datasets

To demonstrate utility in MRI, anisotropic 3D MRI data was retrospectively collected from $$$60$$$ patients who undertook abdominal VIBE imaging. The local research ethics committee approved the study (Ref: 10/H0/720/91). A breath-hold of $$$21$$$s was used, with the following imaging parameters: coronal orientation, matrix size $$$\approx 185 \times 330 \times 23$$$, in-plane resolution $$$\approx 1.2 \times 1.2mm$$$ (range: $$$0.8$$$, $$$1.6mm$$$) with through-plane resolution $$$\approx 6.6mm$$$ (range: $$$5.2$$$, $$$7.6mm$$$). Of these 3D VIBE datasets, $$$45$$$ were used for training the networks, and $$$15$$$ were reserved for testing.

To demonstrate utility in CT, anisotropic 3D CT data was extracted from $$$36$$$ patients in the DeepLeision¹ CT dataset, with the following image parameters: transverse/axial orientation, matrix size $$$\approx 512 \times 512 \times 52$$$, in-plane resolution $$$\approx 0.95 \times 0.95mm$$$ (range $$$0.91, 0.98mm$$$) with through-plane resolution of $$$5mm$$$. Of these CT datasets, $$$27$$$ were used for training and $$$9$$$ for testing.

Methodology (Standard CycleGAN)

The CycleGAN² architecture (see Fig. 1a) imposes a Cycle-Consistency Loss to perform image-to-image translation with unpaired data. The goal is to take images in the low-resolution (LR) domain, $$$X$$$, and translate them to the high-resolution (HR) domain, $$$Y$$$, through a generator mapping $$$G_{X}: X \rightarrow Y$$$.

The Cycle-Consistency Loss (Fig. 1b and 1c) is necessary as the generator mappings $$$G_{X}$$$ and $$$G_{Y}$$$ are severely under-constrained. To mitigate this, an inverse mapping is introduced which enforces $$$G_{Y}(G_{X}(x)) = x'$$$, where $$$x' \approx x$$$.

Methodology (Proposed / Modified CycleGAN)

We implemented two main changes to the standard CycleGAN to improve the resultant image quality: 1) modification to the loss, 2) replacement of the instance normalisation in the convolution layers. We modified the Cycle-Consistency Loss to also impose a "Gradient Mapping", $$$L_{gmap}$$$, which aims to reduce deformation errors. This loss works between the gradient maps of input images, $$$x \in X$$$ and $$$y \in Y$$$, and ``cycled" images $$$x' = (G_{Y}(G_{X}(x))$$$, $$$y' = G_{X}(G_{Y}(y)))$$$, respectively³.

$$
\begin{equation}\begin{aligned}L_{gmap}(G_{X},G_{Y}) = \mathbb{E}_{x \sim \text { pdata }} (x)[ \|S_{x}(G_{Y}(G_{X}(x)))-S_{x}(x) \\+S_{y}(G_{Y}(G_{X}(x)))-S_{y}(x)\|_1 ]\\+\mathbb{E}_{y \sim \text { pdata }}(y)[ \|S_{x}(G_{X}(G_{Y}(y)))-S_{x} (y)\\+S_{y}(G_{X}(G_{Y}(y)))-S_{y}(y)\|_1 ]\\\end{aligned}\label{eq:cycle_consistent_grad_map}\end{equation}
$$
where $$$S_x$$$ and $$$S_y$$$ denote the $$$x$$$-axis and $$$y$$$-axis gradients acquired using a Sobel Operator and $$$||\cdot||_{1}$$$ denotes an $$$L_{1}$$$ norm; the final loss function is then defined as follows:

$$
\begin{equation}\begin{aligned}\mathcal{L}_{\text {total}}(G_{X}, G_{Y}) &=\lambda_{cyc} L_{cyc} + \lambda_{gmap}L_{gmap},\label{eq:final_loss}\end{aligned}\end{equation}
$$

where $$$L_{cyc}$$$ denotes the original Cycle-Consistency Loss and $$$\lambda_{cyc}, \lambda_{gmap}$$$ denote loss weights for the Cycle-Consistency and Gradient Mapping losses respectively. Furthermore, we replaced the instance normalisation in the convolution layers of the standard CycleGAN, with a weight demodulation process, to reduce the presence of droplet or block-like noise artefacts, similarly to StyleGAN2⁴. This step scales the weights of output feature maps, $$$j$$$, to unit standard deviation by a factor of $$$1 /{\sigma_{j}}$$$.

$$
\begin{equation}\begin{aligned}\left. w''_{ijk} = w'_{ijk} \middle/ \sqrt{\sum_{ik} {w'_{ijk}}^{2} + \epsilon}, \right.\label{eq:weight_demod}\end{aligned}\end{equation}
$$
where $$$i$$$ denotes the index of the input feature map, $$$j$$$ enumerates the output feature maps, and $$$k$$$ enumerates the spatial dimensions of the convolutional filter.

Methodology (Patch Reconstruction Algorithm)

The standard CycleGAN and proposed CycleGAN were trained using unpaired, random, 2D patches (size $$$32 \times 32$$$) taken from the low-resolution ($$$x \in X$$$) and high-resolution ($$$y \in Y$$$) imaging planes of the 3D data. The resultant networks can be applied to unseen prospective data not of fixed size, using an overlapping patch reconstruction algorithm.

Quantitative Results

The resultant image quality was compared from the unsupervised networks using the no-reference image quality score: Perception Based Image Quality Evaluator (PIQUE)⁵ (see Table 1 and Figure 2).

Conclusions

Our initial results demonstrate that it is possible to perform super-resolution without paired data. We have demonstrated that our technique performs well on a relatively small training dataset, for both CT and MRI data, with significant improvement in visual image quality and perceptual image scores. We have shown that our proposed method improves image quality over a standard CycleGAN and believe this approach has the potential to improve image quality for anisotropic medical imaging.

Acknowledgements

This work was supported by UK Research and Innovation (MR/S032290/1).