Introduction

Fetal brain magnetic resonance imaging can provide more detailed and accurate 3D information of brain morphology, making it particularly critical for fetal brain malformation diagnosis and development assessment [1]. However, the inevitable fetal motion and maternal breathing during MRI acquisition can result in loss of structural continuity and corrupted volumetric information. Although fast imaging techniques can overcome the in-plane motion of individual 2D slices [2], the motion between slices still exists. Therefore, there is an urgent need to reconstruct high-quality fetal brain 3D volumes from multiple motion-corrupted 2D slices. Although the traditional iterative motion correction and reconstruction approaches have proven effective, they have two major limitations: 1) limited to capturing relatively small motion; and 2) time-consuming. Herein, we propose a novel end-to-end learning-based fetal brain MRI 3D volume reconstruction method. The main contributions are as follows. 1) A multi-scale feature fusion model is proposed to correct arbitrary motion of 2D slices. 2) A residual-based encoder-decoder model with a specifically designed perceptual loss reconstructs the high-resolution fetal brain MRI 3D volume with anatomically realistic details further by using an adversarial loss.

Methods and Materials

We propose an end-to-end highly efficient fetal brain MRI 3D volume reconstruction method based on deep learning, as shown in Fig. 1. It consists of three steps: 1) motion estimation and correction, 2) initial volume reconstruction, and 3) high-quality high-resolution volume reconstruction.
First, a multi-scale feature fusion model is proposed to predict motions of fetal brain MRI 2D slices. It includes a bottom-up path, a top-down path, and lateral connections, as shown in Fig. 2. The bottom-up path is the feature extraction backbone of the model, which is composed of basic residual blocks to generate feature maps at four different scales. The top-down path generates higher resolution features through up-sampling. Then, the feature maps of the bottom-up and top-down paths are fused by lateral connections. The model can capture both fine-grained details and global context information at different scales, thus enabling the model to handle different slices with different appearances. Similar to SVRnet [3], anchor points are used to define motion parameters, and L1 loss is used to train the model.
Then, according to the predicted motion parameters, an initial 3D volume is estimated by point spread function. To improve the initial 3D volume quality, we develop a residual-based encoder-decoder model, which can learn an enriched set of textural and structural features, as shown in Fig. 3(a). To optimize the model for achieving better performance, we design a perceptual loss, as shown in Fig. 3(b), which measures the difference between two 3D volumes in terms of the feature map similarity, rather than pixel-wise differences to preserve more textural and structural details. To produce anatomically more realistic details on the reconstructed 3D volume, the adversarial loss is introduced as in WGAN. A simple discriminator network is proposed to calculate the Wasserstein distance between the ground truth 3D volume and the reconstructed 3D volume, as shown in Fig. 3(b).
We developed the model based on 279 T2-weighted fetal brain MRI volumes between 21 and 38 weeks of gestational age. For training purpose, these fetal volumes were reconstructed using NiftyMIC[4] with 0.8 mm isotropic resolution. The datasets applied to simulate the fetal brain motion were divided into training (194), validation (30), and test (55) sets. The models were implemented with Pytorch, and trained on NVIDIA TITAN Xp GPU using Adam optimizer. The fetal motion simulation method is similar to [4].

Results

The standard metrics Structural Similarity (SSIM) and Peak Signal-to-Noise Ratio (PSNR) are used to evaluate the performance of methods. The run time is used to evaluate efficiency. We adopted NiftyMIC [5] and SVRTK [6] as baselines for comparison. In terms of PSNR, our method improves by 7.14 and 2.96 compared to SVRTK and NiftyMIC, respectively. According to SSIM, our method improves by 0.194 and 0.237, respectively. Compared with these methods, the proposed method achieved the speedup of 167.9 and 21.8 times, respectively. Fig. 4 shows the representative reconstruction results obtained by different methods.

Conclusions

In this work, an end-to-end, highly efficient method based on deep learning is proposed to reconstruct fetal brain MRI 3D volume. The multi-scale feature fusion model can solve arbitrary motion correction of 2D slices. Our designed residual-based encoder-decoder model can learn an enriched set of features to reconstruct high-resolution fetal brain MRI 3D volume. The results demonstrate the proposed method is able to reconstruct high-resolution 3D volumes with high efficiency and accuracy.

Acknowledgements

This work was supported in part by NIH grants (MH116225, MH117943 and MH123202 to G.L.).