Introduction

For quantitative and dynamic MRI, a series of images is always needed to be acquired, while the image resolution and quality would be restricted by the scan time. Typical serial MR sequences include CEST images with multiple saturation offsets, diffusion-weighted images with multiple directions, dynamic imaging with multiple time points or temporal phases, and so on. Recently correlations among the serial images have been utilized for highly-accelerated acquisitions, using either compress sensing methods[1,2], or deep-learning approaches[2,3]. However, restricted by locality of convolution operation, traditional deep-learning techniques could not take full account of the correlation among serial direction and image dimensions.As a novel deep-learning framework, transformer has demonstrated great advantages in vision tasks owing to its learning ability of global information by modeling long-range dependency[5], like Swin-Unet[6]. Herein, we utilized the transformer’s global attention on serial contrast dimension, to reconstruct highly under-sampled MR serial images, which has been initially validated on three datasets of different application scenarios.

Methods

Under-sampling Pattern
Following theoretical hypothesis of compressed sensing, a variable density randomized cartesian under-sampling pattern was generated (Figure 1a), which is different for all series images to efficiently use the complementary information along the serial dimension. Notably, the last serial image was fully-sampled.
Dataset & Preprocessing
We demonstrate the performance on 3 datasets.
1)FastCEST (Fast acquired Chemical Exchange Saturation Transfer MR Dataset): A dataset containing 12 healthy subjects for chemical exchange saturation transfer imaging on 3T, acquired in the Center for Biomedical Imaging Research (CBIR), Tsinghua University, with written consent signed. For each subject, 10 slices and 31 different saturation offsets from -10 to 10ppm were acquired. A 10 times retrospective under-sampling was applied to all slices, with 10 subjects for training and 2 subjects for test.
2)OCMR (Open-Access Multi-Coil k-Space Dataset for Cardiovascular Magnetic Resonance Imaging): The cardiac cine data were interpolated to 32 frames and under-sampled by 8 times in k-space. 58 subjects were taken for training set and 8 subjects for test.
3)HCP (Human Connectome Project):10 subjects were selected for our study. Each case contains 40 slices on 288 gradient directions (including 18 B0 images). An 8 times under-sampling was applied to the k-space data after 2-dimensional Fast Fourier Transform.
Network Architecture
Figure 1 depicts the architecture of the proposed SMR-Transformer, which contains encoder and decoder sub-network like U-Net[7], but the convolution operations were replaced by transformer blocks. In the encoder sub-network, six transformer blocks followed by three series merging operations were performed for learning global contextual features. Symmetric to the encoder, three series-expanding operations followed by six transformer blocks were used to generate recovery feature maps for reconstruction. The transformer first learned global information along the series dimension through multi-head attention module, then encoded or decoded every series image through multi-layer perceptron (MLP) layer.
Network Training
The SMR-transformer network randomly picked the 4X4XN_s patch as input(N_s is image number included in the series). Learning rate was set to 1e-4, then decayed by 10 times at 30 and 150 epochs. The number of training epochs were set as 300. Loss function was mean square error (MSE) and optimizer was Adam. For comparison, the regular Res-U-Net was also trained using N_xXN_yXN_s patch size, in which N_x, N_y were chosen as the maximum number allowed by the GPU memory (FastCEST: 128X112X32, OCMR: 160X144X32, HCP:112X96X288).

Results and Discussion

We evaluated the performance of proposed SMR-Transformer on three datasets, with the 3D Res-U-Net for comparison. Compared with Res-U-Net, our SMR-Transformer achieved higher PSNR and SSIM, and lower MSE on both the HCP and OCMR datasets; while for FastCEST dataset the two networks had similar PSNR and MSE (Table 1). For FastCEST, SMR-Transformer displayed less noisy contrast maps than Res-U-Net, including MTR asymmetry and Lorentzian difference at the amide and the NOE frequency (Figure 2). For HCP dataset, the colored fractional anisotropy (CFA) maps calculated by the software FSL showed that, SMR-Transformer reconstructed more detail information than Res-U-Net (Figure 3). For cardiac cining images of OCMR, SMR-Transformer and Res-U-Net both displayed well reconstruction on test images. However, both methods could not well estimate cardiac motion. A possible reason is that, cine imaging is more of a spatial-temporal imaging modality, that is different from the above two multi-contrast dataset. In future, network architecture of SMR-Transformer may need adjustment to give more attention on spatial dimension.
Overall, SMR-Transformer outperformed Res-U-net for three kinds of serial images. The credits may be transformer’s advantage on directly global attention to series dimension, which overcomes the localized shortage of convolution operations. In current implementation, requires a fully-sampled serial image as a reference input. There are two reasons for that. (1) A high-quality image can help improving the reconstruction of overall series. (2) The spatial size of transformer inputs was only 4*4 now, therefore an unaliased image is needed to de-alias other images effectively.

Conclusion

We proposed a pure transformer network architecture for accelerating serial MR imaging and evaluated it on three datasets. Quantitative results of reconstruction performance, as well as the image visualization, demonstrated that our SMR-Transformer outperformed 3D Res-U-Net. Additionally, the proposed SMR-transformer method could enable the acceleration up to 10 times on different datasets, which is promising in acceleration of serial MR imaging.

Acknowledgements

The authors acknowledged funding from National Natural Science Foundation of China (82071914). and the startup package from Tsinghua University to Dr. Song.