Yue Hu^{1}, Peng Li^{1}, and Dong Nan^{2}

^{1}Harbin Institute of Technology, Harbin, China, ^{2}The First Affiliated Hospital of Harbin Medical University, Harbin, China

We propose a novel algorithm for the super-resolution of brain MR images based on feature regularized DIP network, where no prior training pairs are required. We formulate the network by including the total variation (TV) term as the sparsity regularization and the Laplacian as the sharpness regularization. The network is iteratively updated using the image feature regularizations and the measured image. Numerical experiments demonstrate the improved performance offered by the proposed method.

where $$$f$$$ is an arbitrary continuous transformation function that maps the image $$$\mathbf X$$$ to a low-resolution image, and $$$\mathbf n$$$ is the additive noise. The image super-resolution problem can thus be formulated as: $$ \mathbf X^*=\min_{\mathbf X}\|\mathbf Y-f(\mathbf X) \|_2^2 +\lambda {\cal R}(\mathbf X) $$ Where $$${\cal R} (\mathbf X)$$$ is the regularization term, and $$$\lambda$$$ is the balancing parameter. In the DIP scheme, the CNN itself is used as the regularization, which makes the optimization problem for DIP as follows: $${\theta}^*=\min_{\theta} \|\mathbf y-f(N_{\theta}(\mathbf Z))\|^2_2;\;\rm{s.t.}\mathbf X=N_{{\theta}^*}(\mathbf Z)$$ Here, $$$N_{\theta}$$$ is the deep neural network parameterized by $$$\theta$$$. The input of the network is a noise vector $$$\mathbf Z$$$. Due to the lack of training data, the DIP scheme has sub-optimal performance when the downsampling factor is large. We propose to include total variation (TV) and the Laplacian in the DIP framework to promote the piecewise-smoothness and the sharp features of the image. We formulate the proposed method as follows: $${\theta}^*=\min_{\theta}\|\mathbf Y-f(\mathbf X)\|^2_2 +\lambda_1 {\cal R}_{\rm{TV}}(\mathbf X) +\lambda_2 {\cal R}_{\rm{Laplacian}}(\mathbf X)$$ Here $$$\mathbf X=N_{{\theta}^*}(\mathbf Z)$$$, $$${\cal R}_{\rm{TV}}(\mathbf X)=\sum\limits_{i=1}^N\sqrt{|D_x|^2+|D_y|^2}$$$ is the TV regularization, where $$$D_x$$$ and $$$D_y$$$ are the finite difference operators along $$$x$$$ and $$$y$$$ dimensions. $$${\cal R}_{\rm{Laplacian}}(\mathbf X)=\sum\limits_{i=1}^N|\nabla^2\mathbf X| $$$ is the Laplacian regularization. $$$\lambda_1$$$ and $$$\lambda_2$$$ are balancing parameters. Fig.1 shows the implementation block diagram of the proposed algorithm. Here, we use a UNet-based network architecture, illustrated in Fig.2. Note that the TV and Laplacian priors, along with the difference between the low-resolution image and the output image, are incorporated in the network. The detailed illustration of the network is shown in the caption of Fig.2. We use zero-filled images as the input of the network. Note that the proposed network itself is used as a prior. By including the TV and Laplacian penalties, the method is able to promote the sparsity and the sharpness of the recovered image.

1. Shi F, Cheng J, Wang L, et al. LRTV: MR image super-resolution with low-rank and total variation regularizations. IEEE transactions on medical imaging. 2015;34(12):2459-2466.

2. Pham C H, Ducournau A, Fablet R, et al. Brain MRI super-resolution using deep 3D convolutional networks. 2017 IEEE 14th International Symposium on Biomedical Imaging. 2017:197-200.

3. Shi J, Li Z, Ying S, et al. MR image super-resolution via wide residual networks with fixed skip connection. IEEE journal of biomedical and health informatics, 2018;23(3):1129-1140.

4. Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018:9446-9454.

5. Cherukuri V, Guo T, Schiff S J, et al. Deep MR Image Super-Resolution Using Structural Priors. 2018 25th IEEE International Conference on Image Processing. 2018:410-414.

Figure 1. The implementation block diagram of the proposed method.

Figure 2. The network architecture of the proposed method. The network includes 6 coding blocks, 7 decoding blocks, and 6 skip connections. For each convolutional layer in coding block and decoding block, the convolution kernel size is 3x3. The convolution step and the zero padding layer number is 1. In the skip connection, the convolution kernel size is 1x1 with no padding. We use anAvgpool2d layer with a kernel size of 2x2 and a step size of 2 in the downsampling module of the coding block.

Figure 3. Super-resolution of the brain MR image with downsampling factor of 2. We compare the proposed method with the standard TV and the original DIP method. It is observed that although no prior training data is involved, the proposed method outperforms the other methods in visual and quantitative evaluation.The detailed features are preserved better with minimum artifacts because of the regularizations used.

Figure 4. Super-resolution of the brain MR image with downsampling factor of 4. It is shown that the proposed method is able to recover the details better.