Nicholas Dwork^{1}, John M. Pauly^{1}, and Jorge Balbas^{2}

Fusing a lower resolution color image with a higher resolution monochrome image is a common practice in medical imaging. By incorporating spatial context and/or improving the Signal-to-Noise ratio, the fused image provides clinicians with a single frame of the most complete diagnostic information. In this paper, image fusion is formulated as a convex optimization problem which avoids image decomposition and permits operations at the pixel level. This results in a highly efficient and embarrassingly parallelizable algorithm based on widely available robust and simple numerical methods that realizes the fused image as the global minimizer of the convex optimization problem.

Let $$$X=(R,G,B)$$$ denote the color image, and $$$Y$$$ the monochrome image, with $$$R,G,B,Y\in\mathbb{R}^{M\times N}$$$ corresponding to the red, green, blue, and monochrome channels respectively. All values in $$$X$$$ and $$$Y$$$ are in $$$[0,1]$$$. The fused image $$$F=(F_R,F_G,F_B)$$$ results from solving the following problem:

\begin{align} \text{minimize} & \hspace{8pt} \|X-F\|_F^2 + \gamma \, \|f(F_R,F_G,F_B)-Y\|_F^2 \\ \text{subject to} & \hspace{8pt} 0 \leq F \leq 1, \end{align}

where $$$\|\cdot\|_F$$$ denotes the Frobenius norm and $$$f(F_R,F_G,F_B)=w_R\,F_R + w_G\,F_G + w_B\,F_B$$$ is a weighted average of the three color channels; $$$\gamma$$$ is a parameter set by the user. Note that this problem is completely separable across pixels. The result of each pixel is independent of any other pixel. The value of the $$$(i,j)^{\text{th}}$$$ pixel in the fused image can be found by solving the following problem:

\begin{align}\text{minimize} & \hspace{8pt} \left\| \underbrace{ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ w_R \sqrt{\gamma} & w_G \sqrt{\gamma} & w_B \sqrt{\gamma} \end{bmatrix} }_{\boldsymbol{A}} \, \underbrace{ \begin{bmatrix} \left(F_R\right)_{ij} \\ \left(F_G\right)_{ij} \\ \left(F_B\right)_{ij} \end{bmatrix} }_{\boldsymbol{x}_{ij}} - \underbrace{ \begin{bmatrix} R_{ij} \\ G_{ij} \\ B_{ij} \\ Y_{ij} \sqrt{\gamma} \end{bmatrix} }_{\boldsymbol{b}_{ij}} \right\|_2^2 \\ \text{subject to} & \hspace{8pt} 0 \leq F \leq 1 \end{align}

The solution to this problem can be found by minimizing $$$\|\boldsymbol{A}\,\boldsymbol{x}_{ij}-\boldsymbol{b}_{ij}\|_2$$$ and then performing a projection of the result onto the cube $$$[0,1]^3$$$ along the line $$$l$$$ parameterized by $$$\lambda$$$ in the following equation:

\begin{equation*}l(\lambda) = \left\{ \begin{array}{cl} (R_{ij},G_{ij},B_{ij}) + \lambda\,w & \hspace{4pt} \text{if} \; F_{ij} > 1 \\ (R_{ij},G_{ij},B_{ij}) - \lambda\,w & \hspace{4pt} \text{otherwise} \end{array} \right..\end{equation*}An approximate more computationally efficient solution is to perform a Euclidean projection onto $$$[0,1]^3$$$ (instead of a projection along $$$l$$$). All results are shown using this approximation.

This section presents results of the CLS fusion algorithm for several different applications and compares them to existing fusion algorithms. The weight vector $$$w=(1/3,1/3,1/3)$$$ so that $$$f(X)$$$ yields the intensity channel of the color image [3].

Figure 1 shows results of a 3'-[$$$^{\text{18}}$$$F] fluoro-3'-deoxythymidine PET image and a T1 weighted MR image of a living BALB/c mouse bearing a CT26 colon carcinoma [4]. Increasing $$$\gamma$$$ increases the intensity of the monochrome image in the fused result. These results are compared to alpha-blending [5]. For comparable levels of intensity from the monochrome image, the CLS-fusion algorithm is able to retain much more of the information from the color imagery.

Figure 2 shows MR flow results collected using the Variable-Density Sampling and Radial View-ordering (VDRad) sequence [2]. The velocity in each dimension is represented as a different color; Right-Left (RL) / Anterior-Posterior (AP) / Superior-Inferior (SI) motion are represented by Red/Green/Blue, respectively. Larger velocities are represented with larger color values. Figures 2c and 2d show the result of Wavelet fusion [6] and CLS fusion algorithms, respectively. The Wavelet result shows artifacts resulting from the structure of the Wavelet kernels; these artifacts are absent in the CLS result. Additionally, CLS retains better the color hue better.

[1 ] Gerald Antoch and Andreas Bockisch. Combined PET/MRI: A New Dimension in Whole-body OncologyImaging? European Journal of Nuclear Medicine and Molecular Imaging, 36(1):113–120, 2009.

[2] Joseph Y Cheng, Kate Hanneman, Tao Zhang, Marcus T Alley, Peng Lai, Jonathan I Tamir, Martin Uecker, John M Pauly, Michael Lustig, and Shreyas S Vasanawala. Comprehensive Motion-compensated Highly Accelerated 4D flow MRI with Ferumoxytol Enhancement for Pediatric Congenital Heart Disease. Journal of Magnetic Resonance Imaging, 43(6):1355–1368, 2015.

[3] Jorge Nunez, Xavier Otazu, Octavi Fors, Albert Prades, Vicenc Pala, and Roman Arbiol. Multiresolution-based Image Fusion with Additive Wavelet Decomposition. IEEE Transactions on Geoscience and Remote Sensing, 37(3):1204–1211, 1999.

[4] Martin S Judenhofer, Hans F Wehrl, Danny F Newport, Ciprian Catana, Stefan B Siegel, Markus Becker,Axel Thielscher, Manfred Kneilling, Matthias P Lichy, Martin Eichner, et al. Simultaneous PET-MRI: a New Approach for Functional and Morphological Imaging.Nature Medicine, 14(4):459–465, 2008.

[5] Thomas Porter and Tom Duff. Compositing Digital Images. In ACM Siggraph Computer Graphics,volume 18, pages 253–259. ACM, 1984.

[6] Krista Amolins, Yun Zhang, and Peter Dare. Wavelet Based Image Fusion Techniques — An Introduction,Review and Comparison.ISPRS Journal of Photogrammetry and Remote Sensing, 62(4):249–263, 2007.

(Top) CLS fusion results, (Bottom) Alpha-blending results. (a) $$$\gamma=1$$$, (b) $$$\gamma=3$$$, (c) $$$\gamma=10$$$; (d) $$$\alpha=0.25$$$, (e) $$$\alpha=0.5$$$, (f) $$$\alpha=0.75$$$. Source images were adapted by permission from Macmillan Publishers Ltd: Nature Medicine: Martin S Judenhofer and Hans F Wehrlet al. Simultaneous PET-MRI: a New Approach for Functional and Morphological Imaging. Nature Medicine. 2008;14:459--465, \copyright (2008).

(a) Maximum intensity projection anatomical MR image, (b) maximum intensity projection MR flow image where red/green/blue represent RL/AP/SI motion respectively, (c) Wav-IHS fusion result, (d) CLS fusion result with $$$\gamma=9$$$. The data presented here was gathered with Institutional Review Board (IRB) approval, Health Insurance Portability and Accountability Act (HIPAA) compliance, and patient informed assent/consent.