A typical clinical imaging protocol covers a large number of different image contrasts and, in the age of multi-modality systems, even different imaging modalities. The standard approach of separate data acquisition and reconstruction is inefficient and wasteful, given the substantial amount of anatomical correlations between these datasets. On the other hand, given the different imaging physics and contrast mechanisms, each subset can also contain unique features which may be missing or appear differently in others. In this work, we propose a generalized reconstruction framework for simultaneous reconstruction of images from different contrasts and modalities that takes advantage of the inherent redundancies while ensuring that the unique features and signal values of each individual image data set are preserved. Going beyond mere relative contrast variations, we show that PET and MR-Fingerprinting (MRF) [1] can jointly be reconstructed while preserving the quantitative nature of the underlying methods.

In the proposed framework, we regard images from different contrasts and modalities as different channels of a multi-channel image $$$u=(u_i)_{i=1}^{n}$$$, each $$$u_i$$$ corresponding to one contrast or modality. Reconstruction is achieved by solving the convex optimization problem:

$$\min_{u=(u_i)_{i=1}^{n}}\text{TGV}_{\alpha}^2(u)+\sum_{i=1}^n\lambda_iD(u_i,d_i),$$

where $$$D(u_i,d_i)$$$ is the data fidelity for data $$$d_i$$$. It is given either as $$$D(u_i,d_i)=\left\|E_iu_i-d_i\right\|_2^2$$$ for MR, $$$E_i$$$ being the MR forward operator, or $$$D(u_i,d_i)=\int\left(Au_i-d_i\log(Au_i )\right)$$$ for PET, $$$A$$$ being the PET forward operator. The multi-channel regularizer TGV [2,3] is defined as:

$$\text{TGV}_{\alpha}^2(u)=\min_w\alpha_1\| |\nabla u–w|_\text{nuc}\|_1+\alpha_0\| |\mathcal{E}w|_\text{frob}\|_1,$$

where the gradient $$$\nabla u$$$ of the multi-channel image function is a matrix valued function mapping to $$$\mathbb{R}^{n \times 3}$$$. The symmetrized gradient $$$\mathcal{E}w$$$ is a tensor valued function mapping to $$$\mathbb{R}^{n\times 3\times 3}$$$ and is defined as $$$\mathcal{E}w=\frac{1}{2}(Jw+(Jw)^T)$$$ with $$$Jw$$$ being the Jacobian of $$$w$$$. Coupling of different channels is achieved with a pointwise nuclear-norm $$$|\cdot |_\text{nuc}$$$ and pointwise Frobenius norm $$$|\cdot |_\text{frob}$$$. The nuclear-norm employs a pointwise penalization of the $$$\ell^1$$$ norm of the singular values of each matrix, corresponding to the gradient of the multichannel image at one voxel. The nuclear-norm can be considered as the convex relaxation of the rank, thus, nuclear-norm minimization has a low-rank enforcing effect. As each column of the matrix corresponds to the spatial gradient of one image channel, this enforces aligned edges amongst different channels. The number of non-zero singular values is independent of signal intensity and edge direction and is identical for a unique edge and two aligned edges. The latter makes the regularizer robust against unwanted modification of unique features. This is illustrated conceptually for the case of a single MR+PET contrast in Figure 1.

Numerical simulations were performed using a brain phantom [4]. Specific lesions were added to MR and PET. MPRAGE, T2w and FLAIR contrast were simulated as follows: Matrix size 176×176×30, 8 receive coil channels, radial acquisition from 32 spokes. K-space data were corrupted by adding complex Gaussian noise. PET rawdata was simulated according to the crystal geometry of a whole body MR-PET. The total number of counts corresponded to a 10min FDG PET head scan.

In-vivo data of a high-grade-glioma patient was acquired on a clinical 3T MR-PET system (Siemens Biograph mMR) using a 12-element head coil array. The study was approved by the IRB, written informed consent was obtained. The protocol consisted of 6 min simultaneous MRF (voxel size 1.5×1.5×mm², 176×176 matrix, 30 slices, slice thickness 3mm) and PET data acquisition. Data from 32 compressed MR fingerprint sets [5] containing 75 and 30 spokes were reconstructed using the proposed approach as well as a conventional MRF reconstruction and EM (PET).

Figure 2 shows results of the numerical simulation and RMSE to the ground-truth. Figure 3 visualizes the improved PET resolution by cross-sectional profiles. Simultaneous reconstruction not only preserves quantitative PET values but shows stronger agreement with the underlying ground-truth. This is confirmed by quantitative analysis in Figure 4. Figure 5 shows quantitative MR relaxation parameter maps and PET images of the high-grade-glioma MRF measurement.

The proposed method has several unique features that makes multi-contrast and multi-modality image reconstruction feasible: It can deal with arbitrary sampling operators that describe the different imaging physics of the individual modalities and integrates different data terms for modalities with different noise models. The proposed regularizer makes use of shared features, as demonstrated by the improved resolution of PET by using the sharp edge information from the MR contrasts, without compromising individual signal values and contrasts, as demonstrated by improvements in PET-signal and MRF relaxation parameter values. In the combination with MR Fingerprinting, this approach allows a 6-minute comprehensive whole brain coverage exam that provides both quantitative PET and MR-relaxation parameters.