1499

An AI-based pipeline for automatic image fusion of cardiac CTCA and perfusion MRI
Wenting Jiang1, Ming-Yen Ng1, TsunHei Sin1, and Peng Cao1
1Department of Diagnostic Radiology, the University of Hong Kong, Hong Kong, Hong Kong

Synopsis

Keywords: Myocardium, Cardiovascular

Motivation: The 3D fusion of coronary structure and myocardial blood flow data helps to reduce the misallocation of affected vessels to their associated myocardial territories.

Goal(s): An AI-based pipeline has been developed that uses advanced machine learning algorithms to automatically fuse images from cardiac CTCA and perfusion MRI.

Approach: The pipeline includes an automatic reorientation of 3D CT coronary angiography and fusion with stress cardiovascular magnetic resonance images.

Results: we achieved 3D fusion of CTCA and CMR establishing a correlation between coronary artery stenosis and stress-induced myocardial hypoperfusion.

Impact: the pipeline can assist in clinical assessments of coronary artery disease.

INTRODUCTION

The assessment of coronary artery stenosis and myocardial ischemia is crucial for the diagnosis and treatment of coronary artery disease (CAD). To evaluate both aspects, a multimodality imaging approach is recommended, which includes CT coronary angiography (CTCA), single photon emission CT (SPECT), and stress cardiovascular magnetic resonance (CMR) imaging [1]. Experts match diseased coronary artery segments with corresponding myocardial territories sequentially. Now 3D fusion technique for coronary structure and myocardial blood flow data can reduce potential misallocation of affected vessels to their associated myocardial territories.
Previous studies[2][3][4][5] have tried to fuse CTCA and CMR images, but these methods often require significant amount of time and manual intervention. In light of the rapid advancements in artificial intelligence, our objective is to propose a fully automated framework based on deep learning for cardiac CTCA and CMR fusion. The proposed framework utilizes raw CTCA and CMR data as inputs, and can perform a series of processes, including CTCA reorientation, registration, CMR perfusion quantification, CTCA coronary artery extraction, and 3D fusion of CTCA and CMR, ultimately generating the desired image.
In this study, we proposed an AI-based pipeline for the automatic reorientation of 3D CT coronary angiography (CTCA) and the fusion of CTCA with stress cardiovascular magnetic resonance (CMR) images. The pipeline aims to establish a correlation between coronary artery stenosis and stress-induced myocardial hypoperfusion, thereby assisting in the clinical assessment of coronary artery disease (CAD).

METHODS

This retrospective study obtained approval from the institutional review board. A cohort of 93 patients who underwent both CTCA and stress CMR imaging at our hospital were included for network training and evaluation purposes. The CTCA examinations were performed using the Philips iCT256 scanner, following standardized protocols established locally. Our study utilized the following CMR imaging parameters: 3 short axis slices, 90 time points, field of view (FOV) = 30–40 cm, voxel size = 1.7–2.0 × 2.2–2.7 mm.
Figure 1 illustrates key steps involved in our pipeline. CTCA reorientation using RestNet-STN and CMR corpping to focus on the heart region are applied initially. The reoriented CTCA results are refined by aligning the long axis of the left ventricle (LV) horizontally along the RL direction in the coronal plane. The LV outline on CMR images is generated leveraging the circular shape of LV in short-axis view. To establish spatial alignment between the reoriented CTCA and CMR images, registration is performed based on the LV contour and cropped field of view (FOV). Coronary artery extraction from reoriented CTCA images and myocardial perfusion quantification on CMR are applied simultaneously. 3D rendering fusion results of CTCA and CMR are obtained using 3D Slicer[8], enabling a comprehensive visualization of the fused images.
Within this framework, we proposed a self-supervised network ResNet-STN for CTCA reorientation (Figure 2). The predicted rotation angles by ResNet18 module are used to define a rotation matrix, denoted as R. Since our objective is solely focused on CTCA reorientation without any translational transformations, the translation matrix is set to zero. The obtained rotation matrix R and translation matrix M are combined as [R M]' and serve as the input to the spatial transformer network (STN)[9] for transforming the input CTCA images into reoriented output images. By utilizing STN, the optimal rotation parameters can be learned for the CTCA task, even without ground truth transformation information. Additionally, we introduce a novel approach where corresponding short-axis CMR images are incorporated as cross-modal supervision for the middle slice of the output volume. This cross-modal supervision serves as a guiding signal to facilitate the learning of CTCA reorientation.

RESULTS

Figure 3 displays the final reorientation results of CTCA on the testing cohort. ResNet-STN produced satisfactory reorientation outcomes. The LV's near-circular appearance shows that the original CTCA images are effectively rotated into the correct short-axis view, highlighting the accuracy and effectiveness of our model in achieving the desired reorientation.Figure 4 compares the CTCA reorientation results before and after the refinement process. As depicted, the refinement step significantly improves the results, exhibiting enhanced quality and alignment. This validates the effectiveness of our approach.
Figure 5 showcases the final 3D image fusion for one testing case, which combines the reoriented CTCA scans with the perfusion quantification map obtained from CMR imaging. This fusion enables a comprehensive visualization of the coronary arteries, along with the quantification of myocardial perfusion. The integration of these modalities provides valuable insights into the anatomical and functional aspects of the cardiac structures.

CONCLUSION

We proposed a fully automated framework based on deep learning for cardiac CTCA and CMR fusion.

Acknowledgements

No acknowledgement found.

References

[1] Authors/Task Force members et al., “2014 ESC/EACTS Guidelines on myocardial revascularization: The Task Force on Myocardial Revascularization of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS)Developed with the special contribution of the European Association of Percutaneous Cardiovascular Interventions (EAPCI),” Eur. Heart J., vol. 35, no. 37, pp. 2541–2619, Oct. 2014, doi: 10.1093/eurheartj/ehu278.

[2] J. von Spiczak et al., “Fusion of CT coronary angiography and whole-heart dynamic 3D cardiac MR perfusion: building a framework for comprehensive cardiac imaging,” Int. J. Cardiovasc. Imaging, vol. 34, no. 4, pp. 649–660, Apr. 2018, doi: 10.1007/s10554-017-1260-6.

[3] P. Stolzmann et al., “Image fusion of coronary CT angiography and cardiac perfusion MRI: a pilot study,” Eur. Radiol., vol. 20, no. 5, pp. 1174–1179, May 2010, doi: 10.1007/s00330-010-1746-2.

[4] O. F. Donati et al., “3D Fusion of Functional Cardiac Magnetic Resonance Imaging and Computed Tomography Coronary Angiography: Accuracy and Added Clinical Value,” Invest. Radiol., vol. 46, no. 5, p. 331, May 2011, doi: 10.1097/RLI.0b013e3182056caf.

[5] “Image fusion of coronary CT angiography and cardiac perfusion MRI.” Accessed: Nov. 07, 2023. [Online]. Available: https://publica.fraunhofer.de/entities/publication/79f39d26-b7a3-4e84-9eb0-994c8857b785

[6] V. F. Leavers, “Which Hough Transform?,” CVGIP Image Underst., vol. 58, no. 2, pp. 250–264, Sep. 1993, doi: 10.1006/ciun.1993.1041.

[7] “Physics-informed neural networks for myocardial perfusion MRI quantification - ScienceDirect.” Accessed: Nov. 06, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841522000512

[8] S. Pieper, M. Halle, and R. Kikinis, “3D Slicer,” in 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821), Apr. 2004, pp. 632-635 Vol. 1. doi: 10.1109/ISBI.2004.1398617.

[9] M. Jaderberg, K. Simonyan, A. Zisserman, and koray kavukcuoglu, “Spatial Transformer Networks,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2015. Accessed: Nov. 07, 2023. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2015/hash/33ceb07bf4eeb3da587e268d663aba1a-Abstract.html

Figures

Figure 1. The overall pipeline of our work includes 5 main steps: 1) Orginal CTCA rotation to short-axis view using our RestNet-STN and original CMR cropping to focus primarily on the heart region; 2) CTCA rotation refinement utilizing the LV mask generated by the Frangi Filter and LV contouring on CMR through Hough transformation; 3) Registration between reoriented CTCA and CMR; 4) Perfusion quantification on CMR by employing a physics-informed neural network[7] and coronary artery extraction on CTCA using the Frangi Filter; 5) 3d image fusion of CTCA and CMR using 3D Slicer[8].

Figure 2. The proposed rotation network (ResNet-STN) architecture. With original CTCA volume as input, the ResNet18 module is employed to predict three rotation angles based on the input CTCA volume, then the transformation matrix is defined by the predicted angles, which serves as the input to the spatial transformer network (STN)[9] to transform the input CTCA images into reoriented output images. A cross-modal supervision is introduced between the middle slice of the output volume and corresponding short-axis CMR images.

Figure 3. CTCA reorientation results produced by our pipeline.

Figure 4. Comparison of CTCA reorientation result before and after refinement. The first row is the result generated by the model without refinement, and the second row is the result after refinement.

Figure 5. 3d Image fusion of CTCA and CMR on short-axis view.

Proc. Intl. Soc. Mag. Reson. Med. 32 (2024)
1499
DOI: https://doi.org/10.58530/2024/1499