5152
A motion aware DNN model with edge focus loss and quality control for short-axis left ventricle segmentation of cine MR sequences1Radiology and Nuclear Medicine, Capital Medical University XuanWu Hospital, Beiing, China, 2School of Biomedical Engineering, Capital Medical University, Beijing, China
Synopsis
Motivation: We propose a motion aware DNN model for cardiac sequence segmentation.
Goal(s): We construct an in-house dataset which has three advantages: segmentation annotations covering the cardiac cycle; comprehensive annotations, including the annotations of interventricular groove structure; fine annotations of endocardium.
Approach: We propose an edge focus loss to make the segmented boundaries be consistent with the local gradient of the input images and propose a quality control method based on Image Moments to filter abnormal predictions.
Results: The experimental results highlight the accuracy of the proposed model, and the fine segmentation results could be used to estimate accurate clinical indicators for clinical diagnosis.
Impact: In experiments, we compare the proposed model with 12 state-of-the-art segmentation models, and our model have obtained the highest accuracy for the segmentation and the highest PCC on the 17-segment model.
Purpose
Accurate segmentation of the LV myocardium is the key step of automatic assessment of cardiac function. However, the current models mainly focus on the segmentation of the end-diastolic (ED) and the end-systolic (ES) frames of cine MR, and lack the attention to myocardial motion in the entire cardiac cycle. Besides, due to lack the fine segmentation tools, the simplified approach excluding papillary muscles and trabeculae from the myocardium is applied in clinical routine.Methods
In this paper we propose a motion aware DNN model for short-axis left ventricle segmentation of cine MR sequences. A new motion attention layer based on Bidirectional ConvLSTM is proposed to encode shapes in both temporal directions, and an edge focus loss function is proposed to make the model pay more attention to the pixels on the boundaries. Besides, we propose a quality control method based on Image Moments to filter out the abnormal segmentations of the epicardium before subsequent analyses.Results
Compared with 12 state-of-the-art segmentation models, the proposed model has obtained the high segmentation accuracy with DC of 94%, 98%, HD of 8.68, 7.98 for the endocardium and the epicardium respectively. On the 17-segment model, the proposed model has obtained the highest Pearson Correlation Coefficient (PCC) at 14 of 17 segments, and the mean PCC of 85% for all segments.Conclusion
The experimental results highlight the accuracy of the proposed model, and the fine segmentation results could be used to estimate accurate clinical indicators for clinical diagnosis.Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 61672362).References
[1] D. Ouyang et al., “Video-based AI for beat-to-beat assessment of cardiac function,” Nature, vol. 580, no. 7802, pp. 252–256, Apr. 2020, doi: 10.1038/s41586-020-2145-8.
[2] M. D. Cerqueira et al., “Standardized Myocardial Segmentation and Nomenclature for Tomographic Imaging of the Heart,” Circulation, vol. 105, no. 4, pp. 539–542, Jan. 2002, doi: 10.1161/hc0402.102975.
[3] A. Zakeri, A. Hokmabadi, N. Ravikumar, A. F. Frangi, and A. Gooya, “A probabilistic deep motion model for unsupervised cardiac shape anomaly assessment,” Med Image Anal, vol. 75, Jan. 2022, doi: 10.1016/j.media.2021.102276.
[4] S. Vesal, M. Gu, A. Maier, and N. Ravikumar, “Spatio-Temporal Multi-Task Learning for Cardiac MRI Left Ventricle Quantification,” IEEE J Biomed Health Inform, vol. 25, no. 7, pp. 2698–2709, Jul. 2021, doi: 10.1109/JBHI.2020.3046449.
[5] C. Chen et al., “Deep learning for cardiac image segmentation: A review,” Nov. 2019, doi: 10.3389/fcvm.2020.00025.
[6] F. Y. Li, W. Li, X. Gao, and B. Xiao, “A Novel Framework with Weighted Decision Map Based on Convolutional Neural Network for Cardiac MR Segmentation,” IEEE J Biomed Health Inform, 2021, doi: 10.1109/JBHI.2021.3131758.
[7] W. Xue et al., “Left Ventricle Quantification Challenge: A Comprehensive Comparison and Evaluation of Segmentation and Regression for Mid-Ventricular Short-Axis Cardiac MR Data,” IEEE J Biomed Health Inform, vol. 25, no. 9, pp. 3541–3553, Sep. 2021, doi: 10.1109/JBHI.2021.3064353.
[8] X. Yang, Y. Zhang, B. Lo, D. Wu, H. Liao, and Y. T. Zhang, “DBAN: Adversarial Network with Multi-Scale Features for Cardiac MRI Segmentation,” IEEE J Biomed Health Inform, vol. 25, no. 6, pp. 2018–2028, Jun. 2021, doi: 10.1109/JBHI.2020.3028463.
[9] C. Biffi et al., “Explainable Anatomical Shape Analysis through Deep Hierarchical Generative Models,” IEEE Trans Med Imaging, vol. 39, no. 6, pp. 2088–2099, Jun. 2020, doi: 10.1109/TMI.2020.2964499.
[10] J. Sun, F. Darbehani, M. Zaidi, and B. Wang, “SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation,” MICCAI 2020: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, pp. 797–806, Jan. 2020, [Online]. Available: http://arxiv.org/abs/2001.07645
[11] J. H. Riffel et al., “Cardiovascular magnetic resonance of cardiac morphology and function: impact of different strategies of contour drawing and indexing,” Clinical Research in Cardiology, vol. 108, no. 4, pp. 411–429, Apr. 2019, doi: 10.1007/s00392-018-1371-7.
[12] L. K. Tan, Y. M. Liew, E. Lim, and R. A. McLaughlin, “Convolutional neural network regression for short-axis left ventricle segmentation in cardiac cine MR sequences,” Med Image Anal, vol. 39, pp. 78–86, Jul. 2017, doi: 10.1016/j.media.2017.04.002.
[13] K. N. Wang et al., “AWSnet: An auto-weighted supervision attention network for myocardial scar and edema segmentation in multi-sequence cardiac magnetic resonance images,” Med Image Anal, vol. 77, Apr. 2022, doi: 10.1016/j.media.2022.102362.
[14] A. Ostvik et al., “Myocardial Function Imaging in Echocardiography Using Deep Learning,” IEEE Trans Med Imaging, vol. 40, no. 5, pp. 1340–1351, May 2021, doi: 10.1109/TMI.2021.3054566.
[15] F. Uslu, M. Varela, G. Boniface, T. Mahenthran, H. Chubb, and A. A. Bharath, “LA-Net: A Multi-Task Deep Network for the Segmentation of the Left Atrium,” IEEE Trans Med Imaging, vol. 41, no. 2, pp. 456–464, Feb. 2022, doi: 10.1109/TMI.2021.3117495.
[16] M. H. Jafari, N. van Woudenberg, C. Luong, P. Abolmaesumi, and T. Tsang, “Deep bayesian image segmentation for a more robust ejection fraction estimation,” in Proceedings - International Symposium on Biomedical Imaging, Apr. 2021, vol. 2021-April, pp. 1264–1268. doi: 10.1109/ISBI48211.2021.9433781.
[17] S. Dong et al., “DeU-Net: Deformable U-Net for 3D Cardiac MRI Video Segmentation,” in MICCAI 2020: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020, pp. 98–107. doi: 10.1007/978-3-030-59719-1_10.
[18] S. Dong et al., “DeU-Net 2.0: Enhanced deformable U-Net for 3D cardiac cine MRI segmentation,” Med Image Anal, vol. 78, p. 102389, May 2022, doi: 10.1016/j.media.2022.102389.
[19] F. Uslu and M. Varela, “SA-net: A sequence aware network for the segmentation of the left atrium in cine MRI datasets,” in Proceedings - International Symposium on Biomedical Imaging, Apr. 2021, vol. 2021-April, pp. 766–769. doi: 10.1109/ISBI48211.2021.9434147.
[20] S. Guo, L. Xu, C. Feng, H. Xiong, Z. Gao, and H. Zhang, “Multi-level semantic adaptation for few-shot segmentation on cardiac image sequences,” Med Image Anal, vol. 73, p. 102170, Oct. 2021, doi: 10.1016/j.media.2021.102170.
[21] F. Guo, M. Ng, G. Kuling, and G. Wright, “Cardiac MRI Segmentation With Sparse Annotations: Ensembling Deep Learning Uncertainty and Shape Priors,” Med Image Anal, p. 102532, Jul. 2022, doi: 10.1016/j.media.2022.102532.
[22] Wenjun Yan, Yuanyuan Wang, Zeju Li, Rob J. van der Geest, and Qian Tao, “Left Ventricle Segmentation via Optical-Flow-Net from Short-Axis Cine MRI: Preserving the Temporal Coherence of Cardiac Motion,” Medical Image Computing and Computer Assisted Intervention - MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, p.IV, pp. 613–621, 2018.
[23] Chen Qin et al., “Joint learning of motion estimation and segmentation for cardiac MR image sequences,” Medical Image Computing and Computer Assisted Intervention - MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, p.II, pp. 472–480, 2018.
[24] Ming Li et al., “Recurrent aggregation learning for multi-view echocardiographic sequences segmentation,” MICCAI 2019.
[25] S. Chen, K. Ma, and Y. Zheng, “TAN: Temporal Affine Network for Real-Time Left Ventricle Anatomical Structure Analysis Based on 2D Ultrasound Videos,” Computer Vision and Pattern Recognition, Apr. 2019.
[26] Hongrong Wei et al., “Temporal-consistent segmentation of echocardiography with co-learning from appearance and shape,” MICCAI 2020.
[27] Mohammad H. Jafari et al., “A Unified Framework Integrating Recurrent Fully-Convolutional Networks and Optical Flow for Segmentation of the Left Ventricle in Echocardiography Data,” International Workshop on Deep Learning in Medical Image Analysis, pp. 29–37, 2018.
[28] O. S. Al-Kadi, “Spatio-Temporal Segmentation in 3-D Echocardiographic Sequences Using Fractional Brownian Motion,” IEEE Trans Biomed Eng, vol. 67, no. 8, pp. 2286–2296, Aug. 2020, doi: 10.1109/TBME.2019.2958701.
[29] D. Yang, Q. Huang, K. Mikael, S. al Al’aref, L. Axel, and D. Metaxas, “Mri-Based Characterization of Left Ventricle Dyssynchrony with Correlation to Crt Outcomes,” in Proceedings - International Symposium on Biomedical Imaging, Apr. 2020, vol. 2020-April, pp. 822–825. doi: 10.1109/ISBI45749.2020.9098519.
[30] Robert Robinson et al., “Automatic quality control of cardiac MRI segmentation in large-scale population imaging,” in In: Medical Image Computing and Computer-Assisted Intervention: MICCAI 2017. Cham, Switzerland: Springer International, 2017.
[31] B. Ruijsink et al., “Fully Automated, Quality-Controlled Cardiac Analysis From CMR,” JACC Cardiovasc Imaging, vol. 13, no. 3, pp. 684–695, Mar. 2020, doi: 10.1016/j.jcmg.2019.05.030.
[32] K. Li, L. Yu, and P.-A. Heng, “Towards reliable cardiac image segmentation: Assessing image-level and pixel-level segmentation quality via self-reflective references,” Med Image Anal, vol. 78, p. 102426, May 2022, doi: 10.1016/j.media.2022.102426.
[33] O. Bernard et al., “Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved?,” IEEE Trans Med Imaging, vol. 37, no. 11, pp. 2514–2525, Nov. 2018, doi: 10.1109/TMI.2018.2837502.
[34] A. Suinesiaputra et al., “A collaborative resource to build consensus for automated left ventricular segmentation of cardiac MR images,” Med Image Anal, vol. 18, no. 1, pp. 50–62, Jan. 2014, doi: 10.1016/j.media.2013.09.001.
[35] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. K. Katsaggelos, “Efficient Video Object Segmentation via Network Modulation,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[36] W. Wang, J. Shen, X. Lu, S. C. H. Hoi, and H. Ling, “Paying Attention to Video Object Pattern Understanding,” IEEE Trans Pattern Anal Mach Intell, vol. 43, no. 7, pp. 2413–2428, Jul. 2021, doi: 10.1109/TPAMI.2020.2966453.
[37] J. Cheng, Y. Yuan, Y. Li, J. Wang, and S. Wang, “Learning to Segment Video Object with Accurate Boundaries,” IEEE Trans Multimedia, vol. 23, pp. 3112–3123, 2021, doi: 10.1109/TMM.2020.3020698.
[38] T. Takikawa, D. Acuna, V. Jampani, and S. Fidler, “Gated-SCNN: Gated Shape CNNs for Semantic Segmentation,” Computer Vision and Pattern Recognition, Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.05740
[39] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Perez, “ADVENT: Adversarial entropy minimization for domain adaptation in semantic segmentation,” in in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 2512–2521.
[40] N. Painchaud, Y. Skandarani, T. Judge, O. Bernard, A. Lalande, and P. M. Jodoin, “Cardiac Segmentation With Strong Anatomical Guarantees,” IEEE Trans Med Imaging, vol. 39, no. 11, pp. 3703–3713, Nov. 2020, doi: 10.1109/TMI.2020.3003240.
[41] D. Jha et al., “ResUNet++: An Advanced Architecture for Medical Image Segmentation,” 21st IEEE International Symposium on Multimedia, Nov. 2019.
[42] M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha, and V. K. Asari, “Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation,” Feb. 2018.
[43] Ozan Oktay et al., “Attention U-Net: Learning Where to Look for the Pancreas,” Computer Science - Computer Vision and Pattern Recognition, 2018.
[44] Jieneng Chen et al., “TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation,” Computer Vision and Pattern Recognition, 2021.
[45] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation Networks,” IEEE Trans Pattern Anal Mach Intell, vol. 42, no. 8, pp. 2011–2023, Aug. 2020, doi: 10.1109/TPAMI.2019.2913372.
[46] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation,” IEEE Trans Med Imaging, vol. 39, no. 6, pp. 1856–1867, Dec. 2020.
[47] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Proc. MICCAI. Munich, Germany: Springer, pp. 234–241, May 2015.
[48] H. Huang et al., “UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation,” ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2020.
[49] F. I. Diakogiannis, F. Waldner, P. Caccetta, and C. Wu, “ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 162, pp. 94–114, Apr. 2020, doi: 10.1016/j.isprsjprs.2020.01.013.
[50] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning Spatiotemporal Features with 3D Convolutional Networks,” in 2015 IEEE International Conference on Computer Vision (ICCV), Dec. 2015, pp. 4489–4497. doi: 10.1109/ICCV.2015.510.
[51] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei, Eds., “Eidetic 3D LSTM: A Model for Video Prediction and Beyond,” in International Conference on Learning Representations , 2019.
[52] A. Casella, S. Moccia, D. Paladini, E. Frontoni, E. De Momi, and L. S. Mattos, “A shape-constraint adversarial framework with instance-normalized spatio-temporal features for inter-fetal membrane segmentation,” Med Image Anal, vol. 70, p. 102008, May 2021, doi: 10.1016/j.media.2021.102008.
[53] Y. Xia, N. Ravikumar, J. P. Greenwood, S. Neubauer, S. E. Petersen, and A. F. Frangi, “Super-Resolution of Cardiac MR Cine Imaging using Conditional GANs and Unsupervised Transfer Learning,” Med Image Anal, vol. 71, Jul. 2021, doi: 10.1016/j.media.2021.102037.
Figures
