1024
Visual Pathway Delineation via Correlation-Constrained Feature Decomposition and Consistency-based Sample Selection1Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China, 2University of Chinese Academy of Science, Beijing, China, 3Zhejiang University of Technology, Hangzhou, China, 4Peng Cheng Laboratory, Shenzhen, China
Synopsis
Keywords: Segmentation, Multimodal, Multi-parametric MRI, Deep Learning
Motivation: Accurate segmentation of visual pathway (VP) in multi-parametric MRI is crucial for reliable diagnosis of visual disorders. However, existing methods face challenges due to complex multi-parametric MRI relationships and limited labeled training data.
Goal(s): The goal is to improve automatic VP delineation by developing a new framework that handles complex multi-parametric MRI relationships and incorporates unlabeled data.
Approach: Our framework incorporates a correlation-constrained feature decomposition module to better exploit multi-parametric MRI information and a consistency-based sample selection method for more effective semi-supervised learning.
Results: Experiments on the HCP dataset show that the proposed framework achieved superior VP delineation performance compared to state-of-the-art approaches.
Impact: The results of this study could have a significant impact on scientists, clinicians, and patients by improving the understanding of the human visual system and enhancing the diagnosis accuracy of visual pathway disorders.
Introduction
Accurately delineating the visual pathway (VP) in the human visual system is crucial for understanding and diagnosing visual disorders 1. Manual delineation is time-consuming, so automated methods are desired. Two categories of techniques have emerged: conventional manual-designed methods and deep learning (DL)-based methods.Conventional manual-designed methods, including model-based 2 and atlas-based 3 methods, have achieved encouraging performance. However, they mostly relied on handcraft feature extraction, which is time-consuming.
Recently DL-based methods have been introduced in VP delineation 4,6. These methods have undergone three periods, starting with feature extraction, then pixel/voxel classification, and finally U-Net network-based segmentation approaches. However, most of these approaches focus on single-modal data and have limitations in capturing the complex structure of the VP. Multi-parametric MRI (e.g., combining T1-weighted (T1w) and fractional anisotropy (FA) images) can provide both anatomical and connectivity information, thus improving VP delineation accuracy 6,7. To combine this complementary information, different fusion strategies have been explored 6,7. Nevertheless, they are still limited in effectively capturing the unique characteristics of each imaging sequence. Furthermore, the limited availability of labeled data poses a challenge in VP segmentation. Although abundant semi-supervised learning techniques have been developed for medical image segmentation 8, 9, they may not be applicable for VP segmentation.
To address these challenges, a novel multi-parametric MRI-based VP delineation framework is proposed. Inspired by existing works on feature decomposition and disentanglement 10,11, in our framework, a correlation-constrained feature decomposition (CFD) module is designed to capture each imaging sequence's unique patterns and improve delineation accuracy. Furthermore, a consistency-based sample selection (CSS) method is developed to utilize unlabeled data and enhance the overall delineation performance. The proposed approach is evaluated on the open-source Human Connectome Project (HCP) dataset.
Methodology
The problem of VP delineation was studied using a novel semi-supervised framework. The framework consists of two main components, as shown in Figure 1: a correlation-constrained feature decomposition (CFD) module and a consistency-based sample selection (CSS) method. First, the CFD decomposes features from multi-parametric MR images and selectively retains the unique characteristics of each imaging sequence, such as anatomical structures (tissue contrast, etc.) and information about the integrity and organization of white matter tracts for VP delineation to improve multi-parametric MRI information fusion. Then, it employed a correlation-driven loss to guide the decomposition process. Finally, the CSS method selects high-consistent / reliable unlabeled samples based on a consistency score, enabling the network to learn accurate VP delineation even with limited labeled data.The overall training loss is defined as:
$$ loss = α.l_{sup} + γ.l_{cons} + β.l_{decomp} $$
Where $$$l_{sup}$$$, $$$l_{cons}$$$ and $$$l_{decomp}$$$ are the supervised loss (binary cross-entropy loss + dice loss), the unsupervised consistency loss, and the decomposition loss, respectively. α, γ, and β are hyperparameters.
Results
The proposed framework was compared to two baselines with single-sequence input (T1w and FA), two state-of-the-art fully supervised methods 6,7 for VP delineation trained with limited annotations, and two state-of-the-art semi-supervised methods 8,9.The qualitative and quantitative results obtained from the experiments conducted on the HCP dataset (see Figure 2 and Table 1) demonstrate that the proposed framework surpasses state-of-the-art methods in terms of VP delineation performance. The feature decomposition module effectively separated each modality feature into unique and non-unique characteristics, allowing the network to focus on the most distinctive information for accurate delineation. The consistency-based sample selection method improved the network's accuracy by selecting reliable unlabeled data samples based on consistency scores.
Discussion
The results on the HCP dataset suggest that the proposed semi-supervised multi-parametric MRI-based VP delineation framework mitigates the challenges associated with automatic VP delineation. The incorporation of both T1-weighted and FA images allows for a more comprehensive understanding of the VP's complex structure. The feature decomposition module enhances the network's ability to capture meaningful features, while the semi-supervised framework reduces the reliance on large amounts of annotations.These findings suggest that the proposed approach has the potential to improve clinical practice by providing efficient and reliable automated VP delineation.
Conclusion
In conclusion, this study presents a novel semi-supervised multi-parametric MRI-based VP delineation framework, which effectively addresses the challenges of accurate VP delineation. The proposed approach surpasses existing methods in terms of delineation performance on the HCP dataset. By decomposing image features of different MRI sequences with CFD and incorporating the CSS method to select reliable unlabeled samples, the proposed approach achieves accurate VP delineation while reducing manual annotation efforts.The findings of this research have implications for clinical practice, as automated VP delineation can enhance the understanding of the human visual system and aid in the diagnosis of visual disorders.
Acknowledgements
This research was partly supported by the National Natural Science Foundation of China (62222118, U22A2040), Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application (2022B1212010011), Shenzhen Science and Technology Program (RCYX20210706092104034, JCYJ20220531100213029), and Key Laboratory for Magnetic Resonance and Multimodality Imaging of Guangdong Province (2023B1212060052).References
1. Chan, J.: Optic Nerve Disorders, pp. 130–131 (2007).
2. M. G. Linguraru, “Partitioned Shape Modeling with On-the-Fly Sparse Appearance Learning for Anterior Visual Pathway Segmentation,” in Clinical Image-Based Procedures. Translational Research in Medical Imaging: 4th International Workshop, CLIP 2015, Held in Conjunction with MICCAI 2015, Munich, Germany, October 5, 2015. Revised Selected Papers, vol. 9401. Springer, 2016, p. 104.
3. S. Panda, A. J. Asman, M. P. DeLisi, L. A. Mawn, R. L. Galloway, and B. A. Landman, “Robust optic nerve segmentation on clinically acquired CT,” in Medical Imaging 2014: Image Processing, vol. 9034. SPIE, 2014, pp. 362–371.
4. J. Dolz, H.-A. Leroy, N. Reyns, L. Massoptier, and M. Vermandel, “A fast and fully automated approach to segment optic nerves on MRI and its application to radiosurgery,” in 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI). IEEE, 2015, pp. 1102–1105.
5. A. Mansoor, J. J. Cerrolaza, R. Idrees, E. Biggs, M. A. Alsharid, R. A. Avery, and M. G. Linguraru, “Deep learning guided partitioned shape model for anterior visual pathway segmentation,” IEEE transactions on medical imaging, vol. 35, no. 8, pp. 1856–1865, 2016.
6. S. Li, Z. Chen, W. Guo, Q. Zeng, and Y. Feng, “Two parallel stages deep learning network for anterior visual pathway segmentation,” in Computational Diffusion MRI: International MICCAI Workshop, Lima, Peru, October 2020. Springer, 2021, pp. 279–290.
7. L. Xie, L. Yang, Q. Zeng, J. He, J. Huang, Y. Feng, E. Amelina, and M. Amelin, “Deep Multimodal Fusion Network for the Retinogeniculate Visual Pathway Segmentation,” in The 42nd Chinese Control Conference (CCC 2023), 2023.
8. L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, “Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation,” in Medical Image Computing and Computer Assisted Intervention– MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22. Springer, 2019, pp. 605–613.
9. Y. Bai, D. Chen, Q. Li, W. Shen, and Y. Wang, “Bidirectional Copy-Paste for Semi-Supervised Medical Image Segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11 514–11 524.
10. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool, “Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5906–5916.
11. X. Deng, E. Liu, S. Li, Y. Duan, and M. Xu, “Interpretable Multimodal Image Registration Network Based on Disentangled Convolutional Sparse Coding,” IEEE Transactions on Image Processing, vol. 32, pp. 1078–1091, 2023.
Figures
