Joint self-supervised and supervised contrastive learning for multimodal MRI data: Towards predicting abnormal neurodevelopment
Open accessAbstract
The integration of different imaging modalities, such as structural, diffusion tensor, and functional magnetic resonance imaging, with deep learning models has yielded promising outcomes in discerning phenotypic characteristics and enhancing disease diagnosis. The development of such a technique hinges on the efficient fusion of heterogeneous multimodal features, which initially reside within distinct representation spaces. Naively fusing the multimodal features does not adequately capture the complementary information and could even produce redundancy. In this work, we present a novel joint self-supervised and supervised contrastive learning method to learn the robust latent feature representation from multimodal MRI data, allowing the projection of heterogeneous features into a shared common space, and thereby amalgamating both complementary and analogous information across various modalities and among similar subjects. We performed a comparative analysis between our proposed method and alternative deep multimodal learning approaches. Through extensive experiments on two independent datasets, the results demonstrated that our method is significantly superior to several other deep multimodal learning methods in predicting abnormal neurodevelopment. Our method has the capability to facilitate computer-aided diagnosis within clinical practice, harnessing the power of multimodal data. The source code of the proposed model is publicly accessible on GitHub: https://github.com/leonzyzy/Contrastive-Network.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CLIP and ALIGN use large-scale weakly annotated or natural language supervised learning to learn generalised visual language representations.
“models such as CLIP [181], ALIGN [182], etc., which use large-scale weakly annotated or natural language supervised learning of generalised visual language representations”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [182]
The Introduction describes CLIP-based methods as learning a shared image–text representation through contrastive learning, and reference 24 identifies natural-language supervision. Our copy does not discuss ALIGN or establish the claim about large-scale weakly annotated training. Page numbers are unavailable in the indexed passages. (LLM, conf=0.85)
Experimental — accuracy not yet measuredALIGN trains cross-modal image and text data to enhance matching performance.
“ALIGN [182] follows a similar approach, training cross-modal image and text data to enhance matching performance.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [182]
The Introduction (page unavailable) describes CLIP-based methods that create a shared representation space for images and text, but it does not identify ALIGN or discuss its matching performance. Searches for “ALIGN” and “ALIGN image text matching performance” found no passage about ALIGN; passages about aligning features concern this paper’s MRI method instead. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Computational Radiomics System to Decode the Radiographic Phenotype
(2017) · doi:10.1158/0008-5472.can-17-0339 · Cited by 6,593 · see citing works
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725 · see citing works
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · see citing works
SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1994) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812797926_0003 · Cited by 2,350 · see citing works
Deep Metric Learning Using Triplet Network
Hoffer, E., & Ailon, N. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24261-3_7 · Cited by 2,134 · see citing works
The clinical use of structural MRI in Alzheimer disease
Frisoni, G. B., Fox, N. C., Jack, C. R., Scheltens, P., & Thompson, P. M. (2010) · Nature Reviews Neurology · doi:10.1038/nrneurol.2009.215 · Cited by 2,113 · see citing works
2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(2001) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2001.990916 · Cited by 1,884 · see citing works
Analysis of functional MRI time‐series
Friston, K. J., Jezzard, P., & Turner, R. (1994) · Human Brain Mapping · doi:10.1002/hbm.460010207 · Cited by 1,835 · see citing works
2013 IEEE 10th International Symposium on Biomedical Imaging
(2013) · doi:10.1109/isbi20048.2013 · Cited by 1,637 · see citing works
(2016) · neural information processing systems · Cited by 1,572 · see citing works
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 1× · see citing works
“Multimodal learning aims to construct artificial intelligence (AI) models that can analyze and integrate relevant features extracted from diverse data modalities, with the goal of performing various tasks such as classification and regression [7].”
BackgroundBrainNetCNN: convolutional neural networks for brain networks; towards predicting neurodevelopment
Kawahara, J., Brown, C. J., Miller, S. P., Booth, B. G., Chau, V., Grunau, R. E., Zwicker, J. G., & Hamarneh, G. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.09.046 · Cited by 813 · see citing works
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
Ye, M., Zhang, X., Yuen, P. C., & Chang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00637 · Cited by 629 · see citing works
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis
Dalmaz, O., Yurt, M., & Çukur, T. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167808 · Cited by 613 · see citing works
Makropoulos, A., Robinson, E. C., Schuh, A., Wright, R., Fitzgibbon, S. P., Božek, J., Counsell, S. J., Steinweg, J. K., Vecchiato, K., Passerat‐Palmbach, J., et al. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.01.054 · Cited by 561 · see citing works
Poria, S., Cambria, E., & Gelbukh, A. F. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1303 · Cited by 553 · see citing works
Kidwell, C. S., Alger, J. R., & Saver, J. L. (2003) · Stroke · doi:10.1161/01.str.0000097608.38779.cc · Cited by 474 · see citing works
Contrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · see citing works
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and TextPreprint
Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S., Cui, Y., & Gong, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11178 · Cited by 433 · see citing works
Automatic Whole Brain MRI Segmentation of the Developing Neonatal Brain
Makropoulos, A., Gousias, I. S., Ledig, C., Aljabar, P., Serag, A., Hajnal, J. V., Edwards, A. D., Counsell, S. J., & Rueckert, D. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2322280 · Cited by 381 · see citing works
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
Wang, C., Chai, M., He, M., Chen, D., & Liao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00381 · Cited by 352 · see citing works
Multimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281 · see citing works
5th International Conference on Pattern Recognition
(1979) · IEEE Transactions on Computers · doi:10.1109/tc.1979.1675273 · Cited by 262 · see citing works
Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P. (2020) · Scientific Reports · doi:10.1038/s41598-020-78888-w · Cited by 246 · see citing works
Boulahia, S. Y., Amamra, A., Madi, M. R., & Daikh, S. (2021) · Machine Vision and Applications · doi:10.1007/s00138-021-01249-8 · Cited by 236 · see citing works
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., & Malekshan, K. R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01805 · Cited by 225 · see citing works
Gousias, I. S., Edwards, A. D., Rutherford, M., Counsell, S. J., Hajnal, J. V., Rueckert, D., & Hammers, A. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.05.083 · Cited by 219 · see citing works
A Survey on Canonical Correlation Analysis
Yang, X., Liu, W., Liu, W., & Tao, D. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2958342 · Cited by 202 · see citing works
Ktena, S. I., Parisot, S., Ferrante, E., Rajchl, M., Lee, M., Glocker, B., & Rueckert, D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66182-7_54 · Cited by 196 · see citing works
Deep Multimodal Fusion by Channel ExchangingPreprint
Wang, Y., Huang, W., Sun, F., Xu, T., Rong, Y., & Huang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05005 · Cited by 191 · see citing works
Multimodal Representation Learning for Recommendation in Internet of Things
Huang, Z., Xu, X., Juan, N., Zhu, H., & Wang, C. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2940709 · Cited by 184 · see citing works
Self-Supervised Feature Learning via Exploiting Multi-Modal Data for Retinal Disease Diagnosis
Li, X., Jia, M. J., Islam, M. T., Yu, L., & Xing, L. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3008871 · Cited by 150 · see citing works
Zhang, S., Zhang, J., Tian, B., Lukasiewicz, T., & Xu, Z. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102656 · Cited by 145 · see citing works
Joo, S., Ko, E. S., Kwon, S., Jeon, E., Jung, H., Kim, J.-Y., Chung, M. J., & Im, Y.-H. (2021) · Scientific Reports · doi:10.1038/s41598-021-98408-8 · Cited by 137 · see citing works
Cross-modal attention for multi-modal image registration
Song, X., Chao, H., Xu, X., Guo, H., Xu, S., Türkbey, B., Wood, B. J., Sanford, T., Wang, G., & Yan, P. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102612 · Cited by 133 · see citing works
Multimodal brain and retinal imaging of dopaminergic degeneration in Parkinson disease
Lee, J., Martín‐Bastida, A., Murueta‐Goyena, A., Gabilondo, I., Cuenca, N., Piccini, P. P., & Jeon, B. (2022) · Nature Reviews Neurology · doi:10.1038/s41582-022-00618-9 · Cited by 131 · see citing works
Rotation-Oriented Collaborative Self-Supervised Learning for Retinal Disease Diagnosis
Li, X., Hu, X., Qi, X., Yu, L., Zhao, W., Heng, P.-A., & Xing, L. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3075244 · Cited by 105 · see citing works
Discriminative Multiple Canonical Correlation Analysis for Information Fusion
Gao, L., Qi, L., Chen, E., & Guan, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2765820 · Cited by 101 · see citing works
Co-Attention Fusion Network for Multimodal Skin Cancer Diagnosis
He, X., Wang, Y., Zhao, S., & Chen, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108990 · Cited by 97 · see citing works
Multimodal MRI synthesis using unified generative adversarial networks
Dai, X., Lei, Y., Fu, Y., Curran, W. J., Liu, T., Mao, H. Z., & Yang, X. (2020) · Medical Physics · doi:10.1002/mp.14539 · Cited by 95 · see citing works
Peng, B., Wang, S., Zhou, Z., Liu, Y., Tong, B., Zhang, T., & Dai, Y. (2017) · Neuroscience Letters · doi:10.1016/j.neulet.2017.04.034 · Cited by 88 · see citing works
Multimodal Emotion Recognition Using Deep Canonical Correlation AnalysisPreprint
Liu, W., Qiu, J., Zheng, W., & Lu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05349 · Cited by 83 · see citing works
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Taleb, A., Kirchler, M., Monti, R., & Lippert, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02024 · Cited by 83 · see citing works
Multimodal Triplet Attention Network for Brain Disease Diagnosis
Zhu, Q., Wang, H., Xu, B., Zhang, Z., Shao, W., & Zhang, D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3199032 · Cited by 82 · see citing works
Koonce, B. (2021) · International Linear Collider · Cited by 79 · see citing works
Liang, S., & Gu, Y. (2020) · Sensors · doi:10.3390/s21010220 · Cited by 73 · see citing works
Luo, Y., Alvarez, T. L., Halperin, J. M., & Li, X. (2020) · NeuroImage Clinical · doi:10.1016/j.nicl.2020.102238 · Cited by 72 · see citing works
Deep Multi-Modal Discriminative and Interpretability Network for Alzheimer’s Disease Diagnosis
Zhu, Q., Xu, B., Huang, J., Wang, H., Xu, R., Shao, W., & Zhang, D. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3230750 · Cited by 72 · see citing works
Hu, D., Zhang, H., Wu, Z., Wang, F., Wang, L., Smith, J. K., Lin, W., Li, G., & Shen, D. G. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3013825 · Cited by 68 · see citing works
Deep Multimodal Learning for the Diagnosis of Autism Spectrum Disorder
Tang, M., Kumar, P., Chen, H., & Shrivastava, A. (2020) · Journal of Imaging · doi:10.3390/jimaging6060047 · Cited by 66 · see citing works
Li, M., Tang, D., Zeng, J., Zhou, T., Zhu, H., Chen, B., & Zou, X. (2018) · Computer Speech & Language · doi:10.1016/j.csl.2018.11.002 · Cited by 60 · see citing works
Modality-invariant temporal representation learning for multimodal sentiment classification
Sun, H., Liu, J., Chen, Y., & Lin, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.031 · Cited by 57 · see citing works
SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition
Memmesheimer, R., Theisen, N., & Paulus, D. (2020) · International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413336 · Cited by 52 · see citing works
Multi-modal multiple kernel learning for accurate identification of Tourette syndrome children
Wen, H., Liu, Y., Rekik, I., Wang, S., Chen, Z., Zhang, J., Zhang, Y., Peng, Y., & He, H. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.09.039 · Cited by 50 · see citing works
Joint Representation and Recognition for Ship-Radiated Noise Based on Multimodal Deep Learning
Yuan, F., Ke, X., & Cheng, E. (2019) · Journal of Marine Science and Engineering · doi:10.3390/jmse7110380 · Cited by 49 · see citing works
He, L., Li, H., Chen, M., Wang, J., Altaye, M., Dillman, J. R., & Parikh, N. A. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.753033 · Cited by 48 · see citing works
A multimodal deep learning model for cardiac resynchronisation therapy response prediction
Anton, E. P., Sidhu, B. S., Gould, J., Porter, B., Elliott, M. K., Mehta, V. S., Rinaldi, C. A., & King, A. P. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102465 · Cited by 45 · see citing works
Supervised Contrastive Learning for Multimodal Unreliable News Detection in COVID-19 Pandemic
Zhang, W., Gui, L., & He, Y. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482196 · Cited by 45 · see citing works
Parikh, N. A., Sharma, P. K., He, L., Li, H., Altaye, M., Illapani, V. S. P., Arnsperger, A. L., Beiersdorfer, T., Bridgewater, K., Cahill, T. E., et al. (2020) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2020.11.058 · Cited by 44 · see citing works
VLCDoC: Vision-Language contrastive pre-training model for cross-Modal document classification
Bakkali, S., Ming, Z., Coustaty, M., Rusiñol, M., & Terrades, O. R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109419 · Cited by 44 · see citing works
Classification of sMRI for Alzheimer's disease Diagnosis with CNN
Aderghal, K., Benois‐Pineau, J., & Afdel, K. (2017) · ACM International Conference on Multimedia Retrieval (ICMR) · doi:10.1145/3078971.3079010 · Cited by 43 · see citing works
Multi-Modal Siamese Network for Diagnostically Similar Lesion Retrieval in Prostate MRI
Rossi, A., Hosseinzadeh, M., Bianchini, M., Scarselli, F., & Huisman, H. J. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3043641 · Cited by 39 · see citing works
Jha, A., Bose, S., & Banerjee, B. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00629 · Cited by 38 · see citing works
Multimodal Deep Learning in Early Autism Detection—Recent Advances and Challenges
D’couto, S. S., & Pradeepkandhasamy, J. (2024) · doi:10.3390/engproc2023059205 · Cited by 33 · see citing works
Adaptive Contrast for Image Regression in Computer-Aided Disease Assessment
Dai, W., Li, X., Chiu, W., Kuo, M., & Cheng, K. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3137854 · Cited by 32 · see citing works
Deep Multimodal Complementarity Learning
Wang, D., Zhao, T., Yu, W., Chawla, N. V., & Jiang, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3165180 · Cited by 31 · see citing works
Fischer, M., Hepp, T., Gatidis, S., & Yang, B. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102174 · Cited by 27 · see citing works
D’Souza, N. S., Nebel, M. B., Crocetti, D., Robinson, J., Wymbs, N. F., Mostofsky, S. H., & Venkataraman, A. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118388 · Cited by 18 · see citing works
Multimodal Fusion Using Sparse Cca For Breast Cancer Survival Prediction
Subramanian, V., Syeda-Mahmood, T. F., & Do, M. N. (2021) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi48211.2021.9434033 · Cited by 16 · see citing works
Kline, J. E., Dudley, J., Illapani, V. S. P., Li, H., Kline‐Fath, B. M., Tkach, J. A., He, L., Yuan, W., & Parikh, N. A. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119727 · Cited by 13 · see citing works
Jang, Y. H., Ham, J., Kasani, P. H., Kim, H., Lee, J. Y., Lee, G. Y., Han, T. H., Kim, B.-N., & Lee, H. J. (2024) · Scientific Reports · doi:10.1038/s41598-024-58682-8 · Cited by 13 · see citing works
Multimodal Multitask Deep Learning for X-Ray Image Retrieval
Yu, Y., Hu, P., Lin, J., & Krishnaswamy, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87240-3_58 · Cited by 13 · see citing works
Self-Supervised Multimodal Domino: in Search of Biomarkers for Alzheimer’s Disease
Fedorov, A., Sylvain, T., Geenjaar, E., Luck, M. M., Wu, L., DeRamus, T. P., Kirilin, A., Bleklov, D., Calhoun, V. D., & Plis, S. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00017 · Cited by 12 · see citing works
A novel collaborative self-supervised learning method for radiomic data
Li, Z.-Y., Li, H., Ralescu, A., Dillman, J., Parikh, N., & He, L.-L. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120229 · Cited by 11 · see citing works
Li, H., Wang, J., Li, Z., Cecil, K. M., Altaye, M., Dillman, J. R., Parikh, N. A., & He, L. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120579 · Cited by 10 · see citing works
Prediction of Infant Cognitive Development with Cortical Surface-Based Multimodal Learning
Cheng, J., Zhang, X., Zhao, F., Wu, Z., Yuan, X., Wang, L., Lin, W., & Li, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_58 · Cited by 10 · see citing works
Generalized self-supervised contrastive learning with bregman divergence for image recognition
Li, Z., & Ralescu, A. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.05.020 · Cited by 9 · see citing works
Multi-task Joint Prediction of Infant Cortical Morphological and Cognitive Development
Yuan, X., Cheng, J., Zhao, F., Wu, Z., Wang, L., Lin, W., Zhang, Y., & Li, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43996-4_52 · Cited by 9 · see citing works
Li, Z., Li, H., Braimah, A., Dillman, J. R., Parikh, N. A., & He, L. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119484 · Cited by 7 · see citing works
Kelly, K. J., Hutton, J., Parikh, N. A., & Barnes‐Davis, M. E. (2023) · Frontiers in Pediatrics · doi:10.3389/fped.2023.1083364 · Cited by 5 · see citing works
Learning Generalized Hybrid Proximity Representation for Image Recognition
Li, Z., & Ralescu, A. (2022) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai56018.2022.00138 · Cited by 3 · see citing works
D’Souza, N. S., Nebel, M. B., Crocetti, D., Wymbs, N. F., Robinson, J., Mostofsky, S. H., & Venkataraman, A. (2020) · PubMed · doi:10.48550/arxiv.2008.12410 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [3]Jones DK, Diffusion mri, Oxford University Press, 2010.
- [84]Bayley N, Bayley scales of infant and toddler development–third edition (vol. 2) (2006).
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Such pre-training not only enables better handling of multimodal data but also enhances the model’s overall performance in complex applications [183].”
MCBERT: A multi-modal framework for the diagnosis of autism spectrum disorder
Khan, K., & Katarya, R. (2024) · Biological Psychology · doi:10.1016/j.biopsycho.2024.108976 · Cited by 34 · see citing works
Advances in deep learning for multimodal brain imaging: A comprehensive survey
Balsabti, S. M., Al-Gburi, R. M., gaib, R., Mustafa, A., Ahmed, S. K., Issa, A. M., Al-Naimi, T. M., AlSaad, R., & Elhenidy, A. M. (2025) · Neuroscience Informatics · doi:10.1016/j.neuri.2025.100252 · Cited by 18 · see citing works
Abbas, T., Linjawi, M., Iqbal, I., Alghushairy, O., Alsini, R. A. I., & Daud, A. Y. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108391 · Cited by 3 · see citing works
TRACE-AD: Temporal Representation Learning for Alzheimer's Classification from 3D MRI
Israni, A., & Asadinia, M. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393812
Ali, T., Sheihan, M., Ahmad, F., & Ahmad, M. (2026) · International Conference on Device Intelligence, Computing and Communication Technologies (DICCT) · doi:10.1109/dicct69099.2026.11535989
Chaieb, M., HADHRI, M., & Ameur, N. B. (2026) · Intelligence in Business and Industry (IBI) · doi:10.1109/ibi68858.2026.11604060
Qi, X., Briggs, C., & Marco, J. (2026) · Renewable & Sustainable Energy Reviews · doi:10.1016/j.rser.2026.117395
Objective-Induced Representation Geometry Tradeoffs for Trustworthy Autism Neuro-Imaging
Khan, K., & Wahid, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35579-9_9
Byeon, J., Lee, H., Jang, M., Yoon, S., & Lee, H. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104350
- Authors
- Li, Z., Li, H., Ralescu, A. L., Dillman, J. R., Altaye, M., Cecil, K. M., Parikh, N. A., & He, L.
- Affiliations
- Cincinnati Children's Hospital Medical Center (US)
- University of Cincinnati (US)
- Corresponding author
- He, Lili
- Venue
- Artificial Intelligence in Medicine
- Publisher
- Elsevier BV
- Published
- 2024-11
- Volume
- 157
- Pages
- 102993
- ISSN
- 0933-3657 (ISSN-L)
- PMC
- PMC11560553
- OpenAlex
- W4402980788
- Cited by
- 10
- References
- 93
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 1
- FWCI
- 2.54
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (am, 338 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Artificial Intelligence in Medicine (acceptedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Institutes of Health
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal MRI, contrastive learning, self-supervised learning, multimodal fusion, shared latent representation, diffusion tensor imaging, structural MRI, functional MRI, computer-aided diagnosis
- MeSH terms
- Supervised Machine Learning, Deep Learning, Brain, Diagnosis, Computer-Assisted, Humans, Image Interpretation, Computer-Assisted, Magnetic Resonance Imaging, Multimodal Imaging, Neurodevelopmental Disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
