Cross-modal autoencoder framework learns holistic representations of cardiovascular state
Open accessAbstract
A fundamental challenge in diagnostics is integrating multiple modalities to develop a joint characterization of physiological state. Using the heart as a model system, we develop a cross-modal autoencoder framework for integrating distinct data modalities and constructing a holistic representation of cardiovascular state. In particular, we use our framework to construct such cross-modal representations from cardiac magnetic resonance images (MRIs), containing structural information, and electrocardiograms (ECGs), containing myoelectric information. We leverage the learned cross-modal representation to (1) improve phenotype prediction from a single, accessible phenotype such as ECGs; (2) enable imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECGs; and (3) develop a framework for performing genome-wide association studies in an unsupervised manner. Our results systematically integrate distinct diagnostic modalities into a common representation that better characterizes physiologic state.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Caroline et al. developed a cross-modal analysis method that enables imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECG.
“Caroline et al. developed a cross-modal analysis method that enables the imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECG [154].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [154]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1× · see citing works
“In particular, we used a base architecture of densely connected parallel convolutional blocks 60,61 with 1d convolutional layers for ECGs and 2d convolutional layers for MRIs.”
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · cited 1× · see citing works
“In particular, we used a base architecture of densely connected parallel convolutional blocks 60,61 with 1d convolutional layers for ECGs and 2d convolutional layers for MRIs.”
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249 · cited 1× · see citing works
“We used pandas (version 1.3.3) 76,77 for loading data and matplotlib (version 3.4.2) 78 for generating plots.”
InfluentialSciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · cited 1× · see citing works
“We used numpy (version 1.21.2) 66 , sklearn (version 0.24.2) 72 , scipy (version 1.7.1) 74 , and pytorch (version 1.9.11) with gpu support (cudatoolkit version 11.1.74) 75 for training predictive models.”
InfluentialPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1× · see citing works
“We used numpy (version 1.21.2) 66 , sklearn (version 0.24.2) 72 , scipy (version 1.7.1) 74 , and pytorch (version 1.9.11) with gpu support (cudatoolkit version 11.1.74) 75 for training predictive models.”
InfluentialMomentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1× · see citing works
“A line of recent works utilize autoencoders to learn joint representations of multi-modal data including natural images and captions in computer vision 14,22 – 27 , nuclear images and gene expression in biology 20 , and paired clinical measurements 28 – 31 .”
Comprehensive integration of single-cell dataPreprint
Stuart, T., Butler, A. W., Hoffman, P. J., Hafemeister, C., Papalexi, E., Mauck, W., Hao, Y.-H., Stoeckius, M., Smibert, P., & Satija, R. (2018) · bioRxiv · doi:10.1101/460147 · Cited by 13,858 · cited 2× · see citing works
“Indeed, autoencoders have been observed to perform competitively with other multi-modal integration methods including classical integration approaches using CCA 5 – 7 and generative adversarial networks 32,33 .”
Integrating single-cell transcriptomic data across different conditions, technologies, and species
Butler, A., Hoffman, P., Smibert, P., Papalexi, E., & Satija, R. (2018) · Nature Biotechnology · doi:10.1038/nbt.4096 · Cited by 13,174 · see citing works
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001779 · Cited by 12,728 · cited 16× · see citing works
“To account for population structure and ascertainment biases all GWAS were adjusted for the top 20 principal components of ancestry, the UK Biobank assessment center where the measurements were conducted, the genomic array batch, as well as age and sex of each individual.”
InfluentialData Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 11,511 · cited 1× · see citing works
“We used pandas (version 1.3.3) 76,77 for loading data and matplotlib (version 3.4.2) 78 for generating plots.”
InfluentialA simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
The UK Biobank resource with deep phenotyping and genomic data
Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L. T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O’Connell, J., et al. (2018) · Nature · doi:10.1038/s41586-018-0579-z · Cited by 8,876 · see citing works
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 3× · see citing works
“72, and we applied the following weighting on the loss to account for class imbalances: if there were n total samples of which r had label 1, then we weighted the loss for these samples by nr and the loss for the samples with label 0 by nn (cid:4) r .”
InfluentialDissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1× · see citing works
“In addition, the UK Biobank is known tocontain racialandsocioeconomic biases, whichcanleadto problematic inequities in terms of healthcare 57 .”
Relations Between Two Sets of Variates
Hotelling, H. (1936) · Springer series in statistics · doi:10.1007/978-1-4612-4380-9_14 · Cited by 7,029 · cited 2× · see citing works
“Indeed, autoencoders have been observed to perform competitively with other multi-modal integration methods including classical integration approaches using CCA 5 – 7 and generative adversarial networks 32,33 .”
Yang (2021) · doi:10.1038/s41598-020-79139-8 · Cited by 5,035 · see citing works
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · cited 1× · see citing works
“These models have been successfully applied in a variety of applications including computer vision 12 – 14 , chemistry 15 , and biology 16 – 21 .”
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 2× · see citing works
“These models have been successfully applied in a variety of applications including computer vision 12 – 14 , chemistry 15 , and biology 16 – 21 .”
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · see citing works
Junction Tree Variational Autoencoder for Molecular Graph GenerationPreprint
Jin, W.-G., Barzilay, R., & Jaakkola, T. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.04364 · Cited by 1,783 · cited 1× · see citing works
“These models have been successfully applied in a variety of applications including computer vision 12 – 14 , chemistry 15 , and biology 16 – 21 .”
Scuteri, A., Sanna, S., Chen, W.-M., Uda, M., Albai, G., Strait, J., Najjar, S., Nagaraja, R., Orrú, M., Usala, G., et al. (2007) · PLoS Genetics · doi:10.1371/journal.pgen.0030115 · Cited by 1,724 · cited 1× · see citing works
“4a shows that such GWAS for body mass index (BMI) predicted from cross-modal embeddings identi fi es the gene FTO, which is known to have an effect on BMI and obesity risk 41,42 .”
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01303-4 · Cited by 1,661 · cited 1× · see citing works
“62 are also usable in our framework, but we observed that the optimized models showed improvements in convergence speed, reconstruction, and latent space utility for downstream tasks (see Supplementary Fig.”
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562 · cited 4× · see citing works
“For phenotype prediction from latent space embed-dings, we considered the performance of three models (1) kernel regression with the Neural Tangent Kernel (NTK) 68 ; (2) linear regression ; and (3) logistic regression.”
InfluentialSelf-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · see citing works
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation LearningPreprint
Liang, W., Zhang, Y., Kwon, Y., Yeung, S., & Zou, J. Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2203.02053 · Cited by 908 · cited 1× · see citing works
“A line of recent works utilize autoencoders to learn joint representations of multi-modal data including natural images and captions in computer vision 14,22 – 27 , nuclear images and gene expression in biology 20 , and paired clinical measurements 28 – 31 .”
Leveraging polygenic functional enrichment to improve GWAS power
Kichaev, G., Bhatia, G., Loh, P.-R., Gazal, S., Burch, K. S., Freund, M., Schoech, A., Pasaniuc, B., & Price, A. L. (2017) · bioRxiv · doi:10.1101/222265 · Cited by 875 · cited 1×
“…with ECG or MRI traits, but which have clear associations with the cardiovascular system in general, for example NRP1, previously associated with HDL cholesterol 54 , USP34 previously associated with cardiovascular disease 55 , and NRG1 previously associated with systolic blood pressure 56 .”
Richardson, T. G., Sanderson, E., Palmer, T. M., Ala-Korpela, M., Ference, B. A., Davey Smith, G., & Holmes, M. V. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003062 · Cited by 843 · cited 1× · see citing works
“…with ECG or MRI traits, but which have clear associations with the cardiovascular system in general, for example NRP1, previously associated with HDL cholesterol 54 , USP34 previously associated with cardiovascular disease 55 , and NRG1 previously associated with systolic blood pressure 56 .”
Multi-ethnic genome-wide association study for atrial fibrillation
Roselli, C., Chaffin, M. D., Weng, L.-C. C., Aeschbacher, S., Ahlberg, G., Albert, C. M., Almgren, P., Alonso, Á., Anderson, C. D., Aragam, K. G., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0133-9 · Cited by 754 · cited 1× · see citing works
“…have conducted genome-wide association studies (GWAS) to identify single nucleotide polymorphisms (SNPs) associated with cardio-vascular diseases 34,35 , features measured on ECGs 36,37 , or features measured on cardiac MRI 38,39 , these GWAS approaches have relied on labelled data derived from…”
Self-citationChaitanya, K., Erdil, E., Karani, N., & Konukoglu, E. (2020) · Neural Information Processing Systems · doi:10.3929/ethz-b-000443425 · Cited by 720 · see citing works
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709 · see citing works
Big Self-Supervised Models Advance Medical Image Classification
Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., Chen, T., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00346 · Cited by 685 · see citing works
scGen predicts single-cell perturbation responses
Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-019-0494-8 · Cited by 661 · see citing works
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.07667 · Cited by 655
A Survey of Multi-View Representation Learning
Li, Y., Yang, M., & Zhang, Z. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2872063 · Cited by 637 · see citing works
Bezzina, C. R., Barc, J., Mizusawa, Y., Remme, C. A., Gourraud, J., Simonet, F., Verkerk, A. O., Schwartz, P. J., Crotti, L., Dagradi, F., et al. (2013) · Nature Genetics · doi:10.1038/ng.2712 · Cited by 538 · cited 3× · see citing works
“TheManhattanplotsofsuchunsupervisedGWASidentifyleadSNPsincluding those associated with NOS1AP, TTN, SCN10A, SLC35F1, KCNQ1, which have been previously associated with cardiovascular phenotypes. c The corresponding QQ plots and λ GC factors indicate that there is minimal in fl ation in the unsupervised GWAS of cross-mod…”
UK Biobank's cardiovascular magnetic resonance protocol
Petersen, S. E., Matthews, P. M., Francis, J. M., Robson, M. D., Zemrak, F., Boubertakh, R., Young, A. A., Hudson, S., Weale, P., Garratt, S., et al. (2016) · Journal of Cardiovascular Magnetic Resonance · doi:10.1186/s12968-016-0227-4 · Cited by 481 · cited 3× · see citing works
“All models were optimized with the Adam optimizer 65 To avoid any data leakage the entire test set was made of samples with MRI phenotypes labelled in 39 , ensuring a fair comparison with models trained on downstream prediction tasks.”
Trans-ethnic association study of blood pressure determinants in over 750,000 individuals
Understanding Society Scientific Group, Giri, A., Program, M. V., Hellwege, J. N., Keaton, J. M., Park, J., Qiu, C., Warren, H. R., Torstenson, E. S., Kovesdy, C. P., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0303-9 · Cited by 444 · cited 1× · see citing works
“…with ECG or MRI traits, but which have clear associations with the cardiovascular system in general, for example NRP1, previously associated with HDL cholesterol 54 , USP34 previously associated with cardiovascular disease 55 , and NRG1 previously associated with systolic blood pressure 56 .”
Several common variants modulate heart rate, PR interval and QRS duration
Hólm, H., Gudbjartsson, D., Arnar, D., Thorleifsson, G., Thorgeirsson, G., Stefánsdóttir, H., Gudjonsson, S. A., Jónasdóttir, Á., Mathiesen, E., Njølstad, I., et al. (2010) · Nature Genetics · doi:10.1038/ng.511 · Cited by 441 · cited 1× · see citing works
“We observe that lead SNPs such as NOS1AP 44,45 , TTN 46,47 , SCN10A 48 – 50 , SLC35F1 51 , and KCNQ1 52,53 are consistent with those identi fi ed by prior work.”
Genome-wide association study of PR interval
Pfeufer, A., van Noord, C., Marciante, K. D., Arking, D. E., Larson, M. G., Smith, A. V., Tarasov, K. V., Müller‐Nurasyid, M., Sotoodehnia, N., Sinner, M. F., et al. (2010) · Nature Genetics · doi:10.1038/ng.517 · Cited by 434 · see citing works
Self-citationExploring Single-Cell Data with Deep Multitasking Neural Networks
Amodio, M., van Dijk, D., Srinivasan, K., Chen, W. S., Mohsen, H., Moon, K. R., Campbell, A., Zhao, Y.-J., Wang, X.-M., Venkataswamy, M., et al. (2017) · Nature Methods · doi:10.1038/s41592-019-0576-7 · Cited by 288 · see citing works
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287 · see citing works
Pirruccello, J. P., Bick, A., Wang, M., Chaffin, M., Friedman, S., Yao, J., Guo, X., Venkatesh, B. A., Taylor, K. D., Post, W. S., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15823-7 · Cited by 267 · see citing works
Self-citationMulti-omics integration in the age of million single-cell data
Miao, Z., Humphreys, B. D., McMahon, A. P., & Kim, J. (2021) · Nature Reviews Nephrology · doi:10.1038/s41581-021-00463-x · Cited by 249 · cited 1× · see citing works
“These models have been successfully applied in a variety of applications including computer vision 12 – 14 , chemistry 15 , and biology 16 – 21 .”
Aung, N., Vargas, J., Yang, C.-J., Cabrera, C., Warren, H., Fung, K., Tzanis, E., Barnes, M., Rotter, J., Taylor, K., et al. (2019) · Circulation · doi:10.1161/circulationaha.119.041161 · Cited by 178 · cited 1× · see citing works
“We observe that lead SNPs such as NOS1AP 44,45 , TTN 46,47 , SCN10A 48 – 50 , SLC35F1 51 , and KCNQ1 52,53 are consistent with those identi fi ed by prior work.”
Harnessing the Power of Infinitely Wide Deep Nets on Small-data TasksPreprint
Arora, S., Du, S., Li, Z., Salakhutdinov, R., Wang, R., & Yu, D.-L. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1910.01663 · Cited by 175 · cited 2× · see citing works
“However, we acknowledge that a current limitation of our work is that UK Biobank samples are limited in their diversity with individuals primarily falling between the ages of 40 – 69.”
Arking, D., Junttila, M. J., Goyette, P., Huertas-Vázquez, A., Eijgelsheim, M., Blom, M., Newton-Cheh, C., Reinier, K., Teodorescu, C. A., Uy-Evanado, A., et al. (2011) · PLoS Genetics · doi:10.1371/journal.pgen.1002158 · Cited by 150 · cited 1× · see citing works
“We observe that lead SNPs such as NOS1AP 44,45 , TTN 46,47 , SCN10A 48 – 50 , SLC35F1 51 , and KCNQ1 52,53 are consistent with those identi fi ed by prior work.”
Multi-domain translation between single-cell imaging and sequencing data using autoencoders
Yang, K., Belyaeva, A., Venkatachalapathy, S., Damodaran, K., Katcoff, A., Radhakrishnan, A., Shivashankar, G. V., & Uhler, C. (2021) · Nature Communications · doi:10.1038/s41467-020-20249-2 · Cited by 149 · cited 1× · see citing works
“A line of recent works utilize autoencoders to learn joint representations of multi-modal data including natural images and captions in computer vision 14,22 – 27 , nuclear images and gene expression in biology 20 , and paired clinical measurements 28 – 31 .”
Self-citationRelation-induced multi-modal shared representation learning for Alzheimer’s disease diagnosis
Ning, Z., Xiao, Q., Feng, Q., Chen, W., & Zhang, Y. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3063150 · Cited by 106 · cited 1× · see citing works
“Unlike multi-modal data integration approaches based on classical methods such as canonical correlation analysis (CCA) 4 – 7 or non-negative matrix factorization 8,9 , our approach relies on a class of machine learning models called autoencoders.”
Genetic analysis of right heart structure and function in 40,000 people
Pirruccello, J. P., Achille, P. D., Nauffal, V. D., Nekoui, M., Friedman, S. F., Klarqvist, M. D. R., Chaffin, M. D., Weng, L.-C. C., Cunningham, J. W., Khurshid, S. S., et al. (2022) · Nature Genetics · doi:10.1038/s41588-022-01090-3 · Cited by 105 · cited 2× · see citing works
“43.”
Self-citationIncomplete multi-modal representation learning for Alzheimer’s disease diagnosis
Liu, Y., Fan, L., Zhang, C., Zhou, T., Xiao, Z., Geng, L., & Shen, D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101953 · Cited by 102 · cited 1× · see citing works
“A line of recent works utilize autoencoders to learn joint representations of multi-modal data including natural images and captions in computer vision 14,22 – 27 , nuclear images and gene expression in biology 20 , and paired clinical measurements 28 – 31 .”
Ntalla, I., Weng, L.-C., Cartwright, J. H., Hall, A., Sveinbjornsson, G., Tucker, N., Choi, S.-H.-A., Chaffin, M., Roselli, C., Barnes, M., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15706-x · Cited by 100 · cited 2× · see citing works
“TheManhattanplotsofsuchunsupervisedGWASidentifyleadSNPsincluding those associated with NOS1AP, TTN, SCN10A, SLC35F1, KCNQ1, which have been previously associated with cardiovascular phenotypes. c The corresponding QQ plots and λ GC factors indicate that there is minimal in fl ation in the unsupervised GWAS of cross-mod…”
Self-citationThe Genetic Makeup of the Electrocardiogram
Verweij, N., Benjamins, J., Morley, M., van de Vegte, Y. J., Teumer, A., Trenkwalder, T., Reinhard, W., Cappola, T., & van der Harst, P. (2019) · bioRxiv · doi:10.1101/648527 · Cited by 83
Finite Versus Infinite Neural Networks: an Empirical StudyPreprint
Lee, J., Schoenholz, S. S., Pennington, J., Adlam, B., Xiao, L., Novak, R., & Sohl‐Dickstein, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15801 · Cited by 80 · cited 1× · see citing works
“We considered the NTK since it was shown to have superior performance on supervised learning problems 69,70 .”
Smith, J. G., Lowe, J. K., Kovvali, S., Maller, J., Salit, J., Daly, M. J., Stoffel, M., Altshuler, D., Friedman, J. M., Breslow, J. L., et al. (2009) · Heart Rhythm · doi:10.1016/j.hrthm.2009.02.022 · Cited by 70 · cited 1× · see citing works
“…association studies (GWAS) to identify single nucleotide polymorphisms (SNPs) associated with cardio-vascular diseases 34,35 , features measured on ECGs 36,37 , or features measured on cardiac MRI 38,39 , these GWAS approaches have relied on labelled data derived from individual modalities.”
Diamant, N., Reinertsen, E., Song, S., Aguirre, A., Stultz, C. M., & Batra, P. (2021) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1009862 · Cited by 68 · see citing works
Self-citationDeep learning of left atrial structure and function provides link to atrial fibrillation risk
Pirruccello, J. P., Achille, P. D., Choi, S. H., Rämö, J. T., Khurshid, S. S., Nekoui, M., Jurgens, S. J., Nauffal, V. D., Kany, S., FinnGen, et al. (2024) · Nature Communications · doi:10.1038/s41467-024-48229-w · Cited by 48 · cited 1× · see citing works
“The MRIs were cropped to the smallest bounding box which contained all cardiac tissues in all 50 frames as determined by the semantic segmentation in 58 .”
Self-citationJeff, J. M., Ritchie, M. D., Denny, J., Kho, A. N., Ramirez, A., Crosslin, D., Armstrong, L. L., Basford, M., Wolf, W. A., Pacheco, J., et al. (2013) · Annals of Human Genetics · doi:10.1111/ahg.12023 · Cited by 37 · cited 1× · see citing works
“We observe that lead SNPs such as NOS1AP 44,45 , TTN 46,47 , SCN10A 48 – 50 , SLC35F1 51 , and KCNQ1 52,53 are consistent with those identi fi ed by prior work.”
van Setten, J., Verweij, N., Mbarek, H., Niemeijer, M., Trompet, S., Arking, D., Brody, J., Gandin, I., Grarup, N., Hall, L. M., et al. (2019) · European Journal of Human Genetics · doi:10.1038/s41431-018-0295-z · Cited by 36 · cited 1× · see citing works
“…association studies (GWAS) to identify single nucleotide polymorphisms (SNPs) associated with cardio-vascular diseases 34,35 , features measured on ECGs 36,37 , or features measured on cardiac MRI 38,39 , these GWAS approaches have relied on labelled data derived from individual modalities.”
Deep Multi-modal Latent Representation Learning for Automated Dementia Diagnosis
Zhou, T., Liu, M., Fu, H., Wang, J., Shen, J., Shao, L., & Shen, D. G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_69 · Cited by 36 · cited 1× · see citing works
“Unlike multi-modal data integration approaches based on classical methods such as canonical correlation analysis (CCA) 4 – 7 or non-negative matrix factorization 8,9 , our approach relies on a class of machine learning models called autoencoders.”
Decision-Making with Auto-Encoding Variational BayesPreprint
Lopez, R., Boyeau, P., Yosef, N., Jordan, M. I., & Regier, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07217 · Cited by 29 · cited 1× · see citing works
“These models have been successfully applied in a variety of applications including computer vision 12 – 14 , chemistry 15 , and biology 16 – 21 .”
Méndez-Giráldez, R., Gogarten, S. M., Below, J. E., Yao, J., Seyerle, A. A., Highland, H. M., Kooperberg, C. L., Soliman, E. Z., Rotter, J. I., Kerr, K. F., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-17136-0 · Cited by 28 · cited 3× · see citing works
“We fi nd several additional clusters corresponding to SNPs affecting multiple cardiac traits such as those associated with BAG3, SLC35F1, or KCND3.”
Wang, M.-Q., Gao, J.-Q., Shi, Y., & Zhao, X. (2021) · Scientific Reports · doi:10.1038/s41598-021-84135-7 · Cited by 2 · cited 3× · see citing works
“TheManhattanplotsofsuchunsupervisedGWASidentifyleadSNPsincluding those associated with NOS1AP, TTN, SCN10A, SLC35F1, KCNQ1, which have been previously associated with cardiovascular phenotypes. c The corresponding QQ plots and λ GC factors indicate that there is minimal in fl ation in the unsupervised GWAS of cross-mod…”
Srivastava (2014) · doi:10.5555/2627435.2670313 · cited 1×
“To ensure that only one modality is needed at test time, we additionally utilized dropout 63 to merge modality-speci fi c embed-dings.”
Flagged first, then by citations
18 of the 82 references couldn’t be matched to a known paper.
Research works printed but not linked: 15
- [5]Andrew, G., Arora, R., Bilmes, J. & Livescu, K. Deep canonical correlation analysis. In International Conference on Machine Learning (eds Dasgupta, S. & McAllester, D.) (Proceedings of Machine Learning Research, 2013).
- [10]Salakhutdinov R Hinton G Reducing the dimensionality of data with neural networks Science 2006 313 504 507 16873662
- [11]Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning, Vol. 1 (MIT Press, 2016).
- [12]Kinga, D. & Welling, M. Auto-encoding variational Bayes. arXiv:1312.6114 (2014). · arXiv
- [19]Klimovskaia, A. et al. Compositional perturbation autoencoder for single-cell response modeling. Preprint at bioRxiv10.1101/2021.04.14.439903 (2021).
- [32]Zhu, J. Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision (IEEE, 2017).
- [33]Amodio, M. & Krishnaswamy, S. MAGAN: aligning biological manifolds. In International Conference on Machine Learning (eds Dy, J. & Krause, A.) (Proceedings of Machine Learning Research, 2018).
- [34]Shah, S. et al. Genome-wide association and mendelian randomisation analysis provide insights into the pathogenesis of heart failure. Nat. Commun.11, 1 (2020). PMC6952380 31919418
- [41]Frayling T A common variant in the FTO gene is associated with body mass index and predisposes to childhood and adult obesity Science 2007 316 889 94 17434869 PMC2646098
- [44]Newton-Cheh C Genome-wide association study of electrocardiographic and heart rate variability traits: the Framingham heart study BMC Med. Genet. 2007 8 1 8 17903306 PMC1995612
- [64]Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 8748–8763 (PMLR, 2021).
- [65]Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In International Conference on Learning Representations (2015).
- [71]Ma, S. & Belkin, M. Kernel machines that adapt to GPUs for effective large batch training. In Conference on Machine Learning and Systems (eds Talwalkar, A. et al.) (2019).
- [73]Khurshid S Deep learning to predict cardiac magnetic resonance-derived left ventricular mass and hypertrophy from 12-lead ECGs Circulation 2021 14 e012281 34126762 PMC8217289
- [80]Seabold, S. & Perktold, J. statsmodels: econometric and statistical modeling with python. In 9th Python in Science Conference (2010).
Cited by
Sorted by citations
Cardiovascular care with digital twin technology in the era of generative artificial intelligence
Thangaraj, P., Benson, S., Oikonomou, E. K., Asselbergs, F. W., & Khera, R. (2024) · European Heart Journal · doi:10.1093/eurheartj/ehae619 · Cited by 185 · see citing works
“96 Moreover, when MRI and ECG data are available, information on genomic variants can also be inferred.”
Svennberg, E., Han, J. K., Caiani, E. G., Engelhardt, S., Ernst, S., Friedman, P. A., Garcia, R., Ghanbari, H., Hindricks, G., Man, S. H., et al. (2025) · EP Europace · doi:10.1093/europace/euaf071 · Cited by 92 · see citing works
“127 An example is a multimodal AI approach (the DEEP RISK model), which integrated DL features from both ECG and LGE-CMR in patients with non-ischemic cardiomyopathy, along with clinical patient data, to predict the 1-year risk of VA (AUROC 0.84).”
InfluentialArtificial Intelligence in Cardiovascular Care—Part 2: Applications
Jain, S. S., Elias, P., Poterucha, T., Randazzo, M., Lopez Jimenez, F., Khera, R., Perez, M., Ouyang, D., Pirruccello, J., Salerno, M., et al. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.03.401 · Cited by 74 · see citing works
A roadmap for the development of human body digital twins
Tang, C., Yi, W., Occhipinti, E., Dai, Y., Gao, S., & Occhipinti, L. G. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-024-00025-w · Cited by 90 · see citing works
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75 · see citing works
“Caroline et al. developed a cross-modal analysis method that enables the imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECG [154].”
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Lan, X., Wu, F., He, K., Zhao, Q.-H., Hong, S.-D., & Feng, M.-L. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2503.06073 · Cited by 55
Kolk, M. Z. H., Ruipérez-Campillo, S., Wilde, A. A. M., Knops, R. E., Narayan, S. M., & Tjong, F. V. Y. (2024) · Heart Rhythm · doi:10.1016/j.hrthm.2024.09.003 · Cited by 53 · see citing works
“74 This cross-modal learning approach not only enhanced the representations obtained from a single modality but also enabled the imputation of challenging-to-acquire modalities, such as MRI, from more readily accessible ECG.”
TMO-Net: an explainable pretrained multi-omics model for multi-task learning in oncology
Wang, F.-A., Zhuang, Z., Gao, F., He, R., Zhang, S., Wang, L., Liu, J., & Li, Y. (2024) · Genome Biology · doi:10.1186/s13059-024-03293-9 · Cited by 51 · see citing works
“Discriminators assist in improving the generative capacity of TMO-Net during the pre-training stage [62].”
BackgroundMuse, E. D., & Topol, E. J. (2024) · Cell Metabolism · doi:10.1016/j.cmet.2024.02.002 · Cited by 50 · see citing works
Integrative Multi-Omics and Artificial Intelligence: A New Paradigm for Systems Biology
Kant, S., Deepika, & Roy, S. (2025) · OMICS A Journal of Integrative Biology · doi:10.1177/15578100251392371 · Cited by 61 · see citing works
“Cross-modality translation networks also facilitate prediction of a second omics layer from a fi rst, for example, metabolomic pro fi les from transcriptomic data (Cao et al., 2024; Radhakrishnan et al., 2023).”
Kolk, M. Z. H., Ruipérez-Campillo, S., Allaart, C. P., Wilde, A. A. M., Knops, R. E., Narayan, S. M., Tjong, F. V. Y., DEEP RISK investigators, Raijmakers, F. D., Van Der Lingen, A.-L. C. J., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-65357-x · Cited by 49 · see citing works
“As such, a promising opportunity for further exploration is to learn a shared latent space across different domains, enabling more holistic latent representations 46,47 .”
BackgroundYun, T., Cosentino, J., Behsaz, B., McCaw, Z., Hill, D., Luben, R., Lai, D.-B., Bates, J., Yang, H., Schwantes-An, T.-H., et al. (2024) · Nature Genetics · doi:10.1038/s41588-024-01831-6 · Cited by 48 · see citing works
“Previous work has explored applying transfer learning 49 and contrastive learning 50 to retinal fundus images, or multimodal autoencoders to cardiac data modalities 51 .”
Deep learning-derived cardiovascular age shares a genetic basis with other cardiac phenotypes
Libiseller-Egger, J., Phelan, J. E., Attia, Z. I., Benavente, E. D., Campino, S. G., Friedman, P. A., López-Jiménez, F. A., Leon, D. A., & Clark, T. G. (2022) · Scientific Reports · doi:10.1038/s41598-022-27254-z · Cited by 35 · see citing works
“…studies investigated the genetic basis of ECG-features (e.g. for the PR interval 22 or the QRS complex 23 ), while others sought to determine the impact of genetic variants on the shape of the ECG traces in general 24 or on a more holistic representation of the cardiac state including the ECG 25 .”
BackgroundMachine learning integrative approaches to advance computational immunology
Curion, F., & Theis, F. J. (2024) · Genome Medicine · doi:10.1186/s13073-024-01350-3 · Cited by 35 · see citing works
“Integration of these with uncoupled single-cell readouts is possible [102, 164].”
BackgroundHoward, F. M., Hieromnimon, H. M., Ramesh, S., Dolezal, J. M., Kochanny, S. E., Zhang, Q., Feiger, B., Peterson, J., Fan, C., Perou, C. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adq0856 · Cited by 33 · see citing works
Turgut, Ö., Müller, P., Hager, P., Shit, S., Starck, S., Menten, M. J., Martens, E., & Rueckert, D. (2025) · Medical Image Analysis · doi:10.1016/j.media.2024.103451 · Cited by 22 · see citing works
Oikonomou, E. K., & Khera, R. (2024) · Hellenic Journal of Cardiology · doi:10.1016/j.hjc.2024.07.003 · Cited by 26 · see citing works
Zhang, Y., Hager, P., Liu, C., Shit, S., Chen, C., Rueckert, D., & Pan, J. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103756 · Cited by 7 · see citing works
“…for cardiac segmentation (Petitjean and Dacher, 2011; Bai et al., 2017; Chen et al., 2020b; Campello et al., 2021; Zhang et al., 2024b), which provide a description of the cardiac anatomy and derive phenotypic indicators (Bai et al., 2020), these models often rely on sparse views of the heart.”
Novel multi-omics deconfounding variational autoencoders can obtain meaningful disease subtyping
Li, Z., Katz, S., Saccenti, E., Fardo, D., Claes, P., dos Santos, V. M. D. M., van Steen, K., & Roshchupkin, G. V. (2024) · bioRxiv · doi:10.1093/bib/bbae512 · Cited by 20 · see citing works
“(1 ) Based on multi-omics pan-cancer TCGA data (section 2.1) different confounding effects were simulated (section 2.2).”
BackgroundInfluentialArtificial Intelligence in Diagnosis of Heart Failure
Xie, Y., Zhang, L., Sun, W., Zhu, Y. C., Zhang, Z., Chen, L., Xie, M., & Zhang, L. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.124.039511 · Cited by 27 · see citing works
Multimodal Wearable Biosensing Meets Multidomain AI: A Pathway to Decentralized Healthcare
Liu, C., Fan, H., Kim, M., Zhou, T., Yang, P., Zhao, L., Wang, Y., Che, Z., Liu, C., Li, B., et al. (2026) · Advanced Science · doi:10.1002/advs.202522900 · Cited by 20 · see citing works
Haq, I., Gong, Z., Liang, H., Zhang, W., Khan, R., Gu, L., Eils, R., Kang, Y., & Huang, B. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105708 · Cited by 16 · see citing works
Deep Learning of radiology-genomics integration for computational oncology: A mini review
Wang, F., Li, Y., & Zeng, T. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.06.019 · Cited by 21 · see citing works
“In intermediate fusion, multimodal features are combined at an intermediate stage of the model architecture, typically after the initial feature extraction but before the final decision level, such as concatenation, ele-mentwise operations or more sophisticated attention mechanisms [46 – 48].”
BackgroundSingle-cell multiomics data integration and generation with scPairing
Niu, J., Vasquez-Rios, C., & Ding, J. (2025) · Cell Reports Methods · doi:10.1016/j.crmeth.2025.101211 · Cited by 11 · see citing works
“Since f RNA and f ATAC both produce a d -dimensional direction, we merge the directions from both encoders using a method derived from dropout 78,79 to produce the μ for the Power Spherical distribution.”
A Representation Fusion Framework for Decoupling Diagnostic Information in Multimodal Learning
Tonekaboni, S., Friedman, S., Zhang, X.-Y., Maddah, M., & Uhler, C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02144-6 · Cited by 12 · see citing works
“In this study, we employ previously introduced ECG and MRI encoders 3 , although the framework is compatible with alternative pretrained models.”
Self-citationEnhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning
Girlanda, F., Demler, O., Menze, B. H., & Davoudi, N. (2024) · British Machine Vision Conference · doi:10.48550/arxiv.2411.05900 · Cited by 12
“Results of generative models such as Autoencoders (AEs) [2] on multi-modal clinical measurements show that they perform well on aligning the embeddings from diverse modalities and constructing a holistic representation for characterizing physiological state [33].”
Partially shared multi-modal embedding learns holistic representation of cell state
Zhang, X.-Y., Shivashankar, G. V., & Uhler, C. (2024) · bioRxiv · doi:10.1101/2024.10.01.615977 · Cited by 12 · see citing works
Self-citationWhole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images
Zhang, Y.-D., Chen, C., Shit, S., Starck, S., Rueckert, D., & Pan, J.-Z. (2024) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.48550/arxiv.2406.00329 · Cited by 12
“In the past decade, a large multitude of CMR representation learning meth-ods have been proposed for cardiac function analysis [31, 2, 30, 23, 24].”
BackgroundDing, Z.-Y., Hu, Y.-J., Li, Z.-Y., Zhang, H.-K., Wu, F., Xiang, Y.-L., Li, T., Liu, Z.-Y., Chu, X.-S., & Huang, Z.-X. (2024) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.1007/978-3-031-72384-1_11 · Cited by 12 · see citing works
“Furthermore, the innovative proposal by Radhakrishnan et al [21]. for a cross-modal autoencoder that bridges ECG and CMR technologies aims to furnish a thorough cardiovascular health assessment utilizing solely ECG data.”
BackgroundMethodZhou, Y., Khasentino, J., Yun, T., Biradar, M. I., Shreibati, J. B., Lai, D., Schwantes-An, T.-H. L., Luben, R., McCaw, Z. R., Engmann, J., et al. (2025) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2025.05.015 · Cited by 16 · see citing works
“26,27 However, there are multiple ways to model multimodal data using unsupervised learning, including different fusion strategies, and prior work has not systematically explored or compared their effectiveness.”
Wang, M., Li, Z., Tian, Y., Wei, X., Jin, Y., & Liu, C. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103302 · Cited by 7 · see citing works
Sergouniotis, P. I., Diakite, A., Gaurav, K., Allen, N., Aslam, T. M., Atan, D., Barman, S. A., Barrett, J., Bishop, P., Black, G. C. M., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae732 · Cited by 18 · see citing works
“Deep learning approaches have been shown to be able to detect imaging patterns that are not amenable to human identification and which can assist with prediction tasks (Patel et al. 2024, Radhakrishnan et al. 2023, Zhou et al. 2024, Yun et al. 2024).”
The Use of Artificial Intelligence to Predict the Development of Atrial Fibrillation
Pipilas, D. C., Friedman, S. F., & Khurshid, S. S. (2023) · Current Cardiology Reports · doi:10.1007/s11886-023-01859-w · Cited by 14 · see citing works
“…focused largely on single modalities (e.g., ECG), future models may leverage a combination of clinical factors with multiple raw diagnostic modalities (e.g., ECG, photoplethys-mography, echocardiography, cardiac magnetic resonance imaging) to deliver even more accurate AF risk predictions [46].”
BackgroundSelf-citationTechnical survey of end-to-end signal processing in BCIs using invasive MEAs
Erbslöh, A., Buron, L., Ur-Rehman, Z., Musall, S., Hrycak, C., Löhler, P., Klaes, C., Seidl, K., & Schiele, G. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad8031 · Cited by 10 · see citing works
“Autoencoders showed success as FE in different application cases [77, 95].”
Song, J., Jang, J.-H., Hong, D.-G., Kwon, J.-M., & Jo, Y. (2024) · doi:10.48550/arxiv.2407.07110 · Cited by 10
“These results suggest that either only contrastive learning or only generative learning may hinder the robustness of the ECG representation [23, 25].”
InfluentialHoward, F. M., Hieromnimon, H. M., Ramesh, S., Dolezal, J., Kochanny, S., Zhang, Q., Feiger, B., Peterson, J., Fan, C., Perou, C., et al. (2024) · bioRxiv · doi:10.1101/2024.03.22.586306 · Cited by 10
“Additionally, reconstruction of histology from base features enables development of accurate cross-modal autoencoders to reconstruct histology from other forms of data 15,16 , such as sequencing or MRI imaging, enabling a ‘virtual biopsy’.”
BackgroundMethodLiu, K.-C., Peng, S.-Y., Tsao, Y., Liu, C.-Y., Chen, Z.-A., Han, Z., Chen, W.-C., Hsieh, P.-Q., Li, Y.-J., Hsu, Y.-J., et al. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3486154 · Cited by 8 · see citing works
“A better representation of cross-modal embedding can improve different downstream tasks, such as classification and prediction [29], and the decoders have the potential to mitigate discrepancies between the source and target modalities [30].”
OTIS: Learning High-Quality Time Series Features With Tiny Encoders
Turgut, Ö., Müller, P., Menten, M., & Rueckert, D. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2410.07299 · Cited by 8
“To this end, early approaches leveraged contrastive learning , aligning augmented views of a sample in the feature space (Kiyasseh et al., 2021; Zhang et al., 2022; Woo et al., 2022; Wang et al., 2022; Radhakrishnan et al., 2023; Demirel & Holz, 2023).”
Data Augmentation with Cross-Modal Variational Autoencoders (DACMVA) for Cancer Survival Prediction
Rajaram, S., & Mitchell, C. S. (2023) · Inf. · doi:10.3390/info15010007 · Cited by 8 · see citing works
“Specifically, the inclusion of a contrastive loss, as suggested by Radhakrishnan et al. [24], could improve latent space alignment.”
BackgroundFang, X.-C., Ding, Z.-Y., Cai, J., Xiao, Y.-J., Liu, B., Jin, J., Wang, H.-Y., Nie, G., Huang, S., Chen, T., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.20904 · Cited by 8
Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG
Selivanov, A., Müller, P., Turgut, Ö., Stolt-Ansó, N., & Rueckert, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04927-8_21 · Cited by 4 · see citing works
“To integrate information from both modalities, researchers have applied contrastive learning to align ECG and CMR embeddings globally [5,18,25], pulling same-patient representations closer while pushing apart different patients.”
Advancing Stroke Risk Prediction Using a Multi-modal Foundation Model
Delgrange, C., Demler, O., Mora, S., Menze, B. H., de la Rosa, E., & Davoudi, N. (2024) · arXiv.org · doi:10.48550/arxiv.2411.09822 · Cited by 5
“While there was an extensive work done for cardiovascular diseases prediction [27, 28, 29, 30], stroke risk prediction through volumetric brain images and clinical health records remains underexplored.”
Zhou, S., Liu, Y., Turnbull, A., Tapparello, C., Adeli, E., & Lin, F. (2025) · Ageing Research Reviews · doi:10.1016/j.arr.2025.102877 · Cited by 5 · see citing works
Integration of multi-modal measurements identifies critical mechanisms of tuberculosis drug action
Johnson, W. C., Alivisatos, A. P., Smith, T. C., Van, N., Soni, V., Wallach, J. B., Clark, N. A., Fitzgerald, T. A., Whiteley, J. J., Tan, S., et al. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101348 · Cited by 5 · see citing works
Generating Cardiac Magnetic Resonance Images from Electrocardiograms — A Multicenter Study
Ding, Z., Li, Z., Hu, Y., Xu, Y., Zhao, C., Mao, Y., Li, H., Li, Z., Li, Q., Wang, J., et al. (2026) · NEJM AI · doi:10.1056/aioa2500549 · Cited by 6 · see citing works
“We further compared CardioNets with two recent state-of-the-art methods, Cross-AE 24 and MMCL 41 , and observed that CardioNets significantly outperformed both in predicting cardiac phenotypes (Supplementary Fig.”
InfluentialLatent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration
Kwon, B. C., Friedman, S., Xu, K., Lubitz, S., Philippakis, A. A., Batra, P., Ellinor, P., & Ng, K. (2023) · arXiv.org · doi:10.48550/arxiv.2312.00857 · Cited by 5
“A line of recent works uses autoencoders to learn joint representations of multi-modal data including natural images and captions [12,16,23,24,26,32], gene expression in biology [35], and paired clinical measurements [6,11,14,25].”
BackgroundMethodSelf-citationYun, T., Cosentino, J., Behsaz, B., McCaw, Z., Hill, D., Luben, R., Lai, D.-B., Bates, J., Yang, H., Schwantes-An, T.-H., et al. (2023) · medRxiv · doi:10.1101/2023.04.28.23289285 · Cited by 5
“Prior work has explored applying transfer learning [44] and contrastive learning [45] to retinal fundus images, or multimodal autoencoders to cardiac data modalities [46].”
Dabija, M., Tătaru, C., Dumitru, A., Munteanu, O., Radoi, M., Ciurea, A., & Petrescu, I. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms262311444 · Cited by 4 · see citing works
“…development of cross-species computational models that incorporate multimodal imaging (PET, fMRI, elastography) and longitudinal biomarkers (plasma NE metabolites, soluble AQP4 fragments) to determine if the quantitative relationships demonstrated in animal models are preserved in humans [108,109].”
MetaWearS: A Shortcut in Wearable Systems Lifecycle with Only a Few Shots
Amirshahi, A., Toosi, M. H., Mohammadi, S., Albini, S., Schiavone, P. D., Ansaloni, G., Aminifar, A., & Atienza, D. (2024) · arXiv.org · doi:10.48550/arxiv.2408.01988 · Cited by 4
Trajectories of sentiment in 11,816 psychoactive narratives
Friedman, S. F., & Ballentine, G. (2023) · Human Psychopharmacology Clinical and Experimental · doi:10.1002/hup.2889 · Cited by 5 · see citing works
“…tag is consistently elevated for the stimulants cocaine and methamphetamine, see Supplementary Figure 3. predictions through iterative removal of confounded subspaces (Radhakrishnan et al., 2023; Ravfogel et al., 2020), allowing equitable application of these models in healthcare settings.”
BackgroundSelf-citationThe cardiac rhythm invariance Principle: A unified framework for multimodal cardiac signal analysis
Abdessalem, K. B. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.109148 · Cited by 4 · see citing works
Genetic analysis of imaging-derived phenotypes
Bian, Y., & Akey, J. M. (2026) · Nature Reviews Genetics · doi:10.1038/s41576-026-00989-5 · Cited by 2 · see citing works
Ometto, S., Chatterjee, S., Vergani, A. M., Landini, A., Sharapov, S., Giacopuzzi, E., Visconti, A., Bianchi, E., Santonastaso, F., Soda, E., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-74575-y · Cited by 2 · see citing works
““ MRI latents" refers to the unsupervisedGWASof cross-modal representations from the study by Radhakrishnan et al. 19 Previously unreported loci are labelled as novel.”
InfluentialHaq, I., Liang, H., Zeng, K., Wang, T., Uddin, I., Lin, J., Kang, Y., & Huang, B. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109899 · Cited by 3 · see citing works
Fang, X., Ding, Z., Nie, G., Cai, J., Xiao, Y., Liu, B., Jin, J., Wang, H., Huang, S., Chen, T., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818910
AEM: An interpretable multi-task multi-modal framework for cardiac disease prediction
Peng, J., Beetz, M., Banerjee, A., Chen, M., & Grau, V. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.103951 · Cited by 2 · see citing works
Yue, A.-X., Lu, Z., Hu, Y., Cao, Z., & Chen, Y. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3770693 · Cited by 3 · see citing works
Causal Structure and Representation Learning with Biomedical Applications
Uhler, C., & Zhang, J.-Q. (2025) · arXiv.org · doi:10.48550/arxiv.2511.04790 · Cited by 3
“We developed various practical approaches based on autoencoders, where modality-specific encoders and decoders are used to map between observed data from each modality O m , m ∈ [ M ] , and the shared latent space X L [73, 74, 48, 80].”
Self-citationYeung, M. W., van de Leur, R. R., Benjamins, J. W., Vessies, M. B., Ruijsink, B., Anton, E. P., van Tintelen, J. P., Verweij, N., van Es, R., & van der Harst, P. (2025) · iScience · doi:10.1016/j.isci.2025.113226 · Cited by 4 · see citing works
“29 To integrate biological information of the heart from both the 12-lead ECG and MRI, they developed cross-modal 256-dimensional representations with VAE models by enforcing a contrastive loss on the 2 models for 2 modalities.”
Lutsker, G., Rossman, H., Godiva, N., & Segal, E. (2024) · medRxiv · doi:10.1101/2024.03.17.24304415 · Cited by 3 · see citing works
“SSL with multi-modal data are particularly potent, with each modality enhancing the model’s capabilities and utility across various health signals [Radhakrishnan et al., 2023].”
BackgroundGenetic Architectures of Medical Images Revealed by Registration of Multiple Modalities
Friedman, S., Moran, G. E., Rakic, M., & Phillipakis, A. (2024) · Bioinformatics and Biology Insights · doi:10.1177/11779322241282489 · Cited by 3 · see citing works
“For example, two modalities registered in space by VoxelMorph can later be cross-modally fused with DropFuse.”
BackgroundInfluentialSelf-citationGenetic Architectures of Medical Images Revealed by Registration and Fusion of Multiple ModalitiesPreprint
Friedman, S., Moran, G. E., Rakic, M., & Phillipakis, A. (2023) · bioRxiv · doi:10.1101/2023.07.27.550885 · Cited by 3 · see citing works
“Contrastive fusion can be thought of as a learned registration between distinct modalities and, like simpler registration techniques, it too reveals more genetic and phenotypic associations(Radhakrishnan et al. 2023).”
BackgroundMethodSelf-citationWang, F., Liu, J., Gao, F., & Li, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.05.547886 · Cited by 3 · see citing works
Language Models Learn Sentiment and Substance from 11,000 Psychoactive ExperiencesPreprint
Friedman, S. F., & Ballentine, G. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.06.02.494544 · Cited by 1 · see citing works
“Accurate detection of sensitive features, such as these, is a critical first step towards de-biasing predictions through iterative removal of confounded subspaces [28,29].”
BackgroundSelf-citationTonekaboni, S., Schuster, V., & Uhler, C. (2026) · arXiv.org · doi:10.48550/arxiv.2607.16789 · Cited by 2 · see citing works
“…modeling (Chen et al., 2024; Sun et al., 2025), medical diagnostics (Steyaert et al., 2023; Zhou et al., 2023; Tonekaboni et al., 2025; Radhakrishnan et al., 2023), and biology (Cui et al., 2025; Uhler & Shivashankar, 2022; Yang 1 Massachusetts Institute of Technology, Cambridge, MA,…”
Self-citationYang, W., Li, H., Wang, Y.-K., & Wang, Y.-Q. (2026) · Expert systems with applications · doi:10.1016/j.eswa.2025.131083 · Cited by 2 · see citing works
Li, F., Li, S., Qian, Y., Chen, B., Brody, J., Yogeswaran, V., Wiggins, K., Sitlani, C., Bis, J., Shojaie, A., et al. (2026) · medRxiv · doi:10.64898/2026.03.16.26348526 · Cited by 2 · see citing works
Alvarez-Florez, L., Bujalance-Gomez, A., Raijmakers, F. D., Ruipérez-Campillo, S., Kolk, M. Z. H., Wiers, J., Vogt, J. E., Bekkers, E. J., Išgum, I., & Tjong, F. V. Y. (2026) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.3086069 · Cited by 1 · see citing works
Deep learning to predict left ventricular hypertrophy from the electrocardiogram
Naderi, H., Kaplan, T., van Duijvenboden, S., Pujadas, E., Aung, N., Chahal, C., Lekadir, K., Chamling, B., Dörr, M., Markus, M. R. P., et al. (2026) · Europace · doi:10.1093/europace/euag015 · Cited by 2 · see citing works
Yu, Q., Azariah, J. L., Ahmad, Z. S., Anilkumar, R., Calvert, P. A. R., Chen, Y., Zheng, Y., Meng, Y., Bahuleyan, C. G., & Lip, G. Y. H. (2026) · The Lancet Regional Health - Southeast Asia · doi:10.1016/j.lansea.2025.100715 · Cited by 2 · see citing works
Asuai, C., Andrew, M., Arinomor, A., Ogheneochuko, D., Joseph-Brown, A. A., Merit, I., & Collins, A. (2025) · Journal of Computing Theories and Applications · doi:10.62411/jcta.14661 · Cited by 2 · see citing works
“Moreover, developing architectures that incorporate domain knowledge, remain computationally efficient, and provide transparent decision-making through explainable AI (XAI) mechanisms such as attention maps will be crucial for improving reliability and user trust [17], [18].”
Sergouniotis, P., Diakité, A., Gaurav, K., Birney, E., & Fitzgerald, T. (2023) · medRxiv · doi:10.1101/2023.06.15.23291410 · Cited by 2
Physiological World Models for Human State Transitions
Zhang, C.-Y., Wang, R., Zheng, H., Zhang, H., Liu, Y., Wei, X., & Chong, B. (2026) · arXiv.org · doi:10.48550/arxiv.2608.15309 · Cited by 1
Biosignal Fingerprinting: A Cross-Modal PPG-ECG Foundation ModelPreprint
Liu, Z., Liu, C., Liu, F., Chen, Y., Yang, Y., Clifton, D., & Gu, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.09579
Cardiovascular-kidney-metabolic syndrome: candidate subtypes and genetic risk factors
Donker, H. C., Bisht, V., Dwivedi, O. P., Mueller, S., Neijzen, D., Ding, Z., & Lunter, G. A. (2026) · BMC Medical Genomics · doi:10.1186/s12920-026-02315-8 · Cited by 3 · see citing works
Sampaio, I. W., Tassi, E., Bellani, M., Benedetti, F., Nenadić, I., Phillips, M. L., Piras, F., Yatham, L., Bianchi, A. M., Brambilla, P., et al. (2025) · Artif. Intell. Medicine · doi:10.1016/j.artmed.2024.103063 · Cited by 1 · see citing works
“Another innovative aspect is the integration of multimodal data, facilitated by the multivariate nature of AEs [62-64].”
Sampaio, I. W., Tassi, E., Bellani, M., Benedetti, F., Nenadić, I., Phillips, M. L., Piras, F., Yatham, L., Bianchi, A. M., Brambilla, P., et al. (2024) · bioRxiv · doi:10.1101/2024.09.04.611239 · Cited by 1 · see citing works
“AEs are suitable for integrating data and have been successfully applied to multimodal datasets [63]–[65].”
MethodECG Signal Construction From Heart Sounds via Single Node, Surface Acoustic Sensing
Li, K. Y., Iravantchi, Y., Park, H., Liu, Y., & Sample, A. P. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc53108.2024.10781974 · Cited by 1 · see citing works
“Utilizing advanced machine learning techniques, researchers have employed cross-modal autoencoders to produce cardiac MRIs from ECGs [11] and have used generative adversarial networks (GANs) to generate multi-lead ECGs from single-lead ECG input data [12].”
A review of self-supervised imaging phenotypes for genome-wide association studies
Patel, K. A., Zhao, X.-Z., Xie, Z.-Q., & Zhi, D. (2026) · Communications Biology · doi:10.1038/s42003-026-11199-4
“Genetic InfoMaxpairsimagereconstructionwithcontrastivealignmentbetween MRIandgeneticembeddings 45 , while the cross-modal autoencoder combines modality-speci fi c reconstruction losses with a contrastive loss across ECG and cMRI 74 .”
InfluentialDesigning evaluation rubrics for the development of virtual human twins in healthcare
Krishnamurthi, V., Edirisinghe, R. K., John, S. E., Wickramasinghe, N. S., & Plebanski, M. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag174
Lu, N. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104795
Amirshahi, A., Toosi, M. H., Mohammadi, S., Albini, S., Schiavone, P. D., Ansaloni, G., Aminifar, A., & Atienza, D. (2026) · Communications Medicine · doi:10.1038/s43856-026-01740-1
“This direct transfer method is common in medical research across various data types, such as medical images 38 – 41 , biomedical signals 42 – 47 , electronic health records 48 – 51 , and their combinations 52 – 55 .”
Xia, Y., Wei, Y.-R., Lan, Z., Wu, N., Li, Y.-X., & Ji, X.-Q. (2026) · Frontiers in Pediatrics · doi:10.3389/fped.2026.1888687
Causal Structure and Representation Learning with Biomedical Applications
Uhler, C. (2026) · Volume 7: Invited Lectures: Sections 15–20 · doi:10.1137/25m1805849
Self-citationPrioritizing Discovery and Advancements in Arrhythmia Therapies
Hanna, P., Boyle, P. M., Caldwell, J. L., Refaey, M. E., Goodyer, W. R., Khurshid, S. S., Mesubi, O. O., Palatinus, J. A., Pfenniger, A., Ajijola, O. A., et al. (2026) · JACC. Clinical electrophysiology · doi:10.1016/j.jacep.2026.05.005
Self-citationXu, C., Li, S., Xuan, W., & Wang, H. (2026) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp55912.2026.11464795
Tu, Y., Yang, X.-R., Liu, F., Ai, R.-Y., Luo, Y., Ran, C.-M., Yang, G.-C., Zhang, T.-J., & Jiang, L. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03466-5
Wu, Z.-Y., Ji, L., Tang, W., Zhou, S.-J., Yan, Q., Chen, X.-M., Zhu, J., Ma, J., Zhao, Y., & He, W. (2026) · BMC Cardiovascular Disorders · doi:10.1186/s12872-026-05660-9
“The successful deployment of such foundational models requires training on massive, robustly annotated datasets that exceed the capacity of typical single-center studies [60].”
Transcending the Annotation Bottleneck: AI-Powered Discovery in Biology and Medicine
Chatterjee, S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-17216-7_19
“Building upon this, Radhakrishnan et al. [17] utilised cross-modal autoen-coders to learn holistic representations of the cardiovascular state.”
Position: Evaluation of ECG Representations Must Be Fixed
Berger, Z., Prakah-Asante, D., Guttag, J. V., & Stultz, C. M. (2026) · arXiv.org · doi:10.48550/arxiv.2602.17531
“There is also growing interest in multi-modal representations that include ECG as a component (e.g., Radhakrishnan et al., 2023; Thapa et al., 2024); evaluating such models should be addressed in future studies.”
Minh, L. H. N., Nguyen, T.-H., Li, T., Li, T., Huynh, H. H., Raj, M., Yang, C., Xu, M., Vinh, T., & Le, N. Q. K. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag365 · Cited by 1 · see citing works
“Designing modular, open-source implementations that separate encoders, fusion layers, and evaluation modules—and that ship with pretrained weights where licensing permits—will be important to make imaging-anchored multiomics analyses transparent and reusable across cardiovascular centres.”
InfluentialPhenotype-driven parallel embedding for microbiome multi-omic data integrationPreprint
Bamberger, T., & Borenstein, E. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.05.692119 · Cited by 2 · see citing works
“For example, a recently proposed approach for integrating multiple modalities involved training a model on paired ECG and cardiac MRI data to map each modality into a common latent representation 34 .”
Theoretical Framework and Key Technologies of Electric Power Intelligent Computing Systems
Fang, F., Wu, X.-Y., Cheng, L., Hu, Y., Sun, L., Xu, Y.-H., & Zhang, Y. (2025) · Scientia Sinica Technologica · doi:10.1360/sst-2025-0152
“无监 督对齐通过典型相关分析（ CCA ）优化模态间协方差 矩阵 , 或采用循环一致性自编码器（ CycleAE ）构建 则建立局部特征关联 [88] .”
Contrastive Cross-Modal Learning for Infusing Chest X-ray Knowledge into ECGs
Punyamoorty, V., Malusare, A., & Aggarwal, V. (2025) · arXiv.org · doi:10.48550/arxiv.2506.19329
“…framework that adds explicit supervision from labels to enhance task-relevant representation learning; ECCL (Ding et al., 2024) and Cross-modal AE (Radhakrishnan et al., 2023) which are cross-modal transfer frameworks from Cardiac Magnetic Resonance (CMR) images to ECGs; BIOT (Yang et al., 2023),…”
Self-Supervised Learning for Biomedical SignalProcessing: A Systematic Review on ECG andPPG Signals
Ding, C., & Wu, C. (2024) · medRxiv · doi:10.1101/2024.09.30.24314588
“Catering to the core challenge in diagnostics, [36] addresses the harmonious integration of diverse modalities, forging a unified portrayal of the physiological state.”
BackgroundGenetic underpinnings of predicted changes in cardiovascular function using self supervised learningPreprint
Levine, Z., Lutsker, G., Godneva, A., Weinberger, A., Pompan, M., Talmor-Barkan, Y., Reisner, Y., Rossman, H., & Segal, E. (2024) · bioRxiv · doi:10.1101/2024.08.15.608061
“There are a few previous works that combine cardiac SSL and genetic studies, though existing studies with ECG are largely cross-sectional, and focus on the factors that cause an between-9 patient differences in cardiovascular state at a single time point 1,21 .”
BackgroundMethod
- Authors
- Radhakrishnan, A., Friedman, S. F., Khurshid, S., Ng, K., Batra, P., Lubitz, S. A., Philippakis, A. A., & Uhler, C.
- Affiliations
- Massachusetts Institute of Technology (US)
- Broad Institute (US)
- IBM (United States) (US)
- Corresponding author
- Lubitz, Steven A.
- Philippakis, Anthony A.
- Uhler, Caroline
- ORCID
- Venue
- Nature Communications
- Published
- 2023-04-28
- Volume
- 14
- Issue
- 1
- Pages
- 2436-2436
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC10140057
- OpenAlex
- W4367318518
- Cited by
- 82
- References
- 82
- Citations by year
- 2026: 26 · 2025: 23 · 2024: 27 · 2023: 5 · 2022: 1
- FWCI
- 15.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Eric and Wendy Schmidt Center at the Broad Institute
- American Heart Association (18SFRN34250007)
- National Science Foundation (DMS-1651995)
- United States Department of Defense | United States Navy | Office of Naval Research (N00014-17-1-2147)
- Simons Foundation (Simons Investigator Award)
- AstraZeneca, MIT-IBM Watson AI Lab
- Topics
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Cardiac electrophysiology and arrhythmias
- Keywords
- cardiac magnetic resonance imaging, electrocardiogram, multimodal data integration, phenotype prediction
- MeSH terms
- Cardiovascular System, Electrocardiography, Heart, Learning, Genome-Wide Association Study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
