Multimodal contrastive learning for spatial gene expression prediction using histology images
Open accessThe DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images stained with Hematoxylin and Eosin (H&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose mclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a “word”, integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. We conducted an extensive evaluation of highly variable genes in two breast cancer datasets and a skin squamous cell carcinoma dataset, and the results demonstrate that mclSTExp exhibits superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Figure 2 Evaluation of gene expression prediction on the HER2+ datasets by the PCC (ACGs) between the observed and predicted gene expression by STnet [21], HisToGene [15], His2ST [22], THItoGene [23], BLEEP [24], and mclSTExp.](https://figures.cochat.ai/paper-figures/eed4d9dafca54d768e758b1c355cca56/multimodal-contrastive-learning-for-spatial-gene-expression-prediction-using-figure-2.jpg)
![Figure 3 Evaluation of gene expression prediction on the cSCC datasets by the PCC (ACGs) between the observed and predicted gene expression by STnet [21], HisToGene [15], His2ST [22], THItoGene [23], BLEEP [24], and mclSTExp.](https://figures.cochat.ai/paper-figures/9e6ed288e5f6467d90caa63c3d8f94f1/multimodal-contrastive-learning-for-spatial-gene-expression-prediction-using-figure-3.jpg)
![Figure 4 Evaluation of gene expression prediction on the Alex+10x datasets by the PCC (ACGs) between the observed and predicted gene expression by STnet [21], HisToGene [15], His2ST [22], THItoGene [23], BLEEP [24], and mclSTExp.](https://figures.cochat.ai/paper-figures/e692a2c699de402aaba493ab171e099c/multimodal-contrastive-learning-for-spatial-gene-expression-prediction-using-figure-4.jpg)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“mclSTExp integrates image features with spatial context via a Transformer encoder, achieving superior performance in predicting highly variable genes in breast cancer and skin squamous cell carcinoma datasets (Min et al., 2024).”
From Spatial pharmacology of anti-cancer immunomodulators: a comprehensive review (2026) · cites this paper as Min et al., 2024
Experimental — accuracy not yet measuredIn the citing paper
“Several studies have explored predicting gene expression from H&E images using AI models59–62.”
From A visual–omics foundation model to bridge histopathology with spatial transcriptomics (2025) · cites this paper as [62]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays
Chen, A., Liao, S., Cheng, M., Ma, K., Wu, L., Lai, Y., Qiu, X., Yang, J., Xu, J., Hao, S., et al. (2022) · Cell · doi:10.1016/j.cell.2022.04.003 · Cited by 1,784
A single-cell and spatially resolved atlas of human breast cancers
Wu, S. Z., Al-Eryani, G., Roden, D. L., Junankar, S., Harvey, K., Andersson, A., Thennavan, A., Wang, C., Torpy, J. R., Bartonicek, N., et al. (2021) · Nature Genetics · doi:10.1038/s41588-021-00911-1 · Cited by 1,775
Self-citationExploring tissue architecture using spatial transcriptomics
Rao, A., Barkley, D., França, G. S., & Yanai, I. (2021) · Nature · doi:10.1038/s41586-021-03634-9 · Cited by 1,700
2007 IEEE Conference on Computer Vision and Pattern Recognition
(2007) · doi:10.1109/cvpr12085.2007 · Cited by 1,688
Multimodal Analysis of Composition and Spatial Architecture in Human Squamous Cell Carcinoma
Ji, A. L., Rubin, A. J., Thrane, K., Jiang, S., Reynolds, D. L., Meyers, R. M., Guo, M. G., George, B. M., Mollbrink, A., Bergenstråhle, J., et al. (2020) · Cell · doi:10.1016/j.cell.2020.05.039 · Cited by 1,117 · cited 1×
“• The lower resolution (100 μ m per spot) cSCC dataset [29] contains 12 tissue sections obtained from four patients, with each patient contributing three sections.”
Research works printed but not linked: 2
- Xie, 2024, Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning, 1
- Dosovitskiy, 2021, An image is worth 16x16 words: transformers for image recognition at scale, 1
Cited by
A visual–omics foundation model to bridge histopathology with spatial transcriptomics
Chen, W., Zhang, P., Tran, T. N., Xiao, Y., Li, S., Shah, V. V., Cheng, H., Brannan, K. W., Youker, K., Lai, L., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02707-1 · Cited by 118
Machine learning methods for histopathological image analysis: updates in 2024
Komura, D., Ochi, M., & Ishikawa, S. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.033 · Cited by 76
SpaMask: Dual masking graph autoencoder with contrastive learning for spatial transcriptomics
Min, W., Fang, D., Chen, J., & Zhang, S. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012881 · Cited by 63
Self-citationDiffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology ImagesPreprint
Zhu, S.-C., Zhu, Y., Tao, M., & Qiu, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.15598 · Cited by 49
AI-predicted spatial transcriptomics unlocks breast cancer biomarkers from pathology
Shulman, E. D., Campagnolo, E. M., Lodha, R., Chung, Y., Stemmer, A., Cantore, T., Ru, B., Chang, T., Biswas, S., Dhruba, S. R., et al. (2026) · Cell · doi:10.1016/j.cell.2026.04.023 · Cited by 50
Self-citation
Metadata
- Authors
- Min, W., Shi, Z., Zhang, J., Wan, J., & Wang, C.
- Affiliations
- Yunnan University (CN)
- Zhongnan University of Economics and Law (CN)
- Shenzhen Research Institute of Big Data (CN)
- Corresponding author
- Min, Wenwen
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-09-23
- Volume
- 25
- Issue
- 6
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- PMC
- PMC11952928
- OpenAlex
- W4403853489
- Cited by
- 90
- References
- 44
- Citations by year
- 2026: 55 · 2025: 26 · 2024: 9
- FWCI
- 12.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Briefings in Bioinformatics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62262069)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- spatial transcriptomics, gene expression prediction, histological images, H&E staining, multimodal contrastive learning, Transformer encoder, DenseNet121, spatial context, breast cancer, cutaneous squamous cell carcinoma, highly variable genes, immune-related genes, spatial domain
- MeSH terms
- Machine Learning, Algorithms, Breast Neoplasms, Female, Humans, Image Processing, Computer-Assisted, Gene Expression Regulation, Neoplastic, Computational Biology, Gene Expression Profiling, Transcriptome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal contrastive learning for spatial gene expression prediction using histology images. https://app.cochat.ai/papers/10.1093/bib/bbae551
- The paper
- Min, W., Shi, Z., Zhang, J., Wan, J., & Wang, C. (2024). Multimodal contrastive learning for spatial gene expression prediction using histology images. Briefings in Bioinformatics. https://doi.org/10.1093/bib/bbae551
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.