Multimodal contrastive learning for spatial gene expression prediction using histology images
Open accessThe DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images stained with Hematoxylin and Eosin (H&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose mclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a “word”, integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. We conducted an extensive evaluation of highly variable genes in two breast cancer datasets and a skin squamous cell carcinoma dataset, and the results demonstrate that mclSTExp exhibits superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Figure 2 Evaluation of gene expression prediction on the HER2+ datasets by the PCC (ACGs) between the observed and predicted gene expression by STnet [21], HisToGene [15], His2ST [22], THItoGene [23], BLEEP [24], and mclSTExp.](https://figures.cochat.ai/paper-figures/eed4d9dafca54d768e758b1c355cca56/multimodal-contrastive-learning-for-spatial-gene-expression-prediction-using-figure-2.jpg)
![Figure 3 Evaluation of gene expression prediction on the cSCC datasets by the PCC (ACGs) between the observed and predicted gene expression by STnet [21], HisToGene [15], His2ST [22], THItoGene [23], BLEEP [24], and mclSTExp.](https://figures.cochat.ai/paper-figures/9e6ed288e5f6467d90caa63c3d8f94f1/multimodal-contrastive-learning-for-spatial-gene-expression-prediction-using-figure-3.jpg)
![Figure 4 Evaluation of gene expression prediction on the Alex+10x datasets by the PCC (ACGs) between the observed and predicted gene expression by STnet [21], HisToGene [15], His2ST [22], THItoGene [23], BLEEP [24], and mclSTExp.](https://figures.cochat.ai/paper-figures/e692a2c699de402aaba493ab171e099c/multimodal-contrastive-learning-for-spatial-gene-expression-prediction-using-figure-4.jpg)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“mclSTExp integrates image features with spatial context via a Transformer encoder, achieving superior performance in predicting highly variable genes in breast cancer and skin squamous cell carcinoma datasets (Min et al., 2024).”
From Spatial pharmacology of anti-cancer immunomodulators: a comprehensive review (2026) · cites this paper as Min et al., 2024
Experimental — accuracy not yet measuredIn the citing paper
“Several studies have explored predicting gene expression from H&E images using AI models59–62.”
From A visual–omics foundation model to bridge histopathology with spatial transcriptomics (2025) · cites this paper as [62]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“In contrast to the skip connections in ResNet [30], the dense connectivity mechanism in DenseNet-121 enhances the reusability of features, aiding the neural network in capturing image features more effectively and thereby strengthening the model’s expressive capability [26].”
ResultDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 1×
“Bleep utilizes a contrastive learning [28] approach, introducing image and gene expression encoders to learn joint embedding in space.”
Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays
Chen, A., Liao, S., Cheng, M., Ma, K., Wu, L., Lai, Y., Qiu, X., Yang, J., Xu, J., Hao, S., et al. (2022) · Cell · doi:10.1016/j.cell.2022.04.003 · Cited by 1,784
A single-cell and spatially resolved atlas of human breast cancers
Wu, S. Z., Al-Eryani, G., Roden, D. L., Junankar, S., Harvey, K., Andersson, A., Thennavan, A., Wang, C., Torpy, J. R., Bartonicek, N., et al. (2021) · Nature Genetics · doi:10.1038/s41588-021-00911-1 · Cited by 1,775 · cited 1×
“• The higher resolution (55 μ m per spot) Alex + 10x dataset contains 9 breast cancer tissue samples, including 3 samples (2 fresh-frozen and 1 formalin-fixed-paraffin-embedded, FFPE) from 10x Genomics [30] and six samples obtained from Swarbrick’s laboratory [31].”
Self-citationExploring tissue architecture using spatial transcriptomics
Rao, A., Barkley, D., França, G. S., & Yanai, I. (2021) · Nature · doi:10.1038/s41586-021-03634-9 · Cited by 1,700 · cited 1×
“With the rapid development of spatial transcriptomics (ST) technology, we may gain a more comprehensive understanding of gene expression patterns within complex biological systems [1].”
2007 IEEE Conference on Computer Vision and Pattern Recognition
(2007) · doi:10.1109/cvpr12085.2007 · Cited by 1,688
Multimodal Analysis of Composition and Spatial Architecture in Human Squamous Cell Carcinoma
Ji, A. L., Rubin, A. J., Thrane, K., Jiang, S., Reynolds, D. L., Meyers, R. M., Guo, M. G., George, B. M., Mollbrink, A., Bergenstråhle, J., et al. (2020) · Cell · doi:10.1016/j.cell.2020.05.039 · Cited by 1,117 · cited 1×
“• The lower resolution (100 μ m per spot) cSCC dataset [29] contains 12 tissue sections obtained from four patients, with each patient contributing three sections.”
Research works printed but not linked: 2
- Xie, 2024, Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning, 1
- Dosovitskiy, 2021, An image is worth 16x16 words: transformers for image recognition at scale, 1
Cited by
Items 101–137 of 137
Zhang, T., Li, S., Zhang, H., Zhang, R., Zhao, Z., Wang, R., Teng, X., Wan, L., Jiang, Y., Lyu, J., et al. (2026) · Genome biology · doi:10.1186/s13059-026-04160-5
“In addition, recent methods such as SpaMask [16], SpaCross [17], SpaBatch [18], and mclSTExp [19] have further advanced spatial transcriptomics analysis from the perspectives of self-supervised spatial representation learning, cross-section integration, batch-effect correction, and multimodal…”
Huang, J., Feng, X. H., Qu, L., & Zheng, L. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.15.732498
SpaBiT: enhancing spatial transcriptomics resolution via bidirectional attention transformers
Liu, X. W., Li, A., & Min, W. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag443
“However, experimental resolution enhancement is costly and time-consuming for large-scale studies (Min et al. 2024).”
Self-citationTang, L., Han, W., Yang, X., Chen, X., & Li, H. (2026) · Research Square · doi:10.64898/2026.06.29.735170
DFCE-KanT: predicting spatial gene expression from histology images via contrastive learning
Li, F., Wang, P., Shen, J., & Wu, Y. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag502
“In addition, BLEEP (Xie et al. 2024) and mclSTExp (Min et al. 2024) apply contrastive learning to align image and gene expression features in a joint embedding space, but still exhibit suboptimal performance.”
Structural-information guided fusion for spatial domain identification from spatial transcriptomics
Zhang, M., Gao, P., Chen, C., Ma, X. Y., Chen, X., & Feng, S. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag508
“Quantitatively, SGFST achieves the highest ARI score of 0.751, substantially out performing competing approaches, including BANKSY (0.338), stLearn (0.161), SCANPY (0.105), STAGATE (0.480), SpaGCN (0.443), DeepST (0.494), PROST (0.611), MNMST (0.619), ST-SCSR (0.675), Spatial-MGCN (0.271), MuCST (0.507) and SiDMGF (0.…”
InfluentialMao, Y., Kang, H., Wang, L., & Zhao, M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3498-1_41
Yin, G., Liu, S., Zhang, Z., Zhang, S., & Li, L. (2026) · Small Methods · doi:10.1002/smtd.70851
“…studies have developed a range of graph neural network-based frameworks for spatial transcriptomic analysis, incorporating strategies such as topology reconstruction, curriculum-guided fusion, consensus representation learning, cross-slice alignment, and multi-modal contrastive learning [ 16–21 ].”
Mao, Y., & Min, W. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3716-6_26
Self-citationAI-enabled reconstruction of 3D spatial multi-omics at single-cell resolutionPreprint
Wang, Z., Yan, Y., Yang, X., Zhang, D., Han, C., Zou, Q., Du, Y., Hu, Z., & Yuan, Z. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.09.737490
“To In theory, any H&E-based omics prediction methods 6,8-14 can function as SPONGE by training multiple single-omics prediction models and then inferring on the whole tissue volume. assess SPONGE in diagonal integration, we implemented extensive assessments (Fig.”
PRISM: Prior-enhanced Inference for Spatial Transcriptomic Cell Type Mapping
Xu, Y., Wang, X., Liu, S. J., Ge, C., Chen, X., Wang, Y., Yu, B., & Li, X.-M. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag515
“…meth-ods for spatial transcriptomics has accordingly emerged, including SpaMask (Min et al. 2025) for spatial-domain representation learning, mclSTExp (Min et al. 2024) for cross-modal expression prediction, and scstGCN (Xue et al. 2024) for single-cell-resolution inference from spot-based ST.”
Asymmetric dual pathway fusion for histology driven spatial transcriptomic prediction
Su, X., Liu, R., Zhang, K., Ou, Z., & Gao, Y. (2026) · Scientific Reports · doi:10.1038/s41598-026-63426-x
HistAgent: an evidence-grounded agent for spatial molecular reasoning from routine histologyPreprint
Liu, J., Li, W., Liu, H., Lu, Y., Liao, X., Sun, X., Zhai, W., & Lin, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10677502/v1
Chen, T., Unjitwattana, T., Guo, X., Thakur, P., Zhou, S., Zheng, J., Yang, Y., Du, Y., Zou, W., & Garmire, L. X. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01754-z
BioKERN: Biological Kernel Regularization for Histology-to-Transcriptomics Neighborhood RetrievalPreprint
Cho, S. Y., & Orcan-Ekmekci, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.24823
“…the use of an explicit molecular–spatial biological reference geometry for both retrieval supervision and representation regularization. mclSTExp [Yang et al., 2024] extends this with multimodal contrastive learning incorporating spatially contextualized expression representations, while ConGcR…”
st2traj : deconvolution-informed trajectory inference for multi-timepoint spatial transcriptomics
Wang, Z., & Zhang, C. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag645
Liu, X., Hu, Y., Yang, S., Wei, J., & Li, X. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01252-0
“Methods such as SpaMask [21], SpaBatch [22], SpaCross [23], mclSTExp [24], and SpaICL [25] utilize graph neural networks or graph attention mechanisms to model spatial dependencies, integrate multi-modal data, and correct batch effects.”
Unifying physical and molecular coordinate systems across modalities in spatial biologyPreprint
Dai, B., Yan, Y., Wang, Z., Liang, Y., Li, S., Hu, P., Yang, X., Wang, C., Yi, L., Sun, C., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.01.748536
Self-citationscGSI: Graph-guided self-supervised integration of paired single-cell multi-omics
Chen, X., Yang, Z., Liu, X., Xie, Z., & Guo, W. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014773
Abuhassan, Q., Al‐Khreisat, M. J., Roopashree, R., Chanania, K., Sudhakar, T., Sharma, V., Chauhan, A. S., & Shermatov, K. (2026) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2026.101949
Integrating AI-Based Digital Pathology with Spatial Transcriptomics for Deeper Insight
Rezania, M. A., Yi, H., Choi, J., Lee, D., & Noh, M. (2026) · Cancer Research and Treatment · doi:10.4143/crt.2026.0795
Liu, X., Wang, Y., & Min, W. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114981
Self-citation葛志广, & Cai, H. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.14.751379
Predicting spatial transcriptomics from histology images: Progress, challenges, and directions
Xu, K., Koohy, H., & Antanaviciute, A. (2026) · doi:10.70401/exo.2026.0020
Towards Trustworthy Biological Alignment in TabPFN-Probed Pathology Foundation ModelsPreprint
Bhattacharjee, U., Das, A., Hannan, S., Roy, T., Howlader, K., & Sarkar, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.29523
“Histology-to-expression methods such as STNet (He et al. 2020), BLEEP (Xie et al. 2023), and mclSTExp (Min et al. 2024) predict molecular states from tissue morphology, while multimodal approaches such as TANGLE (Jaume et al. 2024b), SPADE (Redekop and others 2025), SEAL (Anony-mous 2026), and…”
Ruan, S., Ling, X., He, Q., Yan, Z., Yuan, H., Guan, T., Xiao, Y., Guan, X., & He, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.28563
“BLEEP and mclST-Exp moved toward contrastive image–expression alignment, whereas GenST explored cross-modal latent generation (Xie et al. 2023; Min et al. 2024; Wood et al. 2026).”
Li, Y., Peng, X., Ding, W., Zhou, Y., & Zhang, X. (2026) · Journal of Computational Science · doi:10.1016/j.jocs.2026.103026
Vila-Bagaria, S., Teixidó, M. D., Piñol, M., Vilardell, F., Montal, R., & Vilaplana, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05157
“Emerging approaches instead leverage contrastive learning to align modalities [12], integrating spatial transcriptomics [16] or enforcing semantic consistency [24] to enhance robustness.”
Wang, Y., Guo, S., Qian, J., Liang, H., Su, Y., Yu, Z., & Li, X. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c02880
Zhi, D., Guan, X., Wang, L., & Chen, W. (2026) · Physiologia Plantarum · doi:10.1111/ppl.71143
Alpha Spatial: Language-guided prediction of single-cell spatial gene expression from histologyPreprint
YANG, X., Guo, Z., GUO, X., Wang, W., Bai, B., Liu, X., Zhang, C., He, Y., & LIU, Y. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.10.08.756444
Liu, Z., Liu, J., Li, S., Feng, Y., & Wang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.28339
“ST-Net (He et al. 2020) established an early image-based prediction framework, while BLEEP (Xie et al. 2023) and mclSTExp (Min et al. 2024) explored cross-modal alignment for expression prediction.”
Zheng, H., Lyu, Q., Zhu, X., Zhang, H., Wang, W., & Li, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134846
An, V., Nguyen, L. D., Luong, H. V. D., & Nguyen, B. P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.35167
“The method baselines include MERGE [8], TRIPLEX [5], mclSTExp [23], EGN [34], STFlow [14], and M2OST [29].”
InfluentialHiFi-ST: Histology-Conditioned Local Coordinate Modeling for Spatial Gene Expression Prediction
Tang, L. (2026) · Frontiers in Computing and Intelligent Systems · doi:10.54097/6pg9t237
Zhu, J., Deng, R., Qu, C., Lu, Z., Zhu, Y., Yang, Y., Wang, Y., Zhao, S., Yang, H., & Huo, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.39151
“…1.515 0.137 0.728 0.227 His2ST [4] 0.603 0.244 0.740 0.148 1.667 0.117 0.776 0.148 EGN [9] 0.603 0.268 0.677 0.124 1.601 0.137 0.770 0.118 mclSTExp [10] 0.593 0.259 0.745 0.128 1.804 0.142 0.871 0.201 OmiCLIP [6] 0.469 0.305 0.662 0.140 1.487 0.165 0.732 0.282 BLEEP [7] 0.456 0.287 0.590 0.169…”
InfluentialSpatial omics in cancer: a review of computational approaches and vision of AI-driven analysis
Arvind, A., Duhon, M., Su, Y.-C., Kim, J., & Li, X. (2026) · Genomics · doi:10.1016/j.ygeno.2026.111340
Metadata
- Authors
- Min, W., Shi, Z., Zhang, J., Wan, J., & Wang, C.
- Affiliations
- Yunnan University (CN)
- Zhongnan University of Economics and Law (CN)
- Shenzhen Research Institute of Big Data (CN)
- Corresponding author
- Min, Wenwen
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-09-23
- Volume
- 25
- Issue
- 6
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- PMC
- PMC11952928
- OpenAlex
- W4403853489
- Cited by
- 90
- References
- 44
- Citations by year
- 2026: 55 · 2025: 26 · 2024: 9
- FWCI
- 12.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Briefings in Bioinformatics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62262069)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- spatial transcriptomics, gene expression prediction, histological images, H&E staining, multimodal contrastive learning, Transformer encoder, DenseNet121, spatial context, breast cancer, cutaneous squamous cell carcinoma, highly variable genes, immune-related genes, spatial domain
- MeSH terms
- Machine Learning, Algorithms, Breast Neoplasms, Female, Humans, Image Processing, Computer-Assisted, Gene Expression Regulation, Neoplastic, Computational Biology, Gene Expression Profiling, Transcriptome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.