Integrating spatial gene expression and breast tumour morphology via deep learning
Abstract
Spatial transcriptomics allows for the measurement of RNA abundance at a high spatial resolution, making it possible to systematically link the morphology of cellular neighbourhoods and spatially localized gene expression. Here, we report the development of a deep learning algorithm for the prediction of local gene expression from haematoxylin-and-eosin-stained histopathology images using a new dataset of 30,612 spatially resolved gene expression data matched to histopathology images from 23 patients with breast cancer. We identified over 100 genes, including known breast cancer biomarkers of intratumoral heterogeneity and the co-localization of tumour growth and immune activation, the expression of which can be predicted from the histopathology images at a resolution of 100 µm. We also show that the algorithm generalizes well to The Cancer Genome Atlas and to other breast cancer gene expression datasets without the need for re-training. Predicting the spatially resolved transcriptome of a tissue directly from tissue images may enable image-based screening for molecular biomarkers with spatial variation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Datasets combining single-cell transcriptomics with histology can help understand how cells interact and what molecular signatures underpin the formation of specialized spatial niches.
“For example, datasets that bridge molecular and spatial scales, such as single-cell transcriptomics data combined with histology to understand how cells interact and what molecular signatures underpin the formation of specialized spatial niches.107”
From How to build the virtual cell with artificial intelligence: Priorities and opportunities (2024) · cites this paper as [107]
Researchers have utilized deep learning to leverage histopathology images to predict spatial gene expression from these images alone, pointing to morphological features not captured by human experts that could potentially enhance utility and lower costs.
“Of note, researchers have utilized deep learning to leverage histopathology images to predict spatial gene expression from these images alone, pointing to morphological features in these images not captured by human experts that could potentially enhance the utility and lower the costs of this technology 25,26 .”
From Multimodal biomedical AI (2022) · cites this paper as [25]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources
Huang, D. W., Sherman, B. T., & Lempicki, R. A. (2009) · Nature Protocols · doi:10.1038/nprot.2008.211 · Cited by 35,890 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Comprehensive molecular portraits of human breast tumours
The Cancer Genome Atlas Network (2012) · Nature · doi:10.1038/nature11412 · Cited by 12,129 · see citing works
UMAP: Uniform Manifold Approximation and Projection
McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018) · Journal of Open Source Software · doi:10.21105/joss.00861 · Cited by 9,826 · see citing works
Intratumor Heterogeneity and Branched Evolution Revealed by Multiregion Sequencing
Gerlinger, M., Rowan, A. J., Horswell, S., Larkin, J., Endesfelder, D., Gronroos, E., Martinez, P., Matthews, N., Stewart, A., Tarpey, P., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1113205 · Cited by 7,774 · see citing works
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,673 · see citing works
CellProfiler: image analysis software for identifying and quantifying cell phenotypes
Carpenter, A. E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang, I. H., Friman, O., Guertin, D. A., Chang, J. H., Lindquist, R. A., Moffat, J., et al. (2006) · Genome biology · doi:10.1186/gb-2006-7-10-r100 · Cited by 5,172 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
Visualization and analysis of gene expression in tissue sections by spatial transcriptomics
Ståhl, P. L., Salmén, F., Vickovic, S., Lundmark, A., Navarro, J. F., Magnusson, J., Giacomello, S., Asp, M., Westholm, J. O., Huss, M., et al. (2016) · Science · doi:10.1126/science.aaf2403 · Cited by 3,834 · see citing works
Self-citationSpatially resolved, highly multiplexed RNA profiling in single cells
Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S., & Zhuang, X. (2015) · Science · doi:10.1126/science.aaa6090 · Cited by 2,969 · see citing works
Coudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö, D., Moreira, A. L., Razavian, N., & Tsirigos, A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0177-5 · Cited by 2,930 · see citing works
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · see citing works
Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH
Eng, C.-H. L., Lawson, M., Zhu, Q., Dries, R., Koulena, N., Takei, Y., Yun, J., Cronin, C., Karp, C., Yuan, G.-C., et al. (2019) · Nature · doi:10.1038/s41586-019-1049-y · Cited by 1,754 · see citing works
Kamentsky, L. D., Jones, T. R., Fraser, A. N., Bray, M., Logan, D. J., Madden, K. L., Ljosa, V., Rueden, C., Eliceiri, K. W., & Carpenter, A. E. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr095 · Cited by 1,002 · see citing works
Highly Multiplexed Subcellular RNA Sequencing in Situ
Lee, J. H., Daugharthy, E. R., Scheiman, J., Kalhor, R., Yang, J. L., Ferrante, T. C., Terry, R., Jeanty, S. S. F., Li, C., Amamoto, R., et al. (2014) · Science · doi:10.1126/science.1250212 · Cited by 973 · see citing works
Yu, K.-H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016) · Nature Communications · doi:10.1038/ncomms12474 · Cited by 972 · see citing works
Detecting Cancer Metastases on Gigapixel Pathology Images
Liu, Y., Gadepalli, K., Norouzi, M., Dahl, G. E., Kohlberger, T., Boyko, A., Venugopalan, S., Timofeev, A., Nelson, P. Q., Corrado, G. S., et al. (2017) · arXiv · doi:10.48550/arxiv.1703.02442 · Cited by 506 · see citing works
1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset
Litjens, G. J. S., Bándi, P., Bejnordi, B. E., Geessink, O. G. F., Balkenhol, M. C. A., Bult, P., Halilović, A., Hermsen, M., van de Loo, R., Vogels, R. J., et al. (2018) · GigaScience · doi:10.1093/gigascience/giy065 · Cited by 414 · see citing works
Deep Convolutional Neural Networks Enable Discrimination of Heterogeneous Digital Pathology Images
Khosravi, P., Kazemi, E., Imielinski, M., Elemento, O., & Hajirasouliha, I. (2018) · EBioMedicine · doi:10.1016/j.ebiom.2017.12.026 · Cited by 334 · see citing works
Sparse PCA corrects for cell type heterogeneity in epigenome-wide association studies
Rahmani, E., Zaitlen, N., Baran, Y., Eng, C. S., Hu, D., Galanter, J., Oh, S. S., Burchard, E. G., Eskin, E., Zou, J., et al. (2016) · Nature Methods · doi:10.1038/nmeth.3809 · Cited by 268 · see citing works
Self-citationMacrophage-Derived SPARC Bridges Tumor Cell-Extracellular Matrix Interactions toward Metastasis
Sangaletti, S., Carlo, E. D., Gariboldi, S., Miotti, S., Cappetti, B., Parenza, M., Rumio, C., Brekken, R. A., Chiodoni, C., & Colombo, M. P. (2008) · Cancer Research · doi:10.1158/0008-5472.can-08-1327 · Cited by 208 · see citing works
Barcoded solid-phase RNA capture for Spatial Transcriptomics profiling in mammalian tissue sections
Salmén, F., Ståhl, P. L., Mollbrink, A., Navarro, J. F., Vicković, S., Frisén, J., & Lundeberg, J. (2018) · Nature Protocols · doi:10.1038/s41596-018-0045-2 · Cited by 193 · see citing works
Self-citationCheng, J., Zhang, J., Han, Y., Wang, X., Ye, X., Meng, Y., Parwani, A. V., Han, Z., Feng, Q., & Huang, K. (2017) · Cancer Research · doi:10.1158/0008-5472.can-17-0313 · Cited by 142 · see citing works
Yu, K., Wang, F., Berry, G. J., Ré, C., Altman, R. B., Snyder, M. P., & Kohane, I. S. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz230 · Cited by 123 · see citing works
Biglycan is a specific marker and an autocrine angiogenic factor of tumour endothelial cells
Yamamoto, K., Ohga, N., Hida, Y., Maishi, N., Kawamoto, T., Kitayama, K., Akiyama, K., Osawa, T., Kondoh, M., Matsuda, K., et al. (2012) · British Journal of Cancer · doi:10.1038/bjc.2012.59 · Cited by 107 · see citing works
Kurozumi, S., Matsumoto, H., Kurosumi, M., Inoue, K., Fujii, T., Horiguchi, J., Shirabe, K., Oyama, T., & Kuwano, H. (2019) · Oncology Letters · doi:10.3892/ol.2019.9938 · Cited by 58 · see citing works
Ladha, J., Sinha, S., Bhat, V., et al. (2012) · Molecular Cancer Research · doi:10.1158/1541-7786.mcr-11-0488 · Cited by 41 · see citing works
MRNA stability and overexpression of fatty acid synthase in human breast cancer cell lines.
Hunt, D. A., Lane, H. M., Zygmont, M. E., et al. (2007) · PubMed · Cited by 34 · see citing works
Modeling Spatial Correlation of Transcripts with Application to Developing Pancreas
Liu, R., Mignardi, M., Jones, R. C., Enge, M., Kim, S. K., Quake, S., & Zou, J. (2018) · Scientific Reports · doi:10.1038/s41598-019-41951-2 · Cited by 16 · see citing works
Self-citationYu, K., Wang, F., Berry, G. J., Ré, C., Altman, R. B., Snyder, M. P., & Kohane, I. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/530360 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Exploring tissue architecture using spatial transcriptomics
Rao, A., Barkley, D., França, G. S., & Yanai, I. (2021) · Nature · doi:10.1038/s41586-021-03634-9 · Cited by 1,700 · see citing works
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · see citing works
Best practices for single-cell analysis across modalities
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00586-w · Cited by 1,051 · see citing works
“Tangram 224 , GridNet 230 and eggplant 231 generate common coordinate frameworks across samples to mitigate this issue 232 .”
BackgroundIntegrating single-cell and spatial transcriptomics to elucidate intercellular tissue dynamics
Longo, S. K., Guo, M. G., Ji, A. L., & Khavari, P. A. (2021) · Nature Reviews Genetics · doi:10.1038/s41576-021-00370-8 · Cited by 1,059 · see citing works
Pan-cancer integrative histology-genomic analysis via multimodal deep learning
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.004 · Cited by 660 · see citing works
How to build the virtual cell with artificial intelligence: Priorities and opportunities
Bunne, C., Roohani, Y., Rosen, Y., Gupta, A., Zhang, X., Roed, M., Alexandrov, T., AlQuraishi, M., Brennan, P., Burkhardt, D. B., et al. (2024) · Cell · doi:10.1016/j.cell.2024.11.015 · Cited by 393 · see citing works
The impact of site-specific digital histology signatures on deep learning model accuracy and bias
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24698-1 · Cited by 330 · see citing works
Zeng, Y., Wei, Z., Yu, W., Yin, R., Yuan, Y., Li, B., Tang, Z., Lu, Y., & Yang, Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac297 · Cited by 211 · see citing works
Statistical and machine learning methods for spatially resolved transcriptomics data analysis
Zeng, Z., Li, Y., Li, Y., & Luo, Y. (2022) · Genome Biology · doi:10.1186/s13059-022-02653-7 · Cited by 186 · see citing works
Levy-Jurgenson, A., Tekpli, X., Kristensen, V. N., & Yakhini, Z. (2020) · Scientific Reports · doi:10.1038/s41598-020-75708-z · Cited by 155 · see citing works
Hoang, D.-T., Dinstag, G., Shulman, E. D., Hermida, L. C., Ben-Zvi, D. S., Elis, E., Caley, K., Sammut, S.-J., Sinha, S., Sinha, N., et al. (2024) · Nature Cancer · doi:10.1038/s43018-024-00793-2 · Cited by 157 · see citing works
Novel research and future prospects of artificial intelligence in cancer diagnosis and treatment
Zhang, C., Xu, J., Tang, R., Yang, J., Wang, W., Yu, X., & Shi, S. (2023) · Journal of Hematology & Oncology · doi:10.1186/s13045-023-01514-5 · Cited by 148 · see citing works
Pang, M., Su, K., & Li, M. (2021) · doi:10.1101/2021.11.28.470212 · Cited by 130 · see citing works
Multimodal contrastive learning for spatial gene expression prediction using histology images
Min, W., Shi, Z., Zhang, J., Wan, J., & Wang, C. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae551 · Cited by 90 · see citing works
Androvic, P., Schifferer, M., Perez Anderson, K., Cantuti-Castelvetri, L., Jiang, H., Ji, H., Liu, L., Gouna, G., Berghoff, S. A., Besson-Girard, S., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-39447-9 · Cited by 67 · see citing works
Zhou, J., Chen, S., Wu, Y., Li, H., Zhang, B., Zhou, L., Hu, Y., Xiang, Z., Li, Z., Chen, N., et al. (2024) · Science Advances · doi:10.1126/sciadv.adh8601 · Cited by 63 · see citing works
Deep learning in spatial transcriptomics: Learning from the next next-generation sequencing
Heydari, A. A., & Sindi, S. S. (2023) · Biophysics Reviews · doi:10.1063/5.0091135 · Cited by 44 · see citing works
Zheng, Y., Conrad, R. D., Green, E. J., Burks, E. J., Betke, M., Beane, J. E., & Kolachalama, V. B. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3386108 · Cited by 44 · see citing works
“Several methods combining spatial omic data and extracted pathology features have been published [1], [2], [3] to improve cell type identification, cell type deconvolution, spatial pattern recognition, and predict omic features on pathology images alone.”
MethodPatkar, S., Chen, A., Basnet, A., Bixby, A., Rajendran, R., Chernet, R., Faso, S., Kumar, P. A., Desai, D., El-Zammar, O., et al. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00765-w · Cited by 41 · see citing works
- Authors
- He, B., Bergenstråhle, L., Stenbeck, L., Abid, A., Andersson, A., Borg, Å., Maaskola, J., Lundeberg, J., & Zou, J.
- Affiliations
- Stanford University (US)
- KTH Royal Institute of Technology (SE)
- Lund University (SE)
- Science for Life Laboratory (SE)
- Chan Zuckerberg Biohub San Francisco (US)
- Corresponding author
- Maaskola, Jonas
- Lundeberg, Joakim
- Zou, James
- ORCID
- Venue
- Nature Biomedical Engineering
- Published
- 2020-06-22
- Volume
- 4
- Issue
- 8
- Pages
- 827-834
- ISSN
- 2157-846X (ISSN-L)
- OpenAlex
- W3036882247
- Cited by
- 507
- References
- 37
- Citations by year
- 2026: 101 · 2025: 117 · 2024: 106 · 2023: 75 · 2022: 57 · 2021: 44 · 2020: 7
- FWCI
- 17.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Nature Biomedical Engineering (publishedVersion)
- PubMed (publishedVersion)
- Lund University Publications (Lund University) (submittedVersion)
- Funders
- Svenska Forskningsrådet Formas
- Chan-Zuckerberg Biohub Investigator Award
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- spatial transcriptomics, breast cancer, histopathological images, H&E staining, deep learning, spatial gene expression prediction, intratumoral heterogeneity, molecular biomarkers, image-based screening, TCGA
- MeSH terms
- Deep Learning, Algorithms, Breast Neoplasms, Female, Humans, Image Processing, Computer-Assisted, Biomarkers, Tumor, Reproducibility of Results, Gene Expression Profiling, Transcriptome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
