The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index, and 1 are not checked yet.
A visual–language foundation model for pathology image analysis using medical Twitter
Abstract
The lack of annotated publicly available medical images is a major barrier for computational research and education innovations. At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter. Here we harness these crowd platforms to curate OpenPath, a large dataset of 208,414 pathology images paired with natural language descriptions. We demonstrate the value of this resource by developing pathology language-image pretraining (PLIP), a multimodal artificial intelligence with both image and text understanding, which is trained on OpenPath. PLIP achieves state-of-the-art performances for classifying new pathology images across four external datasets: for zero-shot classification, PLIP achieves F1 scores of 0.565-0.832 compared to F1 scores of 0.030-0.481 for previous contrastive language-image pretrained model. Training a simple supervised classifier on top of PLIP embeddings also achieves 2.5% improvement in F1 scores compared to using other supervised model embeddings. Moreover, PLIP enables users to retrieve similar cases by either image or natural language search, greatly facilitating knowledge sharing. Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to develop medical artificial intelligence for enhancing diagnosis, knowledge sharing and education.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“AI-driven medical imaging analysis has emerged as a pivotal tool in modern healthcare, significantly improving the accuracy and efficiency of diagnostic processes.66,74,76,77,78,79,80,81,82,83”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [76]
In the citing paper
“Contrastive vision language models in pathology, such as PLIP12 and CONCH13, align text and images at the patch-level.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [12]
In the citing paper
“They are zero-shot learners, i.e., they can solve downstream tasks for which they have not been trained (e.g., cancer subtyping).”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [12]
In the citing paper
“PLIP [194] and CONCH [195] were pretrained on paired pathology images and text descriptions using contrastive learning methods.”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [194]
In the citing paper
“Several studies have developed visual–language foundation models using paired histopathology images and captions11,12.”
From A visual–omics foundation model to bridge histopathology with spatial transcriptomics (2025) · cites this paper as [12]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799
Image analysis and machine learning in digital pathology: challenges and opportunities
Madabhushi, A., & Lee, G. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.06.037 · Cited by 1,093
Kather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.-A., Gaiser, T., Marx, A., Valous, N. A., Ferber, D., et al. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002730 · Cited by 1,082
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
Research works printed but not linked: 2
- Palatucci, M., Pomerleau, D., Hinton, G. & Mitchell, T. M. Zero-shot learning with semantic output codes. In Proc. 22nd International Conference on Neural Information Processing Systems (eds Bengio, Y. et al.) 1410–1418 (Curran Associates, 2009).
- Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
Cited by
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966
A whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921
“Prior work on pathology vision–language modelling tends to focus on tile-level alignment of pathology images and text, as their studies were limited by the sources of image–text pairs (textbook examples 7 or Twitter data 8 ).”
InfluentialA pathology foundation model for cancer diagnosis and prognosis prediction
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07894-z · Cited by 580
A foundation model for clinical-grade computational pathology and rare cancers detection
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03141-0 · Cited by 572
“…large datasets is even greater for applications with inadequate quantities of data to develop bespoke models, as is the case for the detection of uncommon or rare tumor types, as well as for less common diagnostic tasks such as the prediction of specific genomic alterations, clinical outcomes…”
Metadata
- Authors
- Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J., & Zou, J.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Corresponding author
- Zou, James
- Venue
- Nature Medicine
- Published
- 2023-08-17
- Volume
- 29
- Issue
- 9
- Pages
- 2307-2316
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4385948838
- Cited by
- 693
- References
- 44
- Citations by year
- 2026: 221 · 2025: 293 · 2024: 169 · 2023: 10
- FWCI
- 62.90
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- pathology image analysis, multimodal pre-training, zero-shot classification, similar case retrieval, medical knowledge sharing
- MeSH terms
- Artificial Intelligence, Humans, Image Processing, Computer-Assisted, Language, Knowledge, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A visual–language foundation model for pathology image analysis using medical Twitter. https://app.cochat.ai/papers/10.1038/s41591-023-02504-3
- The paper
- Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J., & Zou, J. (2023). A visual–language foundation model for pathology image analysis using medical Twitter. Nature Medicine. https://doi.org/10.1038/s41591-023-02504-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.