A Vision-Language Foundation Model for Precision Oncology
Open accessAbstract
Clinical decision-making is driven by multimodal data, including clinical notes and pathological characteristics. Artificial intelligence approaches that can effectively integrate multimodal data hold significant promise in advancing clinical care 1,2. However, the scarcity of well-annotated multimodal datasets in clinical settings has hindered the development of useful models. In this study, we developed the Multimodal transformer with Unified maSKed modeling (MUSK), a vision-language foundation model designed to leverage large-scale, unlabelled, unpaired image and text data. MUSK was pretrained on 50 million pathology images from 11,577 patients and one billion pathology-related text tokens using unified masked modelling. It was further pretrained on one million pathology image-text pairs to efficiently align the vision and language features. With minimal or no further training, MUSK was tested in a wide range of applications and demonstrated superior performance across 23 patch-level and slide-level benchmarks, including image-to-text and text-to-image retrieval, visual question answering, image classification and molecular biomarker prediction. Furthermore, MUSK showed strong performance in outcome prediction, including melanoma relapse prediction, pan-cancer prognosis prediction and immunotherapy response prediction in lung and gastro-oesophageal cancers. MUSK effectively combined complementary information from pathology images and clinical reports and could potentially improve diagnosis and precision in cancer therapy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MUSK, a vision-language system, pretrains on large, largely unpaired corpora of pathology image patches and clinical text, then aligns image-text pairs for downstream tasks.
“The Multimodal transformer with Unified maSKed modeling (MUSK) from Stanford exemplifies this approach as a vision-language system that pretrains on large, largely unpaired corpora of pathology image patches and clinical text, then aligns image–text pairs for downstream tasks [32].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R. L., Torre, L. A., & Jemal, A. (2018) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21492 · Cited by 84,109 · see citing works
Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R. L., Soerjomataram, I., & Jemal, A. (2024) · CA: a cancer journal for clinicians · doi:10.3322/caac.21834 · Cited by 27,064 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092 · see citing works
Pembrolizumab plus chemotherapy in metastatic non–small-cell lung cancer
Gandhi, L., Rodríguez-Abreu, D., Gadgeel, S., Esteban, E., Felip, E., De Angelis, F., Domine, M., Clingan, P., Hochmair, M. J., Powell, S. F., et al. (2018) · New England Journal of Medicine · doi:10.1056/nejmoa1801005 · Cited by 6,607 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Tumor mutational load predicts survival after immunotherapy across multiple cancer types
Samstein, R. M., Lee, C.-H., Shoushtari, A. N., Hellmann, M. D., Shen, R., Janjigian, Y. Y., Barron, D. A., Zehir, A., Jordan, E. J., Omuro, A., et al. (2019) · Nature Genetics · doi:10.1038/s41588-018-0312-8 · Cited by 4,071 · see citing works
Mok, T. S. K., Wu, Y.-L. L. C., Kudaba, I., Kowalski, D. M., Cho, B. C., Turna, H. Z., de Castro, G., Srimuninnimit, V., Лактионов, К. К., Bondarenko, I., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(18)32409-7 · Cited by 3,319 · see citing works
Nivolumab plus ipilimumab in advanced non-small-cell lung cancer
Hellmann, M. D., Paz-Ares, L., Bernabe Caro, R., Zurawski, B., Kim, S.-W., Carcereny Costa, E., Park, K., Alexandru, A., Lupinacci, L., de la Mora Jimenez, E., et al. (2019) · New England Journal of Medicine · doi:10.1056/nejmoa1910231 · Cited by 2,823 · see citing works
CA: A Cancer Journal for Clinicians
American Cancer Society (2010) · CA A Cancer Journal for Clinicians · doi:10.3322/(issn)1542-4863 · Cited by 2,538 · see citing works
Pan-tumor genomic biomarkers for PD-1 checkpoint blockade–based immunotherapy
Cristescu, R., Mogg, R., Ayers, M., Albright, A., Murphy, E., Yearley, J., Sher, X., Liu, X. Q., Lu, H., Nebozhyn, M., et al. (2018) · Science · doi:10.1126/science.aar3593 · Cited by 2,301 · see citing works
Data-efficient and weakly supervised computational pathology on whole-slide images
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00682-w · Cited by 2,282 · see citing works
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · cited 1× · see citing works
“Foundation models represent a new frontier in medical AI research and development 6,7 .”
Atezolizumab for First-Line Treatment of PD-L1–Selected Patients with NSCLC
Herbst, R. S., Giaccone, G., de Marinis, F., Reinmuth, N., Vergnenégre, A., Barrios, C. H., Morise, M., Felip, E., Andric, Z. G., Geater, S. L., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa1917346 · Cited by 1,667 · see citing works
The immune contexture and Immunoscore in cancer prognosis and therapeutic efficacy
Bruni, D., Angell, H. K., & Galon, J. (2020) · Nature reviews. Cancer · doi:10.1038/s41568-020-0285-7 · Cited by 1,586 · see citing works
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · cited 2× · see citing works
“2a and Supplementary Tables 1 and 2).”
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · see citing works
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · see citing works
Conserved pan-cancer microenvironment subtypes predict response to immunotherapy
Bagaev, A., Kotlov, N., Nomie, K., Svekolkin, V., Gafurov, A., Isaeva, O., Osokin, N., Kozlov, I., Frenkel, F., Gancharova, O., et al. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.04.014 · Cited by 1,302 · see citing works
Kather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.-A., Gaiser, T., Marx, A., Valous, N. A., Ferber, D., et al. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002730 · Cited by 1,082 · see citing works
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · see citing works
A foundation model for generalizable disease detection from retinal images
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06555-x · Cited by 1,014 · see citing works
Immune-checkpoint inhibitors: long-term implications of toxicity
Johnson, D. B., Nebhan, C. A., Moslehi, J. J., & Balko, J. M. (2022) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-022-00600-w · Cited by 982 · see citing works
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966 · cited 15× · see citing works
“A key feature of a foundation model is its ability to perform downstream tasks without further training, that is, zero-shot learning capability 15,16 .”
InfluentialA whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921 · see citing works
Artificial Intelligence in Cancer Research and Precision Medicine
Bhinder, B., Gilvary, C., Madhukar, N. S., & Elemento, O. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0090 · Cited by 781 · see citing works
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777 · see citing works
Transformer-based unsupervised contrastive learning for histopathological image classification
Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., & Han, X. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102559 · Cited by 751 · see citing works
Self-citationA visual–language foundation model for pathology image analysis using medical twitter
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J., & Zou, J. (2023) · Nature Medicine · doi:10.1038/s41591-023-02504-3 · Cited by 693 · see citing works
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662 · see citing works
Pan-cancer image-based detection of clinically actionable genetic alterations
Kather, J. N., Heij, L. R., Grabsch, H. I., Loeffler, C. M. L., Echle, A., Muti, H. S., Krause, J., Niehues, J., Sommer, K. A. J., Bankhead, P., et al. (2020) · Nature Cancer · doi:10.1038/s43018-020-0087-6 · Cited by 660 · see citing works
Multi-omic machine learning predictor of breast cancer therapy response
Sammut, S.-J., Crispin-Ortuzar, M., Chin, S.-F., Provenzano, E., Bardwell, H. A., Ma, W., Cope, W., Dariush, A., Dawson, S.-J., Abraham, J. E., et al. (2022) · Nature · doi:10.1038/s41586-021-04278-5 · Cited by 654 · see citing works
A pathology foundation model for cancer diagnosis and prognosis prediction
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07894-z · Cited by 580 · see citing works
Self-citationA foundation model for clinical-grade computational pathology and rare cancers detection
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03141-0 · Cited by 572 · see citing works
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561 · see citing works
A Multi-Organ Nucleus Segmentation Challenge
Kumar, N., Verma, R., Anand, D., Zhou, Y., Onder, O. F., Tsougenis, E., Chen, H., Heng, P., Li, J., Hu, Z., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2947628 · Cited by 559 · see citing works
Self-citationThompson, A. K., Kelley, B. F., Prokop, L. J., Murad, M. H., & Baum, C. L. (2016) · JAMA Dermatology · doi:10.1001/jamadermatol.2015.4994 · Cited by 521 · see citing works
Rotation Equivariant CNNs for Digital Pathology
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., & Welling, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00934-2_24 · Cited by 476 · see citing works
A multimodal generative AI copilot for human pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. (2024) · Nature · doi:10.1038/s41586-024-07618-3 · Cited by 461 · see citing works
Vanguri, R. S., Luo, J., Aukerman, A. T., Egger, J. V., Fong, C. J., Horvat, N., Pagano, A., Araujo-Filho, J. D. A. B., Geneslaw, L., Rizvi, H., et al. (2022) · Nature Cancer · doi:10.1038/s43018-022-00416-8 · Cited by 395 · see citing works
Self-citationVLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-ExpertsPreprint
Bao, H., Wang, W., Li, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02358 · Cited by 281 · see citing works
Lung and Colon Cancer Histopathological Image Dataset (LC25000)Preprint
Borkowski, A., Bui, M. M., Thomas, L. B., Wilson, C. P., Deland, L. A., & Mastorides, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12142 · Cited by 261 · see citing works
Vision–language foundation model for echocardiogram interpretation
Christensen, M., Vukadinovic, M., Yuan, N., & Ouyang, D. (2024) · Nature Medicine · doi:10.1038/s41591-024-02959-y · Cited by 231 · see citing works
Overcoming Data Limitation in Medical Visual Question Answering
Nguyen, B. D., Do, T.-T., Nguyen, B. X., Do, T., Tjiputra, E., & Tran, Q. D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_57 · Cited by 220 · see citing works
BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images
Brancati, N., Anniciello, A. M., Pati, P., Riccio, D., Scognamiglio, G., Jaume, G., Pietro, G. D., Bonito, M. D., Foncubierta–Rodríguez, A., Botti, G., et al. (2022) · Database · doi:10.1093/database/baac093 · Cited by 189 · see citing works
Benchmarking Self-Supervised Learning on Diverse Pathology Datasets
Kang, M., Song, H., Park, S., Yoo, D., & Pereira, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00326 · Cited by 187 · see citing works
Silva-Rodríguez, J., Colomer, A., Sales, M. A., et al. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105637 · Cited by 167 · see citing works
Han, C., Lin, J., Mai, J., Wang, Y., Zhang, Q., Zhao, B., Chen, X., Pan, X., Shi, Z., Xu, Z., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102487 · Cited by 153 · see citing works
Arunachalam, H. B., Mishra, R., Daescu, O., Cederberg, K., Rakheja, D., Sengupta, A. L., Leonard, D., Hallac, R. R., & Leavey, P. J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210706 · Cited by 146 · see citing works
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.03206 · Cited by 140 · see citing works
Transparent medical image AI via an image–text foundation model grounded in medical literature
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, J. A., Cai, Z. R., Daneshjou, R., & Lee, S.-I. (2024) · Nature Medicine · doi:10.1038/s41591-024-02887-x · Cited by 125 · see citing works
The State of Melanoma: Emergent Challenges and Opportunities
Atkins, M. B., Curiel‐Lewandrowski, C. N., Fisher, D. E., Swetter, S. M., Tsao, H., Aguirre‐Ghiso, J. A., Soengas, M. S., Weeraratna, A. T., Flaherty, K. T., Herlyn, M., et al. (2021) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-20-4092 · Cited by 113 · see citing works
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R. P. N., Wei, M., Valluri, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.00915 · Cited by 102 · see citing works
Xu, F., Zhu, C., Tang, W., Wang, Y., Zhang, Y., Li, J., Jiang, H., Shi, Z., Liu, J., & Jin, M. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.759007 · Cited by 80 · see citing works
Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles
Gamper, J., & Rajpoot, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01628 · Cited by 75 · see citing works
PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology
Sun, Y., Zhu, C., Zheng, S., Zhang, K., Sun, L., Shui, Z., Zhang, Y., Li, H., & Yang, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28308 · Cited by 74 · see citing works
The Digital Brain Tumour Atlas, an open histopathology resource
Roetzer-Pejrimovsky, T., Moser, A., Atli, B., Vogel, C., Mercea, P. A., Prihoda, R., Gelpí, E., Haberler, C., Höftberger, R., Hainfellner, J. A., et al. (2022) · Scientific Data · doi:10.1038/s41597-022-01157-0 · Cited by 70 · see citing works
Shen, Y., Luo, Y., Shen, D. G., & Ke, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16434-7_21 · Cited by 58 · see citing works
Quilt-1M: One Million Image-Text Pairs for Histopathology
Ikezogwo, W. O., Seyfioglu, M. S., Ghezloo, F., Geva, D. S. C., Mohammed, F. S., Anand, P. K., Krishna, R., & Shapiro, L. (2023) · arXiv · doi:10.48550/arxiv.2306.11207 · Cited by 55 · see citing works
Pathological Visual Question AnsweringPreprint
He, X., Cai, Z., Wei, W., Zhang, Y., Mou, L., Xing, E. P., & Xie, P. (2020) · doi:10.36227/techrxiv.13127537.v1 · Cited by 54 · see citing works
Barbano, C. A., Perlo, D., Tartaglione, E., Fiandrotti, A., Bertero, L., Cassoni, P., & Grangetto, M. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506198 · Cited by 52 · see citing works
Kriegsmann, K., Lobers, F., Zgorzelski, C., Kriegsmann, J., Janßen, C., Meliß, R. R., Muley, T., Sack, U., Steinbuss, G., & Kriegsmann, M. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.1022967 · Cited by 50 · see citing works
TCGA-Reports: A machine-readable pathology report resource for benchmarking text-based AI models
Kefeli, J., & Tatonetti, N. P. (2024) · Patterns · doi:10.1016/j.patter.2024.100933 · Cited by 45 · see citing works
The Stanford Medicine data science ecosystem for clinical and translational research
Callahan, A., Ashley, E. A., Datta, S., Desai, P., Ferris, T., Fries, J., Halaas, M., Langlotz, C. P., Mackey, S., Posada, J., et al. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad054 · Cited by 35 · see citing works
K-PathVQA: Knowledge-Aware Multimodal Representation for Pathology Visual Question Answering
Naseem, U., Khushi, M., Dunn, A. G., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3294249 · Cited by 27 · see citing works
Brummer, O., Pölönen, P., Mustjoki, S. M., & Brück, O. E. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.15.503955 · Cited by 4 · see citing works
PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in PathologyPreprint
Sun, Y., Wu, H., Zhu, C., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.16355 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- [24]Kim J-H, Jun J & Zhang B-T Bilinear attention networks. Advances in neural information processing systems 31 (2018).
- [26]Li LH, Yatskar M, Yin D, Hsieh C-J & Chang K-W VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019). · arXiv
- [28]He X, Zhang Y, Mou L, Xing E & Xie P PathVQA: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 (2020). · arXiv
- [59]Shazeer N Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017). · arXiv
- [63]Li J, Li D, Xiong C & Hoi SCH BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (2022).
- [64]Devlin J, Chang M-W, Lee K & Toutanova K BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (2019).
- [65]Ramesh A Zero-shot text-to-image generation. In International conference on machine learning, 8821– 8831 (Pmlr, 2021).
- [66]Peng Z, Dong L, Bao H, Ye Q & Wei F BEiT v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366 (2022). · arXiv
- [70]Loshchilov I & Hutter F Decoupled weight decay regularization. In International Conference on Learning Representations, 1– 18 (2019).
- [71]Ilse M, Tomczak J & Welling M Attention-based deep multiple instance learning. In International conference on machine learning, 2127– 2136 (PMLR, 2018).
- [74]Achiam J GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
Cited by
Sorted by citations
Hou, J., Zhang, R., Xie, Y., Li, C., & Qin, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02257-y · Cited by 38 · see citing works
Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery
Kuno, M., Osumi, H., Udagawa, S., Yoshikawa, K., Ooki, A., Shinozaki, E., Ishikawa, T., Oba, J., Yamaguchi, K., & Sakurada, K. (2025) · Current Oncology · doi:10.3390/curroncol32110588 · Cited by 7 · see citing works
Image Analysis: Understanding and Mitigating Batch Effects in Histopathology
Gindra, R. H., Boushehri, S. S., Wagner, S. J., Tran, M., Winter, D. J. E., Schnabel, J. A., Saur, D., Marr, C., & Peng, T. (2025) · Trillium Pathology · doi:10.47184/tp.2025.01.03 · Cited by 2 · see citing works
- Authors
- Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., Li, Y., Bergstrom, C., Gopaulchan, M., Kim, T., Yu, K.-H., Willens, S., Olguin, F. M., Nirschl, J. J., Neal, J., Diehn, M., Yang, S., & Li, R.
- Affiliations
- Stanford Medicine (US)
- Harvard University (US)
- Corresponding author
- Yang, Sen
- Li, Ruijiang
- ORCID
- Venue
- Nature
- Published
- 2025-01-08
- Volume
- 638
- Issue
- 8051
- Pages
- 769-778
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC12295649
- OpenAlex
- W4406168947
- Cited by
- 256
- References
- 67
- Citations by year
- 2026: 173 · 2025: 83
- FWCI
- 102.32
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Licence
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Heart, Lung, and Blood Institute (R01HL174679)
- National Cancer Institute (R01CA222512, R01CA233578, R01CA285456, R01CA290715, R01CA269559)
- National Institute of General Medical Sciences (R35GM142879)
- National Institute of Dental and Craniofacial Research (R01DE030894)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- precision oncology, multimodal learning, pathology images, clinical notes, image-to-text retrieval, visual question answering, immunotherapy response prediction, lung cancer, gastroesophageal cancer
- MeSH terms
- Clinical Decision-Making, Artificial Intelligence, Humans, Immunotherapy, Lung Neoplasms, Medical Oncology, Melanoma, Neoplasms, Prognosis, Precision Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
