Multimodal Large Language Models are Generalist Medical Image Interpreters
Open accessAbstract
Medicine is undergoing a transformation with the integration of Artificial Intelligence (AI). Traditional AI models, though clinically useful and often matching or surpassing expert clinicians in specific tasks, face a scalability challenge due to the necessity of developing individual models for each task. Therefore, there is a push towards foundation models that are applicable to a wider set of tasks. Our study showcases how non-domain-specific, publicly available vision-language models can be employed as general foundation models for medical applications. We test our paradigm across four medical disciplines - pathology, dermatology, ophthalmology, and radiology - focusing on two use-cases within each discipline. We find that our approach beats existing pre-training methods and is competitive to domain-specific foundation models that require vast amounts of domain-specific training images. We also find that large vision-language models are data efficient and do not require large annotated datasets to reach competitive performance. This allows for the development of new or improved AI models in areas of medicine where data is scarce and will accelerate medical progress towards true multimodal foundation models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Generalist models can inherently understand medical images and, in some medical contexts, even outperform specialized, task-specific pretraining methods.
“The results showed that generalist models can inherently understand medical images and, in some medical contexts, even outperform specialized, task-specific pretraining methods.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [88]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Tschandl, P., Rosendahl, C., & Kittler, H. (2018) · Scientific Data · doi:10.1038/sdata.2018.161 · Cited by 3,628 · see citing works
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148 · see citing works
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167 · see citing works
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · cited 1× · see citing works
“28 This feature of LLMs holds great promise for applications where large labeled datasets are not readily available.”
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · see citing works
Self-citationKather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.-A., Gaiser, T., Marx, A., Valous, N. A., Ferber, D., et al. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002730 · Cited by 1,082 · see citing works
Self-citationThe Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)Preprint
Yang, Z.-Y., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., & Wang, L.-J. (2023) · arXiv.org · doi:10.48550/arxiv.2309.17421 · Cited by 1,036 · see citing works
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · see citing works
A foundation model for generalizable disease detection from retinal images
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06555-x · Cited by 1,014 · see citing works
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.03368 · Cited by 984 · see citing works
A visual–language foundation model for pathology image analysis using medical twitter
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J., & Zou, J. (2023) · Nature Medicine · doi:10.1038/s41591-023-02504-3 · Cited by 693 · see citing works
PadChest: A large chest x-ray image dataset with multi-label annotated reports
Bustos, A., Pertusa, A., Salinas, J.-M., & de la Iglesia-Vayá, M. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101797 · Cited by 691 · see citing works
Lång, K., Josefsson, V., Larsson, A.-M., Larsson, S., Högberg, C., Sartor, H., Hofvind, S., Andersson, I., & Rosso, A. (2023) · The Lancet Oncology · doi:10.1016/s1470-2045(23)00298-x · Cited by 598 · see citing works
A patient-centric dataset of images and metadata for identifying melanomas using clinical context
Rotemberg, V. M., Kurtansky, N. R., Betz‐Stablein, B., Caffery, L. J., Chousakos, E., Codella, N., Combalia, M., Dusza, S. W., Guitera, P., Gutman, D. A., et al. (2021) · Scientific Data · doi:10.1038/s41597-021-00815-z · Cited by 558 · see citing works
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · see citing works
Vanguri, R. S., Luo, J., Aukerman, A. T., Egger, J. V., Fong, C. J., Horvat, N., Pagano, A., Araujo-Filho, J. D. A. B., Geneslaw, L., Rizvi, H., et al. (2022) · Nature Cancer · doi:10.1038/s43018-022-00416-8 · Cited by 395 · see citing works
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2021) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 311 · see citing works
PanNuke: An Open Pan-Cancer Histology Dataset for Nuclei Instance Segmentation and Classification
Gamper, J., Koohbanani, N. A., Benet, K., Khuram, A., & Rajpoot, N. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23937-4_2 · Cited by 289 · see citing works
Singh, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7475259 · Cited by 262 · see citing works
Recent advances in osteoarthritis imaging—the Osteoarthritis Initiative
Eckstein, F., Wirth, W., & Nevitt, M. C. (2012) · Nature Reviews Rheumatology · doi:10.1038/nrrheum.2012.113 · Cited by 254 · see citing works
Use of gpt-4 to diagnose complex clinical cases
Eriksen, A. V., Möller, S., & Ryg, J. (2024) · NEJM AI · doi:10.1056/aip2300031 · Cited by 200 · see citing works
The Multicenter Osteoarthritis Study: Opportunities for Rehabilitation Research
Segal, N. A., Nevitt, M. C., Gross, K. D., Hietpas, J., Glass, N. A., Lewis, C. E., & Torner, J. C. (2013) · PM&R · doi:10.1016/j.pmrj.2013.04.014 · Cited by 186 · see citing works
Med-Flamingo: a Multimodal Medical Few-shot LearnerPreprint
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Zakka, C., Dalmia, Y., Reis, E. P., Rajpurkar, P., & Leskovec, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.15189 · Cited by 111 · see citing works
A reinforcement learning model for AI-based decision support in skin cancer
Barata, C., Rotemberg, V. M., Codella, N., Tschandl, P., Rinner, C., Akay, B. N., Apalla, Z., Argenziano, G., Halpern, A. C., Lallas, A., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02475-5 · Cited by 109 · see citing works
Large language models should be used as scientific reasoning engines, not knowledge databases
Truhn, D., Reis-Filho, J. S., & Kather, J. N. (2023) · Nature Medicine · doi:10.1038/s41591-023-02594-z · Cited by 102 · see citing works
Self-citationA generalizable deep learning regression model for automated glaucoma screening from fundus images
Hemelings, R., Elen, B., Schuster, A. K.-G., Blaschko, M. B., Barbosa‐Breda, J., Hujanen, P., Junglas, A., Nickels, S., White, A. J. R., Pfeiffer, N., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00857-0 · Cited by 98 · see citing works
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16527 · Cited by 72 · see citing works
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022) · arXiv · doi:10.48550/arxiv.2210.13382 · Cited by 61 · see citing works
Language Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46 · see citing works
Image prediction of disease progression for osteoarthritis by style-based manifold extrapolation
Han, T., Kather, J. N., Pedersoli, F., Zimmermann, M. N., Keil, S., Schulze-Hagen, M. F., Terwoelbeck, M., Isfort, P., Haarburger, C., Kiessling, F. M., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00560-x · Cited by 29 · see citing works
Self-citationAIROGS: Artificial Intelligence for RObust Glaucoma Screening ChallengePreprint
de Vente, C., Vermeer, K. A., Jaccard, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.01738 · Cited by 19 · see citing works
Multimodal Foundation Models Exploit Text to Make Medical Image PredictionsPreprint
Buckley, T., Diao, J. A., Rajpurkar, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.05591 · Cited by 18 · see citing works
Han, T., Adams, L. C., Bressem, K. K., Busch, F., Huck, L. C., Nebelung, S., & Truhn, D. (2023) · medRxiv · doi:10.1101/2023.11.03.23297957 · Cited by 12 · see citing works
Self-citationHarnessing medical twitter data for pathology AI
Lu, M. Y., Chen, B., & Mahmood, F. (2023) · Nature Medicine · doi:10.1038/s41591-023-02530-1 · Cited by 4 · see citing works
Han, T., Žigutytė, L., Huck, L. C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.17123 · Cited by 1 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 2
- Alain, G. & Bengio, Y. Understanding intermediate layers using linear classifier probes. Preprint at https://doi.org/10.48550/arXiv.1610.01644 (2018). · DOI · Paper page
- Tiulpin, A., Melekhov, I. & Saarakkala, S. KNEEL: Knee Anatomical Landmark Localization Using Hourglass Networks. Preprint at https://doi.org/10.48550/arXiv.1907.12237 (2019). · DOI
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Previous work [88] evaluated the performance of a general-domain M-LLM in biomedical image classification tasks.”
Large language model for medical images: a survey of taxonomy, systematic review, and future trends
Wang, P., Lu, W., Lu, C., Zhou, R., Li, M., & Qin, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020090 · Cited by 21 · see citing works
- Authors
- Han, T., Adams, L. C., Nebelung, S., Kather, J. N., Bressem, K. K., & Truhn, D.
- Affiliations
- Universitätsklinikum Aachen (DE)
- Technical University of Munich (DE)
- University Hospital Heidelberg (DE)
- Deutsches Herzzentrum München (DE)
- Corresponding author
- Truhn, Daniel
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-12-22
- OpenAlex
- W4390115208
- Cited by
- 18
- References
- 44
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 6
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Funders
- European Commission (101079894, 875329, HORIZON2020, 101057091, 101112053)
- Deutscher Akademischer Austauschdienst (57616814)
- Bundesministerium für Bildung und Forschung (01KD2215A, 01EO2101, 01VSF21048, 57616814, 01KD2215B, 01KD2104C, 031L0312A)
- Bundesministerium für Gesundheit (ZMVI1-2520DAT111, 2520DAT111)
- Deutsche Krebshilfe (#70113864, 2520DAT111)
- Gemeinsame Bundesausschuss (01VSF21048)
- HORIZON EUROPE Framework Programme (101057091, 101079894)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal large language models, vision-language models, foundation models, medical image interpretation, medical image analysis, pathology, dermatology, ophthalmology, radiology, data efficiency, data scarcity, pre-training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
