MetaGP: A generative foundation model integrating electronic health records and multimodal imaging for addressing unmet clinical needs
Open accessAbstract
Artificial Intelligence (AI) has transformed specialized diagnostics, yet current models often lack the cross-disciplinary flexibility required for comprehensive clinical care, potentially limiting their effectiveness in real-world applications. Here, we introduce Meta General Practitioner (MetaGP) , a generative foundation model with 13 billion parameters trained on over ten million electronic health records (EHRs) and a wide range of medical literature and academic books, designed to provide accurate diagnostic support across a spectrum of clinical scenarios, including rare disease diagnosis, emergency condition identification and multimodal medical report generation. MetaGP demonstrated significant performance gains over GPT-4, achieving higher diagnostic accuracy and reducing the incidence of potentially harmful recommendations. In rare disease diagnoses, MetaGP achieved a mean diagnostic score of 1.57, notably exceeding GPT-4’s score of 0.93 and closely matching the performance of experienced clinicians. For emergency condition diagnoses, MetaGP achieved an average score of 1.59, outperforming GPT-4’s 1.19 and proving especially beneficial in supporting junior and mid-level clinicians by improving diagnostic accuracy by 53% and 46%, respectively. We rigorously evaluated MetaGP’s capabilities using a combination of human and automated assessments, benchmarking it against state-of-the-art models, including Med-Flamingo and LLaVA-Med, for chest X-ray (CXR) and computed tomography (CT) report generation. MetaGP consistently delivered high-quality reports, with radiologists rating the AI-generated outputs as comparable or superior to physician-authored reports in a significant proportion of cases. These findings highlight MetaGP’s potential as a reliable and versatile tool for clinical decision support, capable of addressing diagnostic challenges across diverse medical domains.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attention mechanisms such as gated attention, cross-attention, or transformer encoders are increasingly adopted in multimodal AI.
“attention mechanisms such as gated attention [40,41], cross-attention [42–45], or transformer encoders [20,36,46–49] are increasingly adopted.”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401 · cited 1× · see citing works
“33 After removing duplications, we obtained about 14.8 billion tokens.”
InfluentialScalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1× · see citing works
“For example, before applying AI to EHRs, one would normally need to convert the heterogeneous raw data into well-structured inputs, 5,7 which is not only labor-intensive but also prone to information loss.”
Self-citationLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationDisparities in dermatology AI performance on a diverse, curated clinical image set
Daneshjou, R., Vodrahalli, K., Novoa, R. A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S. M., Bailey, E. E., Gevaert, O., et al. (2022) · Science Advances · doi:10.1126/sciadv.abq6147 · Cited by 544 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · see citing works
Self-citationTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483 · see citing works
Self-citationConsiderations for addressing bias in artificial intelligence for health equity
Abràmoff, M. D., Tarver, M. E., Loyo-Berrios, N., Trujillo, S., Char, D., Obermeyer, Z., Eydelman, M. B., Foundational Principles of Ophthalmic Imaging and Algorithmic Interpretation Working Group of the Collaborative Community for Ophthalmic Imaging Foundation, W., & Maisel, W. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00913-9 · Cited by 349 · see citing works
Implications of predicting race variables from medical images
Zou, J., Gichoya, J. W., Ho, D. E., & Obermeyer, Z. (2023) · Science · doi:10.1126/science.adh4260 · Cited by 30 · see citing works
Flagged first, then by citations
Research works printed but not linked: 32
- [1]Liu Y. Jain A. Eng C. Way D.H. Lee K. Bui P. Kanada K. de Oliveira Marinho G. Gallegos J. Gabriele S. A deep learning system for differential diagnosis of skin diseases Nat. Med. 26 2020 900 908 32424212
- [3]Zhang K. Liu X. Xu J. Yuan J. Cai W. Chen T. Wang K. Gao Y. Nie S. Xu X. Deep-learning models for the detection and incidence prediction of chronic kidney disease and type 2 diabetes from retinal fundus images Nat. Biomed. Eng. 5 2021 533 545 34131321
- [4]Kermany D.S. Goldbaum M. Cai W. Valentim C.C.S. Liang H. Baxter S.L. McKeown A. Yang G. Wu X. Yan F. Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning Cell 172 2018 1122 1131.e9 29474911
- [5]Liang H. Tsui B.Y. Ni H. Valentim C.C.S. Baxter S.L. Liu G. Cai W. Kermany D.S. Sun X. Chen J. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence Nat. Med. 25 2019 433 438 30742121
- [6]Jiang L.Y. Liu X.C. Nejatian N.P. Nasir-Moin M. Wang D. Abidin A. Eaton K. Riina H.A. Laufer I. Punjabi P. Health system-scale language models are all-purpose prediction engines Nature 619 2023 357 362 37286606 PMC10338337
- [8]Hannun A.Y. Rajpurkar P. Haghpanahi M. Tison G.H. Bourn C. Turakhia M.P. Ng A.Y. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network Nat. Med. 25 2019 65 69 30617320 PMC6784839
- [9]Liu R. Rizzo S. Whipple S. Pal N. Pineda A.L. Lu M. Arnieri B. Lu Y. Capra W. Copping R. Zou J. Evaluating eligibility criteria of oncology trials using real-world data and AI Nature 592 2021 629 633 33828294 PMC9007176
- [10]Bommasani R. Hudson D.A. Adeli E. Altman R. Arora S. von Arx S. Bernstein M.S. Bohg J. Bosselut A. Brunskill E. On the Opportunities and Risks of Foundation Models Preprint at arXiv 2022 10.48550/arXiv.2108.07258 · DOI · Paper page
- [11]Brown T.B. Mann B. Ryder N. Subbiah M. Kaplan J.D. Dhariwal P. Neelakantan A. Shyam P. Sastry G. Askell A. Language Models are Few-Shot Learners Preprint at arXiv 2020 10.48550/arXiv.2005.14165 · DOI · Paper page
- [12]Singhal K. Azizi S. Tu T. Mahdavi S.S. Wei J. Chung H.W. Scales N. Tanwani A. Cole-Lewis H. Pfohl S. Large language models encode clinical knowledge Nature 620 2023 172 180 37438534 PMC10396962
- [13]Zhou H. Liu F. Gu B. Zou X. Huang J. Wu J. Li Y. Chen S.S. Zhou P. Liu J. A survey of large language models in medicine: Progress, application, and challenge Preprint at arXiv 2023 10.48550/arXiv.2311.05112 · DOI · Paper page
- [14]Wu C. Lin W. Zhang X. Zhang Y. Xie W. Wang Y. PMC-LLaMA: toward building open-source language models for medicine J. Am. Med. Inform. Assoc. 31 2024 1833 1843 38613821 PMC11639126
- [15]Zhang K. Zhou R. Adhikarla E. Yan Z. Liu Y. Yu J. Liu Z. Chen X. Davison B.D. Ren H. A generalist vision-language foundation model for diverse biomedical tasks Nat. Med. 30 2024 3129 3141 39112796 PMC12581140
- [16]Peng C. Yang X. Chen A. Smith K.E. PourNejatian N. Costa A.B. Martin C. Flores M.G. Zhang Y. Magoc T. A study of generative large language model for medical research and healthcare NPJ Digit. Med. 6 2023 210 37973919 PMC10654385
- [18]Moor M. Huang Q. Wu S. Yasunaga M. Dalmia Y. Leskovec J. Zakka C. Reis E.P. Rajpurkar P. Med-Flamingo: a Multimodal Medical Few-shot Learner Preprint at arXiv 2023 10.48550/arXiv.2307.15189 · DOI
- [19]Tiu E. Talius E. Patel P. Langlotz C.P. Ng A.Y. Rajpurkar P. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning Nat. Biomed. Eng. 6 2022 1399 1406 36109605 PMC9792370
- [20]Huang W. Li C. Zhou H.Y. Yang H. Liu J. Liang Y. Zheng H. Zhang S. Wang S. Enhancing representation in radiography-reports foundation model: a granular alignment algorithm using masked contrastive learning Nat. Commun. 15 2024 7620 39223122 PMC11369198
- [21]Yang A. Yang B. Zhang B. Hui B. Zheng B. Yu B. Li C. Liu D. Huang F. Wei H. Qwen2 technical report Preprint at arXiv 2024 10.48550/arXiv.2407.10671 · DOI · Paper page
- [22]Perry T.S. Move over, moore's law. Make way for huang's law [Spectral lines] IEEE Spectr. 55 2018 7
- [30]Zhang K. Liu X. Shen J. Li Z. Sang Y. Wu X. Zha Y. Liang W. Wang C. Wang K. Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography Cell 182 2020 1360 32888496 PMC7470724
- [31]Wang G. Liu X. Shen J. Wang C. Li Z. Ye L. Wu X. Chen T. Wang K. Zhang X. A deep-learning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest X-ray images Nat. Biomed. Eng. 5 2021 509 521 33859385 PMC7611049
- [32]Zhou H.-Y. Yu Y. Wang C. Zhang S. Gao Y. Pan J. Shao J. Lu G. Zhang K. Li W. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics Nat. Biomed. Eng. 7 2023 743 755 37308585
- [34]Gao L. Biderman S. Black S. Golding L. Hoppe T. Foster C. Phang J. He H. Thite A. Nabeshima N. The Pile: An 800GB Dataset of Diverse Text for Language Modeling Preprint at arXiv 2020 10.48550/arXiv.2101.00027 · DOI · Paper page
- [35]Jin Q. Dhingra B. Liu Z. Cohen W.W. Lu X. Pubmedqa: A dataset for biomedical research question answering Preprint at arXiv 2019 10.48550/arXiv.1909.06146 · DOI
- [37]Pal A. Umapathi L.K. Sankarasubbu M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering Conference on health, inference, and learning 2022 PMLR 248 260
- [38]Köpf A. Kilcher Y. Von Rütte D. Anagnostidis S. Tam Z.R. Stevens K. Barhoum A. Nguyen D. Stanley O. Nagyfi R. OpenAssistant Conversations -- Democratizing Large Language Model Alignment Preprint at arXiv 2023 10.48550/arXiv.2304.07327 · DOI · Paper page
- [39]Kim S. Joo S.J. Kim D. Jang J. Ye S. Shin J. Seo M. The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning Preprint at arXiv 2023 10.48550/arXiv.2305.14045 · DOI
- [41]Wei J. Bosma M. Zhao V.Y. Guu K. Yu A.W. Lester B. Du N. Dai A.M. Le Q.V. Finetuned Language Models Are Zero-Shot Learners Preprint at arXiv 2022 10.48550/arXiv.2109.01652 · DOI · Paper page
- [42]Liu Z. Lin Y. Cao Y. Hu H. Wei Y. Zhang Z. Lin S. Guo B. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 2021 9992 10002 10.1109/ICCV48922.2021.00986 · DOI · Paper page
- [44]Chen Z. Agarwal D. Aggarwal K. Safta W. Balan M.M. Brown K. Masked image modeling advances 3d medical image analysis 2023 IEEE/CVF Winter Conference on Applications of Computer Vision 2023 10.1109/WACV56688.2023.00201 · DOI
- [45]Loshchilov I. Hutter F. Decoupled Weight Decay Regularization Preprint at arXiv 2019 10.48550/arXiv.1711.05101 · DOI · Paper page
- [47]Ba J.L. Kiros J.R. Hinton G.E. Layer Normalization Preprint at arXiv 2016 10.48550/arXiv.1607.06450 · DOI
Cited by
Sorted by citations
Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review
Yang, M., Shi, N., Chen, H., Po, E. Y. P., Ng, P. Y., Lai, P. C. K., Wu, Y.-C., & Fong, D. Y. T. (2026) · Health Data Science · doi:10.34133/hds.0356 · Cited by 3 · see citing works
“5 [ 46 ] and Google-T5 [ 90 ], have also been leveraged in clinical multimodal studies.”
Influential
- Authors
- Liu, F., Zhou, H., Wang, K., Yu, Y., Gao, Y., Sun, Z., Liu, S., Sun, S., Zou, Z., Li, Z., Li, B., Miao, H., Liu, Y., Hou, T., Fok, M., Patil, N. G., Xue, K., Li, T., Oermann, E., Yin, Y., Duan, L., Qu, J., Huang, X., Jin, S., & Zhang, K.
- Affiliations
- Macau University of Science and Technology (MO)
- Peking University (CN)
- Wenzhou Medical University (CN)
- Southern Medical University (CN)
- Hospital Conde S. Januário (CN)
- University of Oxford (GB)
- NYU Langone Health (US)
- City University of Macau (MO)
- Chinese PLA General Hospital (CN)
- Corresponding author
- Duan, Lian
- Qu, Jia
- Huang, Xiaoying
- Jin, Shengwei
- Zhang, Kang
- ORCID
- Venue
- Cell Reports Medicine
- Publisher
- Elsevier BV
- Published
- 2025-04
- Volume
- 6
- Issue
- 4
- Pages
- 102056
- ISSN
- 2666-3791 (ISSN-L)
- PMC
- PMC12047458
- OpenAlex
- W4409199153
- Cited by
- 22
- References
- 15
- Citations by year
- 2026: 16 · 2025: 6
- FWCI
- 5.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Cell Reports Medicine (publishedVersion, cc-by-nc-nd)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Beijing Nova Program (20240484627)
- China Postdoctoral Science Foundation (2023T160061)
- National Natural Science Foundation of China (32470964, 32100631, W2431057)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (0003/2021/AKP, 0007/2020/AFJ, 0 070/2020/A2)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- generative foundation models, electronic health records, multimodal medical imaging, rare disease diagnosis, medical report generation, chest radiography, computed tomography, clinical decision support, large language models in medicine
- MeSH terms
- Artificial Intelligence, Humans, Tomography, X-Ray Computed, Electronic Health Records, Multimodal Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
