Large language models leverage external knowledge to extend clinical insight beyond language boundaries
Open accessAbstract
Objectives: Large Language Models (LLMs) such as ChatGPT and Med-PaLM have excelled in various medical question-answering tasks. However, these English-centric models encounter challenges in non-English clinical settings, primarily due to limited clinical knowledge in respective languages, a consequence of imbalanced training corpora. We systematically evaluate LLMs in the Chinese medical context and develop a novel in-context learning framework to enhance their performance. Materials and Methods: The latest China National Medical Licensing Examination (CNMLE-2022) served as the benchmark. We collected 53 medical books and 381 149 medical questions to construct the medical knowledge base and question bank. The proposed Knowledge and Few-shot Enhancement In-context Learning (KFE) framework leverages the in-context learning ability of LLMs to integrate diverse external clinical knowledge sources. We evaluated KFE with ChatGPT (GPT-3.5), GPT-4, Baichuan2-7B, Baichuan2-13B, and QWEN-72B in CNMLE-2022 and further investigated the effectiveness of different pathways for incorporating LLMs with medical knowledge from 7 distinct perspectives. Results: Directly applying ChatGPT failed to qualify for the CNMLE-2022 at a score of 51. Cooperated with the KFE framework, the LLMs with varying sizes yielded consistent and significant improvements. The ChatGPT’s performance surged to 70.04 and GPT-4 achieved the highest score of 82.59. This surpasses the qualification threshold (60) and exceeds the average human score of 68.70, affirming the effectiveness and robustness of the framework. It also enabled a smaller Baichuan2-13B to pass the examination, showcasing the great potential in low-resource settings. Discussion and Conclusion: This study shed light on the optimal practices to enhance the capabilities of LLMs in non-English medical scenarios. By synergizing medical knowledge through in-context learning, LLMs can extend clinical insight beyond language barriers in healthcare, significantly reducing language-related disparities of LLM applications and ensuring global benefit in this field.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Advanced LLMs often rely on general domain datasets and lack exposure to real-world EHR notes, which limits their effectiveness in clinical settings.
“Furthermore, due to data access and privacy concerns34, advanced LLMs often rely on general domain datasets and lack exposure to real-world EHR notes, which limits their effectiveness in clinical settings24,36.”
From Probabilistic medical predictions of large language models (2024) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
Self-citationBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
Complexity-Based Prompting for Multi-Step ReasoningPreprint
Fu, Y., Peng, H.-C., Sabharwal, A., Clark, P., & Khot, T. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.00720 · Cited by 670 · see citing works
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
Self-citationIs ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.85 · Cited by 519 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Bhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460 · see citing works
Health system-scale language models are all-purpose prediction engines
Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06160-y · Cited by 460 · see citing works
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
Creation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · see citing works
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · see citing works
Self-citationUsing AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397 · see citing works
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 1× · see citing works
“11 The insufficiency of data and training impedes LLMs in understanding the intricate concepts 29 and encoding the necessary clinical expertise in non-English environment, 30 which is essential for accurate clinical inference.”
BackgroundBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Self-citationLim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 316 · see citing works
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · see citing works
Korngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259 · see citing works
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 206 · see citing works
ChatGPT and antimicrobial advice: the end of the consulting infection doctor?
Howard, A., Hope, W., & Gerada, A. (2023) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(23)00113-5 · Cited by 202 · see citing works
Unsupervised Commonsense Question Answering with Self-Talk
Shwartz, V., West, P., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.373 · Cited by 186 · see citing works
Chatbot vs medical student performance on free-response clinical reasoning examinations.
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2909 · Cited by 169 · see citing works
ChatGPT Performs on the Chinese National Medical Licensing Examination
Wang, X., Gong, Z., Wang, G., Jia, J., Xu, Y., Zhao, J., Fan, Q., Wu, S., Hu, W., & Li, X. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01961-0 · Cited by 162 · see citing works
ChatGPT: promise and challenges for deployment in low- and middle-income countries
Wang, X., Sanders, H. M., Liu, Y., Seang, K., Tran, B. X., Atanasov, A. G., Qiu, Y., Tang, S., Car, J., Wang, Y. X., et al. (2023) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2023.100905 · Cited by 143 · see citing works
Considering Biased Data as Informative Artifacts in AI-Assisted Health Care
Ferryman, K. S., Mackintosh, M. E., & Ghassemi, M. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2214964 · Cited by 142 · see citing works
Baichuan 2: Open Large-scale Language Models
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.10305 · Cited by 125 · see citing works
Self-citationDeep transfer learning for reducing health care disparities arising from biomedical data inequality
Gao, Y., & Cui, Y. (2020) · Nature Communications · doi:10.1038/s41467-020-18918-3 · Cited by 122 · see citing works
Pre-trained language model representations for language generation
Edunov, S., Baevski, A., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1409 · Cited by 122 · see citing works
Extrapolating Large Language Models to Non-English by Aligning LanguagesPreprint
Zhu, W., Lv, Y., Dong, Q., Yuan, F., Xu, J.-J., Huang, S., Kong, L., Chen, J.-J., & Li, L. (2023) · arXiv.org · doi:10.48550/arxiv.2308.04948 · Cited by 107 · see citing works
Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing ExaminationsPreprint
Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y., & Radev, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.18027 · Cited by 91 · see citing works
The importance of resource awareness in artificial intelligence for healthcare
Jia, Z., Chen, J., Xu, X., Kheir, J. N., Hu, J., Xiao, H., Peng, S., Hu, X. S., Chen, D., & Shi, Y. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00670-0 · Cited by 85 · see citing works
Large Language Models Answer Medical Questions Accurately, but Can’t Match Clinicians’ Knowledge
Harris, E. (2023) · JAMA · doi:10.1001/jama.2023.14311 · Cited by 70 · see citing works
All Languages Matter: On the Multilingual Safety of Large Language ModelsPreprint
Wang, W., Tu, Z., Chen, C., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.00905 · Cited by 61 · see citing works
Assessing Biases in Medical Decisions via Clinician and AI Chatbot Responses to Patient Vignettes
Kim, J. Y., Cai, Z. R., Chen, M. L., Simard, J. F., & Linos, E. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.38050 · Cited by 58 · see citing works
Nayak, A., Alkaitis, M. S., Nayak, K., Nikolov, M. C., Weinfurt, K. P., & Schulman, K. A. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2561 · Cited by 57 · see citing works
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S., Lee, S., & Kim, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.155 · Cited by 55 · see citing works
Master clinical medical knowledge at certificated-doctor-level with deep learning model
Wu, J. Y., Liu, X., Zhang, X., He, Z., & Lv, P. (2018) · Nature Communications · doi:10.1038/s41467-018-06799-6 · Cited by 52 · see citing works
Self-citationLiu, X., Wu, J., Shao, A., Shen, W., Ye, P., Wang, Y., Ye, J., Jin, K., & Yang, J. (2024) · Journal of Medical Internet Research · doi:10.2196/51926 · Cited by 52 · see citing works
Self-citationLanguage Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models
Blevins, T., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.233 · Cited by 27 · see citing works
Cruz, T. M., & Smith, S. A. (2021) · Medical Care · doi:10.1097/mlr.0000000000001507 · Cited by 25 · see citing works
Exploring Social Media for Early Detection of Depression in COVID-19 Patients
Wu, J., Wu, X., Hua, Y., Lin, S.-X., Zheng, Y.-F., Yang, J., & Yang, J. (2023) · The Web Conference · doi:10.1145/3543507.3583867 · Cited by 24 · see citing works
Self-citationChatGPT for low- and middle-income countries: a Greek gift?
Lam, K. (2023) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2023.100906 · Cited by 18 · see citing works
Large Language Models with Retrieval-Augmented Generation for Zero-Shot Disease PhenotypingPreprint
Thompson, W. E., Vidmar, D., Freitas, J. K. D., Pfeifer, J. M., Fornwalt, B. K., Chen, R., Altay, G., Manghnani, K., Nelsen, A. C., Morland, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.06457 · Cited by 15 · see citing works
Implications of Using Chatbots for Future Surgical Education
Grigorian, A., Shipley, J. H., Nahmias, J. T., et al. (2023) · JAMA Surgery · doi:10.1001/jamasurg.2023.3875 · Cited by 13 · see citing works
MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway EncodingPreprint
Wu, J., Wu, X. W., Zheng, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.06611 · Cited by 2 · see citing works
Self-citationGuiding Clinical Reasoning with Large Language Models via Knowledge SeedsPreprint
Jiageng, W. U., Wu, X. W., & Yang, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.06609 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 11
- Radford A, Wu J, Child R, et al. Language models are unsuper- vised multitask learners. OpenAI Blog. 2019;1(8):9.
- Chung HW, Hou L, Longpre S, et al. Scaling instruction-finetuned language models. J Mach Learn Res. 2024;25(70):1-53.
- Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. In: 34th Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2020:1877-1901.
- Wei J, Wang X, Schuurmans D, et al. Chain of thought prompting elicits reasoning in large language models. In: 35th Conference on Neural Information Processing Systems. Neural Information Proc- essing Systems Foundation, Inc. (NeurIPS); 2022:24824-24837.
- Zhang X, Wu J, He Z, Liu X, Su Y. Medical exam question answering with large-scale reading comprehension. In: Proceed- ings of the 32nd AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Confer- ence and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (AAAI'18). 2018:5706-5713.
- Zhang Z, Zhang A, Li M, Smola A. Automatic chain of thought prompting in large language models. In: The 11th International Conference on Learning Representations (ICLR). 2023. Accessed April 11, 2024. https://openreview.net/forum?id=5NTt8GFjUHkr
- Guu K, Lee K, Tung Z, Pasupat P, Chang M. Retrieval augmented language model pre-training. In: International Conference on Machine Learning. PMLR; 2020:3929-3938.
- Wei J, Tay Y, Bommasani R, et al. Emergent abilities of large lan- guage models. In: Transactions on Machine Learning Research (TMLR). 2022. Accessed April 11, 2024. https://openreview.net/ forum?id=yzkSU5zdwD
- Singhal K, Azizi S, Tu T, et al. Large language models encode clini- cal knowledge. Nature. 2023;620(7972):172-180.
- Lehman E, Mahajan D, Wulff J, et al. Do we still need clinical lan- guage models? In: International Conference on Health, Inference, and Learning. PMLR; 2023:578-597.
- All rights reserved. For permissions, please email: journals.permissions@oup.com Journal of the American Medical Informatics Association, 2024, 31, 2054-2064 https://doi.org/10.1093/jamia/ocae079 Research and Applications · DOI
Cited by
Sorted by citations
Probabilistic medical predictions of large language models
Gu, B., Desai, R. J., Lin, K. J., & Yang, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01366-4 · Cited by 50 · see citing works
“Furthermore, due to data access and privacy concerns 34 , advanced LLMsoftenrelyongeneraldomaindatasetsandlackexposuretoreal-world EHRnotes,whichlimitstheireffectivenessinclinicalsettings 24,36 .”
Self-citationWu, J., Liu, X., Li, M., Li, W., Su, Z., Lin, S., Garay, L., Zhang, Z., Zhang, Y., Zeng, Q., et al. (2024) · NEJM AI · doi:10.1056/aira2400012 · Cited by 41 · see citing works
Self-citation
- Authors
- Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J.
- Affiliations
- Zhejiang University (CN)
- Tencent (China) (CN)
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Wu, Xian
- Yuan, Changzheng
- Yang, Jie
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-04-29
- Volume
- 31
- Issue
- 9
- Pages
- 2054-2064
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11339525
- OpenAlex
- W4396498719
- Cited by
- 41
- References
- 60
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 6 · 2023: 2
- FWCI
- 3.65
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, in-context learning, medical knowledge base, few-shot learning, language barriers in healthcare, resource-limited settings, external knowledge integration, medical licensing examinations
- MeSH terms
- China, Humans, Language, Licensure, Medical, Natural Language Processing, Knowledge Bases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
