Evaluating the ChatGPT family of models for biomedical reasoning and classification
Open accessAbstract
OBJECTIVE: Large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates ChatGPT family of models (GPT-3.5, GPT-4) in biomedical tasks beyond question-answering. MATERIALS AND METHODS: We evaluated model performance with 11 122 samples for two fundamental tasks in the biomedical domain-classification (n = 8676) and reasoning (n = 2446). The first task involves classifying health advice in scientific literature, while the second task is detecting causal relations in biomedical literature. We used 20% of the dataset for prompt development, including zero- and few-shot settings with and without chain-of-thought (CoT). We then evaluated the best prompts from each setting on the remaining dataset, comparing them to models using simple features (BoW with logistic regression) and fine-tuned BioBERT models. RESULTS: Fine-tuning BioBERT produced the best classification (F1: 0.800-0.902) and reasoning (F1: 0.851) results. Among LLM approaches, few-shot CoT achieved the best classification (F1: 0.671-0.770) and reasoning (F1: 0.682) results, comparable to the BoW model (F1: 0.602-0.753 and 0.675 for classification and reasoning, respectively). It took 78 h to obtain the best LLM results, compared to 0.078 and 0.008 h for the top-performing BioBERT and BoW models, respectively. DISCUSSION: The simple BoW model performed similarly to the most complex LLM prompting. Prompt engineering required significant investment. CONCLUSION: Despite the excitement around viral ChatGPT, fine-tuning for two fundamental biomedical natural language processing tasks remained the best strategy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent advances in large language models present an opportunity to streamline the informed consent form creation process while improving readability, understandability, and actionability.
“The recent advances in large language models (LLMs) present an opportunity to streamline the ICF creation process while improving readability, understandability, and actionability.”
From Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study (2025) · cites this paper as [14]
The Conclusion describes potential applications of LLMs in biomedical text processing, and the Background mentions research on translating radiology reports into plain language. Neither passage addresses informed consent form creation or shows improvements in readability, understandability, and actionability. (LLM, conf=0.72)
Experimental — accuracy not yet measuredLLMs show significant potential in health informatics tasks such as name entity extraction, patient trial matching, biomedical reasoning and classification, prediction of admissions, and automation of administrative tasks.
“LLMs show significant potential in health informatics, including tasks such as name entity extraction [11], patient trial matching [1213], biomedical reasoning and classification [14], prediction of admissions [15], automation of administrative tasks [16], and so forth.”
From Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study (2025) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605 · see citing works
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · Visual Computing for Industry, Biomedicine, and Art · doi:10.1186/s42492-023-00136-5 · Cited by 367 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
Large language models to identify social determinants of health in electronic health records
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00970-0 · Cited by 278 · see citing works
Self-citationUse of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251 · see citing works
Self-citationCan Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · see citing works
Self-citationLarge Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., Yang, L., Huang, H., Ye, W., Geng, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12095 · Cited by 86 · see citing works
Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnessPreprint
Zuccon, G., & Koopman, B. (2023) · arXiv.org · doi:10.48550/arxiv.2302.13793 · Cited by 54 · see citing works
Large Language Models to Identify Social Determinants of Health in Electronic Health RecordsPreprint
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I. “., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M. H., Harper, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06354 · Cited by 7 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 14
- Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [Internet]. In: NeurIPS Proceedings. 2017. Accessed April 4, 2023. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547d ee91fbd053c1c4a845aa-Paper.pdf
- 0 Sutton RS, Barto AG. Reinforcement Learning: An Introduction. 2nd ed. The MIT Press; 2018.
- Ouyang L, Wu J, Jiang X, et al. Training language models to fol- low instructions with human feedback [Internet]. In: NeurIPS Pro- ceesings. 2022. Accessed April 4, 2023. https://proceedings. neurips.cc/paper_files/paper/2022/hash/b1efde53- be364a73914f58805a001731-Abstract-Conference.html
- Gilson A, Safranek CW, Huang T, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312.
- Singhal K, Azizi S, Tu T, et al. Large Language Models Encode Clinical Knowledge [Internet]. arXiv [cs.CL]. 2022. http://arxiv. org/abs/2212.13138, preprint: not peer reviewed.
- Lehman E, Hernandez E, Mahajan D, et al. Do we still need clini- cal language models? [Internet]. arXiv [cs.CL]. 2023. http://arxiv. org/abs/2302.08091, preprint: not peer reviewed.
- Li Y, Wang J, Yu B. Detecting health advice in medical research lit- erature. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana, Dominican Republic: Association for Computational Linguistics; 2021:6018- 6029.
- Yu B, Li Y, Wang J. Detecting causal language use in science find- ings. In: Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). Hong Kong, China: Association for Computational Lin- guistics; 2019:4664-74.
- Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioin- formatics. 2020;36(4):1234-1240.
- Brown TB, Mann B, Ryder N, et al. Language models are few-shot learners [Internet]. arXiv [cs.CL]. 2020. http://arxiv.org/abs/2005. 14165, preprint: not peer reviewed.
- Wei J, Tay Y, Bommasani R, et al. Emergent abilities of large lan- guage models [Internet]. arXiv [cs.CL]. 2022. http://arxiv.org/abs/ 2206.07682, preprint: not peer reviewed.
- Savova GK, Danciu I, Alamudun F, et al. Use of natural language processing to extract clinical cancer phenotypes from electronic medical records. Cancer Res. 2019;79(21):5463-5470.
- Beam K, Sharma P, Kumar B, et al. Performance of a large lan- guage model on practice questions for the neonatal board exami- nation. JAMA Pediatr. 2023;177(9):977-979.
- Murk W, Goralnick E, Brownstein JS, Landman AB. Quality of layperson CPR instructions from artificial intelligence voice assis- tants. JAMA Netw Open. 2023;6(8):e2331205.
Cited by
Sorted by citations
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615 · see citing works
Self-citationLarge language models to identify social determinants of health in electronic health records
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00970-0 · Cited by 278 · see citing works
Self-citationChen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112 · see citing works
Self-citationPerformance of multimodal artificial intelligence chatbots evaluated on clinical oncology cases
Chen, D., Huang, R. S., Jomy, J., Wong, P., Yan, M., Croke, J., Tong, D., Hope, A., Eng, L., & Raman, S. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.37711 · Cited by 49 · see citing works
Toward Clinical-Grade Evaluation of Large Language Models
Moreno, A. C., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.11.012 · Cited by 28 · see citing works
Self-citationTransforming Informed Consent Generation Using Large Language Models: Mixed Methods Study
Shi, Q., Luzuriaga, K., Allison, J. J., Oztekin, A., Faro, J. M., Lee, J. L., Hafer, N., McManus, M., & Zai, A. H. (2025) · JMIR Medical Informatics · doi:10.2196/68139 · Cited by 23 · see citing works
“LLMs show significant potential in health informatics, including tasks such as name entity extraction [11], patient trial matching [12,13], biomedical reasoning and classification [14], prediction of admissions [15], automation of administrative tasks [16], and so forth.”
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Gallifant, J., Chen, S., Moreira, P. J. F., Munch, N., Gao, M., Pond, J., Celi, L. A., Aerts, H., Hartvigsen, T., & Bitterman, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.726 · Cited by 19 · see citing works
Self-citationLanguage Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Gallifant, J., Chen, S., Moreira, P., Munch, N., Gao, M., Pond, J., Celi, L. A., Aerts, H., Hartvigsen, T., & Bitterman, D. (2024) · arXiv · doi:10.48550/arxiv.2406.12066 · Cited by 19 · see citing works
Self-citation
- Authors
- Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S.
- Affiliations
- Brigham and Women's Hospital (US)
- Boston Children's Hospital (US)
- Technische Universität Darmstadt (DE)
- Corresponding author
- Bitterman, Danielle S
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-01-22
- Volume
- 31
- Issue
- 4
- Pages
- 940-948
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10990500
- OpenAlex
- W4391126287
- Cited by
- 91
- References
- 17
- Citations by year
- 2026: 14 · 2025: 37 · 2024: 33 · 2023: 7
- FWCI
- 8.58
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Institutes of Health (R01GM114355, R01LM013486)
- Woods Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- ChatGPT, large language models, biomedical text classification, chain-of-thought prompting, few-shot prompting, prompt engineering, BioBERT, fine-tuning, biomedical literature
- MeSH terms
- Language, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
