The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
Open accessAbstract
The use of large language models (LLMs) in clinical medicine is currently thriving. Effectively transferring LLMs’ pertinent theoretical knowledge from computer science to their application in clinical medicine is crucial. Prompt engineering has shown potential as an effective method in this regard. To explore the application of prompt engineering in LLMs and to examine the reliability of LLMs, different styles of prompts were designed and used to ask different LLMs about their agreement with the American Academy of Orthopedic Surgeons (AAOS) osteoarthritis (OA) evidence-based guidelines. Each question was asked 5 times. We compared the consistency of the findings with guidelines across different evidence levels for different prompts and assessed the reliability of different prompts by asking the same question 5 times. gpt-4-Web with ROT prompting had the highest overall consistency (62.9%) and a significant performance for strong recommendations, with a total consistency of 77.5%. The reliability of the different LLMs for different prompts was not stable (Fleiss kappa ranged from −0.002 to 0.984). This study revealed that different prompts had variable effects across various models, and the gpt-4-Web with ROT prompt was the most consistent. An appropriate prompt could improve the accuracy of responses to professional medical questions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 6×
“For other LLMs, in the study by Karan et al. 8 , Med-PaLM, which is a version of Flan-PaLMthat has been instruction prompt-tuned and is not currently publicly available, was evaluated by a panel of clinicians.”
InfluentialFleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
When to use the Bonferroni correction
Armstrong, R. A. (2014) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12131 · Cited by 3,031 · cited 1×
“Bonferroni correction was used for multiple comparisons 41 .”
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Safiri, S., Kolahi, A.-A., Smith, E., Hill, C., Bettampadi, D., Mansournia, M. A., Hoy, D., Ashrafi-Asgarabad, A., Sepidarkish, M., Almasi-Hashiani, A., et al. (2020) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2019-216515 · Cited by 1,651
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · cited 1×
“Studies on the topic of prompt engineering in clinical medicine are limited, and most studies primarily apply prompt engineering techniques directly 31 or provide an overview of prompt engineering 14,32,33 in clinical medicine.”
A Review of Methods for Missing Data
Pigott, T. D. (2001) · Educational Research and Evaluation · doi:10.1076/edre.7.4.353.8937 · Cited by 650
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563
MedCalc: a new computer program for medical statistics
Schoonjans, F., Zalata, A., Depuydt, C. E., & Comhaire, F. H. (1995) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(95)01703-8 · Cited by 547
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0200-9 · Cited by 377
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363 · cited 4×
“Walker et al. 6 reported that the internal concordance of the provided information was complete (100”
InfluentialTheodorsson-Norheim, E. (1986) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(86)90081-7 · Cited by 357
A SAS® macro implementation of a multiple comparison post hoc test for a Kruskal–Wallis analysis
Elliott, A. C., & Hynan, L. S. (2011) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2010.11.002 · Cited by 325
Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C.-H., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 310
Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations
Ali, R., Tang, O. Y., Connolly, I. D., Zadnik Sullivan, P. L., Shin, J. H., Fridley, J. S., Asaad, W. F., Cielo, D., Oyelese, A. A., Doberstein, C. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002632 · Cited by 273
Akinci D’Antonoli, T., Stanzione, A., Bluethgen, C., Vernuccio, F., Ugga, L., Klontzas, M. E., Cuocolo, R., Cannella, R., & Koçak, B. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2023.232417 · Cited by 227
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204 · cited 1×
“In computer science, LLMs can obtain ideal and stable answers through prompt engineering, and adopting different prompts will affect the performance of LLMs, which is somewhat re fl ected in mathematical problems 9,11 – 13 .”
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204
GPT-4: a new era of artificial intelligence in medicine
Waisberg, E., Ong, J., Masalkhi, M., Kamran, S. A., Zaman, N., Sarker, P., Lee, A. G., & Tavakkoli, A. (2023) · Irish Journal of Medical Science (1971 -) · doi:10.1007/s11845-023-03377-8 · Cited by 186
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151
Performance of Generative Large Language Models on Ophthalmology Board–Style Questions
Cai, L. Z., Shaheen, A., Jin, A., Fukui, R., Yi, J. S., Yannuzzi, N., & Alabiad, C. (2023) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2023.05.024 · Cited by 139
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128
Osteoarthritis year in review 2023: Epidemiology & therapy
Perruccio, A. V., Young, J. J., Wilfong, J. M., Power, J. D., Cañizares, M., & Badley, E. M. (2023) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2023.11.012 · Cited by 126
Toyama, Y., Harigai, A., Abe, M., Nagano, M., Kawabata, M., Seki, Y., & Takase, K. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01491-2 · Cited by 125
ChatGPT for digital forensic investigation: The good, the bad, and the unknown
Scanlon, M., Breitinger, F., Hargreaves, C., Hilgert, J.-N., & Sheppard, J. (2023) · Forensic Science International: Digital Investigation · doi:10.1016/j.fsidi.2023.301609 · Cited by 118
Performance of ChatGPT on Specialty Certificate Examination in Dermatology multiple-choice questions
Passby, L., Jenko, N., & Wernham, A. G. H. (2023) · Clinical and Experimental Dermatology · doi:10.1093/ced/llad197 · Cited by 104
Pushpanathan, K., Lim, Z. W., Yew, S. M. E., Chen, D. Z., Lin, H. A. H., Goh, J. H. L., Wong, W., Wang, X., Tan, M. C. J., Koh, V., et al. (2023) · iScience · doi:10.1016/j.isci.2023.108163 · Cited by 91
Sun, M., Cole, A. P., Hanna, N., Mucci, L. A., Berry, D. L., Basaria, S. S., Ahern, D. K., Kibel, A. S., Choueiri, T. K., & Trinh, Q. (2018) · The Journal of Urology · doi:10.1016/j.juro.2017.11.136 · Cited by 90
Antaki, F., Milad, D., Chia, M. A., Giguère, C., Touma, S., El‐Khoury, J., Keane, P. A., & Duval, R. (2023) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324438 · Cited by 90
Koga, S., Martin, N. B., & Dickson, D. W. (2024) · Brain Pathology · doi:10.1111/bpa.13207 · Cited by 78
Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions
Fowler, T., Pullen, S., & Birkett, L. (2023) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324091 · Cited by 52
Prompt tuning for parameter-efficient medical image segmentation
Fischer, M., Bartler, A., & Yang, B. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103024 · Cited by 52
Kleinig, O., Gao, C., Kovoor, J. G., Gupta, A. K., Bacchi, S., & Chan, W. O. (2023) · Eye · doi:10.1038/s41433-023-02772-w · Cited by 38
Extracting symptoms from free-text responses using ChatGPT among COVID-19 cases in Hong Kong
Wei, W. I., Leung, C. L. K., Tang, A., McNeil, E. B., Wong, S. Y. S., & Kwok, K. O. (2023) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2023.11.002 · Cited by 30
Yoshiyasu, Y., Wu, F. M., Dhanda, A. K., Gorelik, D., Takashima, M., & Ahmed, O. G. (2023) · International Forum of Allergy & Rhinology · doi:10.1002/alr.23201 · Cited by 28
Investigating the Perception of the Future in GPT-3, -3.5 and GPT-4
Kozachek, D. (2023) · Creativity and Cognition · doi:10.1145/3591196.3596827 · Cited by 25
Smith, J., Choi, P. M. C., & Buntine, P. (2023) · Emergency Medicine Australasia · doi:10.1111/1742-6723.14280 · Cited by 12
Pokutnaya, D. Y., van Panhuis, W. G., Childers, B. R., Hawkins, M., Arcury‐Quandt, A. E., Matlack, M., Carpio, K., & Hochheiser, H. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08729-4 · Cited by 7
Cited by
Gaber, F., Shaik, M., Allega, F., Bilecz, A. J., Busch, F., Goon, K., Franke, V., & Akalin, A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01684-1 · Cited by 165
Prompt Engineering Paradigms for Medical Applications: Scoping Review
Zaghir, J., Naguib, M., Bjelogrlic, M., Névéol, A., Tannier, X., & Lovis, C. (2024) · Journal of Medical Internet Research · doi:10.2196/60501 · Cited by 143
Enhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 147
Self-citationThe Applications of Large Language Models in Mental Health: Scoping Review
Jin, Y., Liu, J., Li, P., Wang, B., Yan, Y., Zhang, H., Ni, C., Wang, J., Li, Y., Bu, Y., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/69284 · Cited by 94
“Previous studies have reported that the use of diverse prompts when applying LLMs can lead to substantial variations in the stability and accuracy of the models’ outputs [112,113].”
Performance of multimodal artificial intelligence chatbots evaluated on clinical oncology cases
Chen, D., Huang, R. S., Jomy, J., Wong, P., Yan, M., Croke, J., Tong, D., Hope, A., Eng, L., & Raman, S. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.37711 · Cited by 49
Metadata
- Authors
- Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J.
- Venue
- npj Digital Medicine
- Published
- 2024-02-20
- Volume
- 7
- Issue
- 1
- Pages
- 41-41
- ISSN
- 2398-6352 (ISSN-L)
- OpenAlex
- W4391971084
- Cited by
- 390
- References
- 45
- FWCI
- 31.27
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.