Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Background: The advancement of large language models (LLMs) offers significant opportunities for health care, particularly in the generation of medical documentation. However, challenges related to ensuring the accuracy and reliability of LLM outputs, coupled with the absence of established quality standards, have raised concerns about their clinical application. Objective: This study aimed to develop and validate an evaluation framework for assessing the accuracy and clinical applicability of LLM-generated emergency department (ED) records, aiming to enhance artificial intelligence integration in health care documentation. Methods: We organized the Healthcare Prompt-a-thon, a competitive event designed to explore the capabilities of LLMs in generating accurate medical records. The event involved 52 participants who generated 33 initial ED records using HyperCLOVA X, a Korean-specialized LLM. We applied a dual evaluation approach. First, clinical evaluation: 4 medical professionals evaluated the records using a 5-point Likert scale across 5 criteria—appropriateness, accuracy, structure/format, conciseness, and clinical validity. Second, quantitative evaluation: We developed a framework to categorize and count errors in the LLM outputs, identifying 7 key error types. Statistical methods, including Pearson correlation and intraclass correlation coefficients (ICC), were used to assess consistency and agreement among evaluators. Results: The clinical evaluation demonstrated strong interrater reliability, with ICC values ranging from 0.653 to 0.887 (P<.001), and a test-retest reliability Pearson correlation coefficient of 0.776 (P<.001). Quantitative analysis revealed that invalid generation errors were the most common, constituting 35.38% of total errors, while structural malformation errors had the most significant negative impact on the clinical evaluation score (Pearson r=–0.654; P<.001). A strong negative correlation was found between the number of quantitative errors and clinical evaluation scores (Pearson r=–0.633; P<.001), indicating that higher error rates corresponded to lower clinical acceptability. Conclusions: Our research provides robust support for the reliability and clinical acceptability of the proposed evaluation framework. It underscores the framework’s potential to mitigate clinical burdens and foster the responsible integration of artificial intelligence technologies in health care, suggesting a promising direction for future research and practical applications in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Ambient artificial intelligence scribes to alleviate the burden of clinical documentation.
Tierney, A. A., Gayre, G., Hoberman, B., Mattern, B., Ballesca, M., Kipnis, P., Liu, V., & Lee, K. (2024) · NEJM Catalyst · doi:10.1056/cat.23.0404 · Cited by 400 · cited 1×
“Moving beyond theoretical knowledge, LLMs are now being applied to practical tasks, including the creation of detailed medical documents, which indicates their potential for direct applications in clinical settings [5-7].”
Self-citationUsing ChatGPT to write patient clinic letters
Ali, S. R., Dobbs, T. D., Hutchings, H. A., & Whitaker, I. S. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00048-1 · Cited by 375 · cited 1×
“Their ability to process vast amounts of medical literature and patient data suggests a transformative impact on both medical research and practice [9,10].”
Ethics of large language models in medicine and medical research
Li, H., Moon, J. T., Purkayastha, S., Celi, L. A., Trivedi, H., & Gichoya, J. W. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00083-3 · Cited by 286 · cited 1×
“Concerns regarding the accuracy of generated content, particularly in scenarios involving hallucinations or fabrications [6,11,12], pose substantial risks in the high-stakes environment of health care.”
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · cited 1×
“This is highlighted by their successful performance on challenging exams, such as the USMLE (United States Medical Licensing Examination), demonstrating their proficiency in medical concepts [3,4].”
Use of gpt-4 to diagnose complex clinical cases
Eriksen, A. V., Möller, S., & Ryg, J. (2024) · NEJM AI · doi:10.1056/aip2300031 · Cited by 200 · cited 1×
“Moving beyond theoretical knowledge, LLMs are now being applied to practical tasks, including the creation of detailed medical documents, which indicates their potential for direct applications in clinical settings [5-7].”
Assessing electronic note quality using the physician documentation quality instrument (PDQI-9)
Bakken, S., Wrenn, J. O., Siegler, E. L., & Stetson, P. D. (2012) · Applied Clinical Informatics · doi:10.4338/aci-2011-11-ra-0070 · Cited by 182 · cited 1×
“One notable reference was the Physician Documentation Quality Instrument (PDQI-9), which provides a comprehensive framework for evaluating the quality of medical records [19,20].”
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90
Cited by
Chow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
Papageorgiou, P. S., Christodoulou, R. C., Pitsillos, R., Petrou, V., Vamvouras, G., Kormentza, E. V., Papagelopoulos, P. J., & Georgiou, M. F. (2025) · Applied Sciences · doi:10.3390/app15169005 · Cited by 8
Artificial intelligence language models for medical text analysis: A systematic review
Sorayaie Azar, A., Bagherzadeh Mohasefi, J., Wiil, U. K., Naemi, A., & Ebrahimi, A. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103441 · Cited by 2
Metadata
- Authors
- Seo, J., Choi, D., Kim, T., Cha, W. C., Kim, M., Yoo, H., Oh, N., Yi, Y., Lee, K. H., & Choi, E.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-11-20
- Volume
- 26
- Pages
- e58329
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W4402765248
- Cited by
- 52
- References
- 8
- FWCI
- 4.17
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.