Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Background: Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)–powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows. Objective: This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model’s performance across different categories. Methods: We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system. Results: Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the “Objective” section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05). Conclusions: Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model’s effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our simulation approach facilitates objective, replicable evaluation of the quality, accuracy, and safety of both ADS-generated transcripts and clinical notes.”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When compared to previous literature AI-driven documentation varied between “bad (29.7)” and “excellent (35.6)” highlighting the variability in documentation produced by AI systems [23, 25, 28].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To assess clinical note accuracy and completeness, we utilized standardized scoring rubrics of key clinical elements previously developed by 2 clinical experts based on simulated encounter recordings and clinical context.16”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“In our previous work using ChatGPT to generate clinical notes using uploaded transcripts, we identified potential accuracy and consistency concerns, with only 53% of clinical elements reported correctly across 3 replicates.16”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“We have previously demonstrated the utility of this method in assessing the quality and accuracy of notes generated by human scribes and ChatGPT4.16,34”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The impact of artificial intelligence in medicine on the future role of the physician
Ahuja, A. S. (2019) · PeerJ · doi:10.7717/peerj.7702 · Cited by 789 · cited 1×
“With the rapid and exponential growth in computing capacity, artificial intelligence (AI) is being increasingly used in health care, holding the promise of revolutionizing medical documentation, thus potentially alleviating the burden on physicians [6].”
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598 · cited 1×
“In the realm of ChatGPT-4, the study conducted by Johnson et al [15] delved into its performance in giving precise and comprehensive medical information.”
The Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP)
Lewis, K. L., Bohnert, C. A., Gammon, W. L., Hölzer, H., Lyman, L., Smith, C., Thompson, T. M., Wallace, A., & Gliva-McConvey, G. (2017) · Advances in Simulation · doi:10.1186/s41077-017-0043-4 · Cited by 580 · cited 1×
“Briefly, for each case, a storyboard was created by subject matter experts and used for training the standardized patient to ensure standard content delivery according to best practices [11].”
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363 · cited 1×
“However, in another study by Walker et al [17] aimed at evaluating the reliability of medical information provided by ChatGPT-4, multiple iterations of their queries executed through the model yielded a remarkable 100% internal consistency among the generated outputs [17].”
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 343 · cited 1×
“Health care providers envision numerous applications for ChatGPT-4, such as answering patient questions, automated insurance prior authorizations, and creating differential diagnoses [9,10].”
Association of Electronic Health Record Design and Use Factors With Clinician Stress and Burnout
Kroth, P. J., Morioka-Douglas, N., Veres, S., Babbott, S., Poplau, S., Qeadan, F., Parshall, C., Corrigan, K., & Linzer, M. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.9609 · Cited by 333 · cited 1×
“Acknowledging the increasing documentation burden contributing to physician burnout, generative AI technology for clinical note documentation may save time [2,19].”
Mobile, social, and wearable computing and the evolution of psychological practice.
Morris, M. E., & Aguilera, A. (2012) · Professional Psychology: Research and Practice · doi:10.1037/a0029041 · Cited by 298
Towards Making the Most of ChatGPT for Machine TranslationPreprint
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4390455 · Cited by 264 · cited 1×
“With this in mind, ChatGPT-4’s temperature allows for variability, but this setting is not accessible to the end user [25,26].”
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Frequency and Types of Patient-Reported Errors in Electronic Health Record Ambulatory Care Notes
Bell, S. K., Delbanco, T., Elmore, J. G., Fitzgerald, P. S., Fossa, A., Harcourt, K., Leveille, S. G., Payne, T. H., Stametz, R. A., Walker, J., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.5867 · Cited by 238 · cited 1×
“In recent years, it has become increasingly evident that inaccuracies in medical notes lead to miscommunication, diagnostic discrepancies, and patients’ perceptions of subpar medical care [1].”
Research works printed but not linked: 2
- Florig, ST, 2021, AMIA Annu Symp Proc, 2021, 457
- Rule, A, 2021, AMIA Annu Symp Proc, 2021, 1059
Cited by
Items 101–118 of 118
Philips, R. H. W., Mahajan, A., Estephan, L. E., Auger, S., Haque, W., Palmer, D., McQueen, J., MacCracken, E., Argyris, P. P., Trzcinska, A., et al. (2026) · Annals of Oncology · doi:10.1016/j.annonc.2026.05.709
McCradden, M. D., Tng, S., Jeyaseelan, D., Leane, C., Campbell, M., Braund, T., Earle-Bandaralage, L., Ebrahimi, M., Sharma, A., & Tang, J. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-111656
Sangalli, L., Mirfarsi, S., Mimms, C., & Prodoehl, J. (2026) · Oral Surgery Oral Medicine Oral Pathology and Oral Radiology · doi:10.1016/j.oooo.2026.06.013
Verma, R., Bains, S. S., Muthani, S. H. R., Arunachalam, A., Mohan, V., & Gold, J. A. (2026) · JMIR Medical Informatics · doi:10.2196/85015
“While the current literature is limited on the use of AI scribes in the ICU, our results suggest that the use of customized AI-assisted scribes in the ICU setting can generate similar outputs compared to outpatient AI-assisted scribe apps [26-28].”
Self-citationOh, C., Hwangbo, Y., Cheon, W., & Lee, J. (2026) · Journal of Medical Internet Research · doi:10.2196/91696
“Kernberg et al [32] found that ChatGPT-4–generated medical notes exhibited considerable quality variability unsuitable for clinical use, whereas our structured pipeline achieves consistent, reproducible results.”
Friedrich, N. A., Das, S. K., Luu, M., Daniels, J. P., Heard, J. R., Barsegyan, A. G., Freedland, S. J., Spiegel, B. M. R., Kokorowski, P. K., Hernandez, G. G., et al. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00199
Oh, D., Cripe, J., Baez-Silva, J., Shaw, C., Gusberg, S., Sniff, J., Gwin, J., & Rezzadeh, K. (2026) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000007970
Michihata, N., Ishii, H., Tsujimura, H., Takano, N., Nomura, T., Takeuchi, Y., Kagawa, S., Asai, S., Kita, E., Komaru, A., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2917-6695
The use of AI scribes across medical fields and why psychiatry is lagging behind: A scoping review
Daignault, C., Audelin-Rinfret, J., Désilets, M., Bahremand, A., & Do, A. (2026) · Santé mentale au Québec · doi:10.7202/1126800ar
Multispecialty Dental EMRs from Chairside Audio: An Exploratory Study
Wei, L., Wang, Y., Lu, J., Wang, H., Cao, Y., Ma, D., Wu, J., Tong, Y., Xu, W., Wang, Y., et al. (2026) · Journal of Dental Research · doi:10.1177/00220345261464508
PRACTALL 2025: Artificial intelligence—application of allergy and immunology to patient care
Shamji, M. H., Rider, N. L., Adcock, I. M., Pongdee, T., Giacco, S. D., Chang, C., Jaen, M. T., & Williams, P. V. (2026) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2026.04.031 · Cited by 1
Super, I., Rezaeian, O., & Asan, O. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106648
Kwolek, K., Laskowska, W., Pilarek, M., Ożga, S., Roszkowska, M., Furtak, M., Janczarski, S., Sarna, D., Wąs, L., & Tryczyński, P. (2026) · International Journal of Innovative Technologies in Social Science · doi:10.31435/ijitss.3(51).2026.6076
Bahl, R., Bansal, H., Buchanan, G. R., & Verspoor, K. M. (2026) · doi:10.2196/preprints.110590
Lin, S., Zhang, C., Zhong, F., Liao, X., Gong, L., & Wang, D. (2026) · Cancer Medicine · doi:10.1002/cam4.72188
“Nevertheless, some studies suggested that while AI tools demonstrated potential, they also presented challenges such as accuracy, educational outcomes, academic integrity, and ethical concerns [11–13].”
Çalışkan, Y., Sarı, M., & Binokay, H. (2026) · Eating and Weight Disorders - Studies on Anorexia Bulimia and Obesity · doi:10.1007/s40519-026-01905-w
“Additionally, AI-powered administrative tools—such as automated clinical documentation—have been explored as a means to reduce clinician burden, though current evidence highlights significant challenges in accuracy and reliability that must be addressed before widespread adoption [21].”
́Kane, R. O., Stonehouse‐Smith, D., Shirazi, M., Hutchinson, R., Seehra, J., & Cobourne, M. T. (2026) · BDJ · doi:10.1038/s41415-026-9951-9 · Cited by 1
Shi, K. X., Creasman, M., Schmajuk, G., Oates, A., Byron, M. E., García-Agúndez, A., & Yazdany, J. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag177
Metadata
- Authors
- Kernberg, A., Gold, J. A., & Mohan, V.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Mohan, Vishnu
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-04-22
- Volume
- 26
- Pages
- e54419
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11074889
- OpenAlex
- W4392632345
- Semantic Scholar
- 78dee4159add8e48b50a2100104fbfae99d2185b
- Cited by
- 120
- References
- 27
- Citations by year
- 2026: 55 · 2025: 52 · 2024: 13
- FWCI
- 11.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Research and Theory › Nursing Diagnosis and Documentation
- Keywords
- ChatGPT-4, SOAP notes, clinical documentation, physician-patient encounters, audio recordings, note quality, medical scribes, physician burnout, large language models, history and physical examination, outpatient care
- MeSH terms
- Artificial Intelligence, Documentation, Humans, Physician-Patient Relations, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.