The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index, and 1 are not checked yet.
Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study
Open accessPublished version of a preprint
Abstract
Background: Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)–powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows. Objective: This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model’s performance across different categories. Methods: We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system. Results: Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the “Objective” section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05). Conclusions: Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model’s effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our simulation approach facilitates objective, replicable evaluation of the quality, accuracy, and safety of both ADS-generated transcripts and clinical notes.”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When compared to previous literature AI-driven documentation varied between “bad (29.7)” and “excellent (35.6)” highlighting the variability in documentation produced by AI systems [23, 25, 28].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To assess clinical note accuracy and completeness, we utilized standardized scoring rubrics of key clinical elements previously developed by 2 clinical experts based on simulated encounter recordings and clinical context.16”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“In our previous work using ChatGPT to generate clinical notes using uploaded transcripts, we identified potential accuracy and consistency concerns, with only 53% of clinical elements reported correctly across 3 replicates.16”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“We have previously demonstrated the utility of this method in assessing the quality and accuracy of notes generated by human scribes and ChatGPT4.16,34”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The impact of artificial intelligence in medicine on the future role of the physician
Ahuja, A. S. (2019) · PeerJ · doi:10.7717/peerj.7702 · Cited by 789 · cited 1×
“With the rapid and exponential growth in computing capacity, artificial intelligence (AI) is being increasingly used in health care, holding the promise of revolutionizing medical documentation, thus potentially alleviating the burden on physicians [6].”
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598 · cited 1×
“In the realm of ChatGPT-4, the study conducted by Johnson et al [15] delved into its performance in giving precise and comprehensive medical information.”
The Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP)
Lewis, K. L., Bohnert, C. A., Gammon, W. L., Hölzer, H., Lyman, L., Smith, C., Thompson, T. M., Wallace, A., & Gliva-McConvey, G. (2017) · Advances in Simulation · doi:10.1186/s41077-017-0043-4 · Cited by 580 · cited 1×
“Briefly, for each case, a storyboard was created by subject matter experts and used for training the standardized patient to ensure standard content delivery according to best practices [11].”
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363 · cited 1×
“However, in another study by Walker et al [17] aimed at evaluating the reliability of medical information provided by ChatGPT-4, multiple iterations of their queries executed through the model yielded a remarkable 100% internal consistency among the generated outputs [17].”
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 343 · cited 1×
“Health care providers envision numerous applications for ChatGPT-4, such as answering patient questions, automated insurance prior authorizations, and creating differential diagnoses [9,10].”
Association of Electronic Health Record Design and Use Factors With Clinician Stress and Burnout
Kroth, P. J., Morioka-Douglas, N., Veres, S., Babbott, S., Poplau, S., Qeadan, F., Parshall, C., Corrigan, K., & Linzer, M. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.9609 · Cited by 333 · cited 1×
“Acknowledging the increasing documentation burden contributing to physician burnout, generative AI technology for clinical note documentation may save time [2,19].”
Mobile, social, and wearable computing and the evolution of psychological practice.
Morris, M. E., & Aguilera, A. (2012) · Professional Psychology: Research and Practice · doi:10.1037/a0029041 · Cited by 298
Towards Making the Most of ChatGPT for Machine TranslationPreprint
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4390455 · Cited by 262 · cited 1×
“With this in mind, ChatGPT-4’s temperature allows for variability, but this setting is not accessible to the end user [25,26].”
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Frequency and Types of Patient-Reported Errors in Electronic Health Record Ambulatory Care Notes
Bell, S. K., Delbanco, T., Elmore, J. G., Fitzgerald, P. S., Fossa, A., Harcourt, K., Leveille, S. G., Payne, T. H., Stametz, R. A., Walker, J., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.5867 · Cited by 238 · cited 1×
“In recent years, it has become increasingly evident that inaccuracies in medical notes lead to miscommunication, diagnostic discrepancies, and patients’ perceptions of subpar medical care [1].”
Research works printed but not linked: 2
- Florig, ST, 2021, AMIA Annu Symp Proc, 2021, 457
- Rule, A, 2021, AMIA Annu Symp Proc, 2021, 1059
Cited by
Biswas, A., & Talukdar, W. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24may1483 · Cited by 1,028
Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review
Bracken, A., Reilly, C., Feeley, A., Sheehan, E., Merghani, K., & Feeley, I. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02157-4 · Cited by 113
“…has been reported with the use of the physician documentation quality instrument (PDQI-9 tool) [34] When compared to previous literature AI-driven documentation varied between “bad (29.7)” and “excellent (35.6)” highlighting the variability in documentation produced by AI systems [23, 25, 28].”
InfluentialThe Impact of AI Scribes on Streamlining Clinical Documentation: A Systematic Review
Sasseville, M., Yousefi, F., Ouellet, S., Naye, F., Stefan, T., Carnovale, V., Bergeron, F., Ling, L., Gheorghiu, B., Hagens, S., et al. (2025) · Healthcare · doi:10.3390/healthcare13121447 · Cited by 110
“One study featured simulated patient-provider encounters across various ambulatory specialties [34].”
InfluentialAI Scribes in Health Care: Balancing Transformative Potential With Responsible Integration
Leung, T. I., Coristine, A. J., & Benis, A. (2025) · JMIR Medical Informatics · doi:10.2196/80898 · Cited by 86
“Other evaluations of stand-alone tools for audio recordings suggest potentially high rates of errors, including incorrect information, omissions, and hallucinations [11,29,38].”
The effect of ambient artificial intelligence notes on provider burnout.
Misurac, J., Knake, L. A., & Blum, J. M. (2025) · Applied Clinical Informatics · doi:10.1055/a-2461-4576 · Cited by 58
“23 Thus, AI technologies must continue to evolve and adapt to the different clinical use cases and needs.”
Metadata
- Authors
- Kernberg, A., Gold, J. A., & Mohan, V.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Mohan, Vishnu
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-04-22
- Volume
- 26
- Pages
- e54419
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11074889
- OpenAlex
- W4392632345
- Semantic Scholar
- 78dee4159add8e48b50a2100104fbfae99d2185b
- Cited by
- 120
- References
- 27
- Citations by year
- 2026: 55 · 2025: 52 · 2024: 13
- FWCI
- 11.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Research and Theory › Nursing Diagnosis and Documentation
- Keywords
- ChatGPT-4, SOAP notes, clinical documentation, physician-patient encounters, audio recordings, note quality, medical scribes, physician burnout, large language models, history and physical examination, outpatient care
- MeSH terms
- Artificial Intelligence, Documentation, Humans, Physician-Patient Relations, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study. https://app.cochat.ai/papers/10.2196/54419
- The paper
- Kernberg, A., Gold, J. A., & Mohan, V. (2024). Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study. Journal of Medical Internet Research. https://doi.org/10.2196/54419
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.