Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 28 of the 29 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Background: Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)–powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows. Objective: This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model’s performance across different categories. Methods: We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system. Results: Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the “Objective” section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05). Conclusions: Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model’s effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our simulation approach facilitates objective, replicable evaluation of the quality, accuracy, and safety of both ADS-generated transcripts and clinical notes.”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When compared to previous literature AI-driven documentation varied between “bad (29.7)” and “excellent (35.6)” highlighting the variability in documentation produced by AI systems [23, 25, 28].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To assess clinical note accuracy and completeness, we utilized standardized scoring rubrics of key clinical elements previously developed by 2 clinical experts based on simulated encounter recordings and clinical context.16”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“In our previous work using ChatGPT to generate clinical notes using uploaded transcripts, we identified potential accuracy and consistency concerns, with only 53% of clinical elements reported correctly across 3 replicates.16”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“We have previously demonstrated the utility of this method in assessing the quality and accuracy of notes generated by human scribes and ChatGPT4.16,34”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The impact of artificial intelligence in medicine on the future role of the physician
Ahuja, A. S. (2019) · PeerJ · doi:10.7717/peerj.7702 · Cited by 789 · cited 1×
“With the rapid and exponential growth in computing capacity, artificial intelligence (AI) is being increasingly used in health care, holding the promise of revolutionizing medical documentation, thus potentially alleviating the burden on physicians [6].”
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598 · cited 1×
“In the realm of ChatGPT-4, the study conducted by Johnson et al [15] delved into its performance in giving precise and comprehensive medical information.”
The Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP)
Lewis, K. L., Bohnert, C. A., Gammon, W. L., Hölzer, H., Lyman, L., Smith, C., Thompson, T. M., Wallace, A., & Gliva-McConvey, G. (2017) · Advances in Simulation · doi:10.1186/s41077-017-0043-4 · Cited by 580 · cited 1×
“Briefly, for each case, a storyboard was created by subject matter experts and used for training the standardized patient to ensure standard content delivery according to best practices [11].”
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363 · cited 1×
“However, in another study by Walker et al [17] aimed at evaluating the reliability of medical information provided by ChatGPT-4, multiple iterations of their queries executed through the model yielded a remarkable 100% internal consistency among the generated outputs [17].”
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 343 · cited 1×
“Health care providers envision numerous applications for ChatGPT-4, such as answering patient questions, automated insurance prior authorizations, and creating differential diagnoses [9,10].”
Association of Electronic Health Record Design and Use Factors With Clinician Stress and Burnout
Kroth, P. J., Morioka-Douglas, N., Veres, S., Babbott, S., Poplau, S., Qeadan, F., Parshall, C., Corrigan, K., & Linzer, M. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.9609 · Cited by 333 · cited 1×
“Acknowledging the increasing documentation burden contributing to physician burnout, generative AI technology for clinical note documentation may save time [2,19].”
Mobile, social, and wearable computing and the evolution of psychological practice.
Morris, M. E., & Aguilera, A. (2012) · Professional Psychology: Research and Practice · doi:10.1037/a0029041 · Cited by 298
Towards Making the Most of ChatGPT for Machine TranslationPreprint
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4390455 · Cited by 264 · cited 1×
“With this in mind, ChatGPT-4’s temperature allows for variability, but this setting is not accessible to the end user [25,26].”
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Frequency and Types of Patient-Reported Errors in Electronic Health Record Ambulatory Care Notes
Bell, S. K., Delbanco, T., Elmore, J. G., Fitzgerald, P. S., Fossa, A., Harcourt, K., Leveille, S. G., Payne, T. H., Stametz, R. A., Walker, J., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.5867 · Cited by 238 · cited 1×
“In recent years, it has become increasingly evident that inaccuracies in medical notes lead to miscommunication, diagnostic discrepancies, and patients’ perceptions of subpar medical care [1].”
Gidwani, R., Nguyen, C., Kofoed, A., Carragee, C., Rydel, T., Nelligan, I., Sattler, A., Mahoney, M., & Lin, S. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2122 · Cited by 202
Dahmen, J., Kayaalp, M. E., Ollivier, M., Pareek, A., Hirschmann, M. T., Karlsson, J., & Winkler, P. W. (2023) · Knee Surgery, Sports Traumatology, Arthroscopy · doi:10.1007/s00167-023-07355-6 · Cited by 193 · cited 1×
“Health care providers envision numerous applications for ChatGPT-4, such as answering patient questions, automated insurance prior authorizations, and creating differential diagnoses [9,10].”
Assessing electronic note quality using the physician documentation quality instrument (PDQI-9)
Bakken, S., Wrenn, J. O., Siegler, E. L., & Stetson, P. D. (2012) · Applied Clinical Informatics · doi:10.4338/aci-2011-11-ra-0070 · Cited by 182 · cited 2×
“Lastly, the Physician Documentation Quality Instrument (PDQI) scoring system, which is a validated tool to assess note quality, was used to evaluate the quality of the generated notes [13].”
InfluentialVan Bulck, L., & Moons, P. (2024) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvad038 · Cited by 143 · cited 1×
“These data are often incomplete and sometimes misleading [18].”
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2023) · arXiv · doi:10.48550/arxiv.2309.08221 · Cited by 140
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623306 · Cited by 136 · cited 1×
“With this in mind, ChatGPT-4’s temperature allows for variability, but this setting is not accessible to the end user [25,26].”
Medical Documentation Burden Among US Office-Based Physicians in 2019: A National Study
Gaffney, A., Woolhandler, S., Cai, C., Bor, D., Himmelstein, J., McCormick, D., & Himmelstein, D. U. (2022) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2022.0372 · Cited by 118 · cited 2×
“Acknowledging the increasing documentation burden contributing to physician burnout, generative AI technology for clinical note documentation may save time [2,19].”
Goss, F. R., Blackley, S. V., Ortega, C. A., Kowalski, L. T., Landman, A., Lin, C., Meteer, M. W., Bakes, S., Gradwohl, S. C., Bates, D. W., et al. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.07.017 · Cited by 82 · cited 1×
“This, combined with software using large language models (LLMs), now enables the creation of structured medical notes in close temporal relation to the clinical encounter, thereby decreasing the clinician documentation burden [7,8].”
Efficiency and safety of speech recognition for documentation in the electronic health record
Hodgson, T., Magrabi, F., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx073 · Cited by 63 · cited 1×
“It should be acknowledged that this study only used transcripts, eliminating the confounder of any potential errors introduced by the speech recognition aspect [23,24].”
Speech Recognition as a Transcription Aid: A Randomized Comparison With Standard Transcription
Mohr, D. N., Turner, D. W., Pond, G. R., Kamath, J. S., Vos, C. B. D., & Carpenter, P. C. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1130 · Cited by 53 · cited 1×
“It should be acknowledged that this study only used transcripts, eliminating the confounder of any potential errors introduced by the speech recognition aspect [23,24].”
Data omission by physician trainees on ICU rounds.
Artis, K. A., Bordley, J., Mohan, V., & Gold, J. A. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003557 · Cited by 36
Self-citationChatGPT is not the solution to physicians’ documentation burden
Preiksaitis, C. M., Sinsky, C. A., & Rose, C. C. (2023) · Nature Medicine · doi:10.1038/s41591-023-02341-4 · Cited by 36 · cited 1×
“This outcome underscores the imperative for exercising caution when solely relying on AI-generated medical information and the need for continuous evaluation, as others have noted [16].”
Pranaat, R., Mohan, V., O'Reilly, M., Hirsh, M., McGrath, K., Scholl, G., Woodcock, D., & Gold, J. A. (2017) · JMIR Medical Informatics · doi:10.2196/medinform.7883 · Cited by 26 · cited 3×
“However, both of these solutions have significant limitations and concerns regarding documentation accuracy and lack of impact on many provider-specific metrics surrounding after-hours charting and burnout [3,4].”
Self-citationCorby, S., Whittaker, K., Ash, J. S., Mohan, V., Becton, J., Solberg, N., Bergstrom, R., Orwoll, B., Hoekstra, C., & Gold, J. A. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01560-4 · Cited by 13 · cited 1×
“In addition, the financial implications of employing medical scribes render them inaccessible to numerous health care practices [5].”
Self-citationChart Completion Time of Attending Physicians While Using Medical Scribes.
Florig, S. T., Corby, S., Rosson, N. T., Devara, T., Weiskopf, N. G., Gold, J. A., & Mohan, V. (2021) · PubMed · Cited by 13
Self-citationMedical Record Closure Practices of Physicians Before and After the Use of Medical Scribes
Florig, S. T., Corby, S., Devara, T., Weiskopf, N. G., Mohan, V., & Gold, J. A. (2022) · JAMA · doi:10.1001/jama.2022.13558 · Cited by 12 · cited 1×
“This mirrors previous studies with human scribes, where widespread adoption had little impact on after-hours charting or chart completion time [20-22].”
Self-citationComparing Scribed and Non-scribed Outpatient Progress Notes.
Rule, A., Florig, S. T., Bedrick, S. D., Mohan, V., Gold, J. A., & Hribar, M. R. (2021) · PubMed · Cited by 10
Self-citationImpact of Scribe Intervention on Documentation in an Outpatient Pediatric Primary Care Practice
Jhaveri, P., Abdulahad, D., Fogel, B. N., Chuang, C. H., Lehman, E. B., Chawla, L., Foley, K. A., Phillips, T. A., & Levi, B. H. (2021) · Academic Pediatrics · doi:10.1016/j.acap.2021.05.004 · Cited by 9 · cited 1×
“This mirrors previous studies with human scribes, where widespread adoption had little impact on after-hours charting or chart completion time [20-22].”
An Empirical Study of the Non-determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023) · arXiv · doi:10.48550/arxiv.2308.02828
Research works printed but not linked: 2
- Florig, ST, 2021, AMIA Annu Symp Proc, 2021, 457
- Rule, A, 2021, AMIA Annu Symp Proc, 2021, 1059
Cited by
Biswas, A., & Talukdar, W. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24may1483 · Cited by 1,028
Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review
Bracken, A., Reilly, C., Feeley, A., Sheehan, E., Merghani, K., & Feeley, I. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02157-4 · Cited by 113
“…has been reported with the use of the physician documentation quality instrument (PDQI-9 tool) [34] When compared to previous literature AI-driven documentation varied between “bad (29.7)” and “excellent (35.6)” highlighting the variability in documentation produced by AI systems [23, 25, 28].”
InfluentialThe Impact of AI Scribes on Streamlining Clinical Documentation: A Systematic Review
Sasseville, M., Yousefi, F., Ouellet, S., Naye, F., Stefan, T., Carnovale, V., Bergeron, F., Ling, L., Gheorghiu, B., Hagens, S., et al. (2025) · Healthcare · doi:10.3390/healthcare13121447 · Cited by 122
“One study featured simulated patient-provider encounters across various ambulatory specialties [34].”
InfluentialAI Scribes in Health Care: Balancing Transformative Potential With Responsible Integration
Leung, T. I., Coristine, A. J., & Benis, A. (2025) · JMIR Medical Informatics · doi:10.2196/80898 · Cited by 98
“Other evaluations of stand-alone tools for audio recordings suggest potentially high rates of errors, including incorrect information, omissions, and hallucinations [11,29,38].”
The effect of ambient artificial intelligence notes on provider burnout.
Misurac, J., Knake, L. A., & Blum, J. M. (2025) · Applied Clinical Informatics · doi:10.1055/a-2461-4576 · Cited by 58
“23 Thus, AI technologies must continue to evolve and adapt to the different clinical use cases and needs.”
Metadata
- Authors
- Kernberg, A., Gold, J. A., & Mohan, V.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Mohan, Vishnu
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-04-22
- Volume
- 26
- Pages
- e54419
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11074889
- OpenAlex
- W4392632345
- Semantic Scholar
- 78dee4159add8e48b50a2100104fbfae99d2185b
- Cited by
- 120
- References
- 27
- Citations by year
- 2026: 55 · 2025: 52 · 2024: 13
- FWCI
- 11.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Research and Theory › Nursing Diagnosis and Documentation
- Keywords
- ChatGPT-4, SOAP notes, clinical documentation, physician-patient encounters, audio recordings, note quality, medical scribes, physician burnout, large language models, history and physical examination, outpatient care
- MeSH terms
- Artificial Intelligence, Documentation, Humans, Physician-Patient Relations, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.