CoChat
Sign Up
Cited by 27Artificial Intelligence in Healthcare and EducationElectronic Health Records SystemsMachine Learning in Healthcare

Generative artificial intelligence writing open notes: a mixed methods assessment of the functionality of GPT 3.5 and GPT 4.0

Open access
Kharko, Anna, McMillan, Brian, Hagström, Josefin, Muli, Irene, Davidge, Gail, Hägglund, Maria, Blease, Charlotte·2024·DIGITAL HEALTH
DOI 10.1177/20552076241291384PMC PMC11528788
Open this paper in CoChat — free account

Abstract

Background: Worldwide, patients are increasingly being offered access to their full online clinical records including the narrative reports written by clinicians (so-called “open notes”). Against these developments, there is growing interest in the use of generative artificial intelligence (AI) such as OpenAI's ChatGPT to co-assist clinicians with patient-facing documentation. Objective: This study aimed to explore the effectiveness of OpenAI's ChatGPT 3.5 and GPT 4.0 in generating three patient-facing clinical notes from fictional general practice narrative reports. Methods: On 1 October 2023 and 1 November 2023, we used ChatGPT 3.5 and 4.0 to generate notes for three validated fictional general practice notes, using a prompt in the style of a British primary care note for three commonly presented conditions: (1) type 2 diabetes, (2) major depressive disorder, and (3) a differential diagnosis for suspected bowel cancer. Outputs were analyzed for reading ease, sentiment analysis, empathy, and medical fidelity. Results: ChatGPT 3.5 and 4.0 wrote longer notes than the original, and embedded more second person pronouns, with ChatGPT 3.5 scoring higher on both. ChatGPT expanded abbreviations, but readability metrics showed that the notes required a higher reading proficiency, with ChatGPT 3.5 demanding the most advanced level. Across all notes, ChatGPT offered higher signatures of empathy across cognitive, compassion/sympathy, and prosocial cues. Medical fidelity ratings varied across all three cases with ChatGPT 4.0 rated superior. Conclusions: While ChatGPT improved sentiment and empathy metrics in the transformed notes, compared to the original they also required higher reading proficiency and omitted details impacting medical fidelity.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/20552076241291384

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI-powered notes written by OpenAI's ChatGPT 3.5 and GPT 4.0 have been found to be more personalized and empathetic.

    “When it comes to clinical documentation, AI-powered notes written by OpenAI’s ChatGPT 3.5 and GPT 4.0 have been found to be more personalized and empathetic [42].”

    From Empathic AI for Patient-Centered Cancer Care: A Scoping Review of Patient Navigation, Support, and Clinical Practice (2026) · cites this paper as [42]

    The Abstract (Conclusions) reports improved empathy metrics in notes generated by ChatGPT 3.5 and 4.0, and the Discussion (Main findings) describes substantially more empathy-signaling language than in the original notes. A GP quoted in Results (Medical fidelity) said one note picked out what interested the patient, but the paper does not establish that the generated notes were more personalized than the originals. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. [1]Blease C Salmi L Rexhepi H, et al. Patients, clinicians and open notes: information blocking as a case of epistemic injustice. J Med Ethics 2021; 48: 785– 793. 33990427 PMC9554023
  2. [2]Kharko A Blease C Johansen M, et al. Mapping patients’ online record access worldwide: preliminary results from an International survey of healthcare experts. In: MEDINFO 2023—the future is accessible. Amsterdam, Netherlands: IOS Press; 2024, [cited 2024 Apr 15]. pp.114–118. Available from: https://ebooks.iospress.nl/volumearticle/66295 38269776
  3. [3]Salmi L Blease C Hägglund M, et al. US policy requires immediate release of records to patients. London, UK: British Medical Journal Publishing Group, 2021. 33602667
  4. [4]Hägglund M McMillan B Whittaker R. Blease C. Patient empowerment through online access to health records. Br Med J 2022; 378. PMC9518004 36175012
  5. [6]Kujala S Hörhammer I Väyrynen A, et al. Patients’ experiences of web-based access to electronic health records in Finland: cross-sectional survey. J Med Internet Res 2022; 24: e37438. PMC9210208 35666563
  6. [7]Hägglund M Scandurra I. Patients’ online access to electronic health records: current status and experiences from the implementation in Sweden. In: Medinfo. Amsterdam, Netherlands: IOS Press, 2017, pp. 723– 727. 29295193
  7. [9]McMillan B Eastham R Brown B, et al. Primary care patient records in the United Kingdom: past, present, and future research priorities. J Med Internet Res 2018; 20: e11293. PMC6315263 30567695
  8. [10]Petersson L Erlingsdóttir G. Open notes in Swedish psychiatric care (part 2): survey among psychiatric care professionals. JMIR Ment Health 2018; 5: e10521. PMC6035347 29929946
  9. [11]Dobscha SK Denneson LM Jacobson LE, et al. VA Mental health clinician experiences and attitudes toward OpenNotes. Gen Hosp Psychiatry 2016; 38: 89– 93. 26380876
  10. [12]DesRoches CM Leveille S Bell SK, et al. The views and experiences of clinicians sharing medical record notes with patients. JAMA Network Open 2020; 3: e201753– e201753. PMC12124735 32219406
  11. [13]Meier-Diedrich E Davidge G Hägglund M, et al. Changes in documentation due to patient access to electronic health records: protocol for a scoping review. JMIR Res Protoc 2023; 12: e46722. PMC10495856 37639298
  12. [14]Blease CR Torous J Hägglund M. Opinion: does patient access to clinical notes change documentation? Front Public Health 2020; 8: 578. PMC7728689 33330320
  13. [15]Muli I Scandurra I Cajander Å, et al. Healthcare professionals’ experiences of the work environment after patients’ access to their electronic health records–a qualitative study in primary care. In: Challenges of trustable AI and added-value on health. Amsterdam, Netherlands: IOS Press, 2022, pp. 530– 534. 35612136
  14. [16]Blease C McMillan B Salmi L, et al. Adapting to transparent medical records: International experience with “open notes”. Br Med J 2022; 379. 36410770
  15. [17]Ralston JD Yu O Penfold RB, et al. Changes in clinician attitudes toward sharing visit notes: surveys pre-and post-implementation. J Gen Intern Med 2021; 36: 3330– 3336. 33886028 PMC8061150
  16. [18]Bernstein J. Not the last word: seeing ourselves as doctors see US. Clin Orthop Relat Res 2022; 480: 1653– 1656. 35916690 PMC9384933
  17. [19]Blease C Torous J McMillan B, et al. Generative language models and open notes: exploring the promise and limitations. JMIR Med Educ 2024; 10: e51183. PMC10797501 38175688
  18. [20]Blease C. Open AI meets open notes: surveillance capitalism, patient privacy and online record access. J Med Ethics 2024; 50: 84– 89. 38050159 PMC10850625
  19. [21]Rosenberg GS Magnéli M Barle N, et al. ChatGPT-4 generates orthopedic discharge documents faster than humans maintaining comparable quality: a pilot study of 6 cases. Acta Orthop 2024; 95: 152. 38597205 PMC10959013
  20. [22]Lee P Goldberg C Kohane I. The AI revolution in medicine: GPT-4 and beyond. London, UK: Pearson Education, Limited, 2023.
  21. [26]Blease C Locher C Gaab J, et al. Generative artificial intelligence in primary care: an online survey of UK general practitioners. BMJ Health Care Inform. 2024; 31. doi: 10.1136/bmjhci-2024-101102 PMC11429366 39288998 · DOI
  22. [28]Cohen IG. What Should ChatGPT Mean for Bioethics? Available at SSRN 4430100. 2023.
  23. [30]Dijkstra AF Verdonk P Lagro-Janssen AL. Gender bias in medical textbooks: examples from coronary heart disease, depression, alcohol abuse and pharmacology. Med Educ 2008; 42: 1021– 1028. 18761614
  24. [31]Duma N Vera Aguilera J Paludo J, et al. Representation of minorities and women in oncology clinical trials: review of the past 14 years. J. Oncol. Pract. 2018; 14: e1– e10. 29099678
  25. [32]Geller SE Koch A Pellettieri B, et al. Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: have we made progress? J Womens Health (Larchmt) 2011; 20: 315– 320. 21351877 PMC3058895
  26. [33]Watts G. Why the exclusion of older people from clinical research must stop. Br Med J 2012; 344: e3445– e3445. 22613873
  27. [34]Bourgeois FT Olson KL Tse T, et al. Prevalence and characteristics of interventional trials conducted exclusively in elderly persons: a cross-sectional analysis of registered clinical trials. PLoS ONE 2016; 11: e0155948. doi: 10.1371/journal.pone.0155948 PMC4873036 27196289 · DOI
  28. [35]Zack T Lehman E Suzgun M, et al. Coding inequity: assessing GPT-4’s potential for perpetuating racial and gender biases in healthcare. medRxiv 2023: 2023–2007.
  29. [38]Goodman KE Paul HY Morgan DJ. AI-generated clinical summaries require more than accuracy. JAMA. 2024 [cited 2024 Feb 29]; Available from: https://jamanetwork.com/journals/jama/article-abstract/2814609 38285439
  30. [41]Nguyen J Pepping CA. The application of ChatGPT in healthcare progress notes: a commentary from a clinical and research perspective. Clin Trans Med 2023; 13: e1324. PMC10315641 37394880
  31. [42]Biswas S. ChatGPT and the future of medical writing. Radiology 2023; 307: e223312. 36728748
  32. [43]Marchandot B Matsushita K Carmona A, et al. ChatGPT: the next frontier in academic writing for cardiologists or a pandora’s box of ethical dilemmas. Eur Heart J Open 2023; 3: oead007. PMC10006694 36915398
  33. [44]Hanna JJ Wakene AD Lehmann CU, et al. Assessing racial and ethnic bias in text generation for healthcare-related tasks by ChatGPT. medRxiv 2023: 2023–2008.
  34. [45]Baker HP Dwyer E Kalidoss S, et al. ChatGPT’s ability to assist with clinical documentation: a randomized controlled trial. J Am Acad Orthop Surg 2022; 32: 123– 129. 37976385
  35. [47]Marks M Haupt CE. AI Chatbots, health privacy, and challenges to HIPAA compliance. JAMA 2023; 330: 309– 310. 37410450
  36. [48]Chen L Zaharia M Zou J. How is ChatGPT’s behavior changing over time?. arXiv; 2023 [cited 2024 Apr 9]. Available from: http://arxiv.org/abs/2307.09009 · arXiv
  37. [51]Mohammad SM Turney PD. Crowdsourcing a word–emotion association lexicon. Comput Intell 2013; 29: 436– 465.
  38. [53]Hall JA Schwartz R Duong F, et al. What is clinical empathy? Perspectives of community members, university students, cancer patients, and physicians. Patient Educ Couns 2021; 104: 1237– 1245. 33234440
  39. [54]Gerger H Munder T Kreuzer N, et al. Lay perspectives on empathy in patient-physician communication: an online experimental study. Health Commun 2023; 39: 1246– 1255. 37219394
  40. [55]Decety J Jackson PL. A social-neuroscience perspective on empathy. Curr Dir Psychol Sci 2006; 15: 54– 58.
  41. [57]Shamay-Tsoory SG Aharon-Peretz J Perry D. Two systems for empathy: a double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain 2009; 132: 617– 627. 18971202
  42. [60]Keselman A Smith CA. A classification of errors in lay comprehension of medical documents. J Biomed Inform 2012; 45: 1151– 1163. 22925723 PMC3504163
  43. [61]Gotlieb R Praska C Hendrickson MA, et al. Accuracy in patient understanding of common medical phrases. JAMA Network Open 2022; 5: e2242972– e2242972. PMC9713608 36449293
  44. [63]Pradhan F Fiedler A Samson K, et al. Artificial intelligence compared with human-derived patient educational materials on cirrhosis. Hepatol Commun 2024; 8: e0367. PMC10871753 38358382
  45. [65]Allen JW Earp BD Koplin J, et al. Consent-GPT: is it ethical to delegate procedural consent to conversational AI? J Med Ethics 2024; 50: 77– 83. 37898550 PMC10850653
  46. [68]Minssen T Vayena E Cohen IG. The challenges for regulating medical use of ChatGPT and other large language models. JAMA 2023; 330: 315– 316. 37410482
  47. [75]AMA. ChatGPT and generative AI: what physicians should consider. American Medical Association. 2023 [cited 2023 Sep 11]. Available from: https://www.ama-assn.org/system/files/chatgpt-what-physicians-should-consider.pdf
  48. [76]Smith H Downer J Ives J. Clinicians and AI use: where is the professional guidance? J Med Ethics. 2024; 50: 437– 441. 37607805 PMC11228205
  49. [78]Sharma A Lin IW Miner AS, et al. Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nat Mach Intell. 2023; 5: 46– 57.
  50. [79]Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res 2023; 25: e50638. PMC10585440 37792434
  51. [80]Jongsma KR Sand M Milota M. Why we should not mistake accuracy of medical AI for efficiency. NPJ Digit Med 2024; 7: 57. 38438477 PMC10912629
  52. [81]Wachter RM Brynjolfsson E. Will generative artificial intelligence deliver on its promise in health care? JAMA 2024; 331: 65– 69. 38032660

Cited by

Sorted by citations

Authors
Kharko, A., McMillan, B., Hagström, J., Muli, I., Davidge, G., Hägglund, M., & Blease, C.
Affiliations
  • Uppsala University (SE)
  • University of Manchester (GB)
Corresponding author
Blease, Charlotte
ORCID
Venue
DIGITAL HEALTH
Publisher
SAGE Publications
Published
2024-01
Volume
10
Pages
20552076241291384-20552076241291384
ISSN
2055-2076 (ISSN-L)
DOI
10.1177/20552076241291384
PMC
PMC11528788
OpenAlex
W4403878225
Cited by
27
References
8
Citations by year
2026: 9 · 2025: 17 · 2024: 1
FWCI
2.52
Type
journal article
Language
en
Open access
Licence
Locations
  • Digital Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Explorer (The University of Manchester) (publishedVersion, cc-by)
  • Uppsala University Publications (Uppsala University) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • Keane Scholar Award (N/A)
  • Forskningsrådet om Hälsa, Arbetsliv och Välfärd (Beyond Implementation of eHealth)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
generative AI, ChatGPT, open notes, primary care documentation, empathy, readability, sentiment analysis, type 2 diabetes, major depressive disorder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review