CoChat
Sign Up
Cited by 24Artificial Intelligence in Healthcare and EducationMobile Health and mHealth ApplicationsArtificial Intelligence in Healthcare

Comparing Artificial Intelligence–Generated and Clinician-Created Personalized Self-Management Guidance for Patients With Knee Osteoarthritis: Blinded Observational Study

Open access
Du, Kai, Li, Ao, Zuo, Qi-Heng, Zhang, Chen-Yu, Guo, Ren, Chen, Ping, Du, Wei-Shuai, Li, Shu-Ming·2025·Journal of Medical Internet Research
DOI 10.2196/67830PMC PMC12096024

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Knee osteoarthritis is a prevalent, chronic musculoskeletal disorder that impairs mobility and quality of life. Personalized patient education aims to improve self-management and adherence; yet, its delivery is often limited by time constraints, clinician workload, and the heterogeneity of patient needs. Recent advances in large language models offer potential solutions. GPT-4 (OpenAI), distinguished by its long-context reasoning and adoption in clinical artificial intelligence research, emerged as a leading candidate for personalized health communication. However, its application in generating condition-specific educational guidance remains underexplored, and concerns about misinformation, personalization limits, and ethical oversight remain. Objective: We evaluated GPT-4’s ability to generate individualized self-management guidance for patients with knee osteoarthritis in comparison with clinician-created content. Methods: This 2-phase, double-blind, observational study used data from 50 patients previously enrolled in a registered randomized trial. In phase 1, 2 orthopedic clinicians each generated personalized education materials for 25 patient profiles using anonymized clinical data, including history, symptoms, and lifestyle. In phase 2, the same datasets were processed by GPT-4 using standardized prompts. All content was anonymized and evaluated by 2 independent, blinded clinical experts using validated scoring systems. Evaluation criteria included efficiency, readability (Flesch-Kincaid, Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook), accuracy, personalization, and comprehensiveness and safety. Disagreements between reviewers were resolved through consensus or third-party adjudication. Results: GPT-4 outperformed clinicians in content generation speed (530.03 vs 37.29 words per min, P<.001). Readability was better on the Flesch-Kincaid (mean 11.56, SD 1.08 vs mean 12.67 SD 0.95), Gunning Fog (mean 12.47, SD 1.36 vs mean 14.56, SD 0.93), and Simple Measure of Gobbledygook (mean 13.33, SD 1.00 vs mean 13.81 SD 0.69) indices (all P<.001), though GPT-4 scored slightly higher on the Coleman-Liau Index (mean 15.90, SD 1.03 vs mean 15.15, SD 0.91). GPT-4 also outperformed clinicians in accuracy (mean 5.31, SD 1.73 vs mean 4.76, SD 1.10; P=.05, personalization (mean 54.32, SD 6.21 vs mean 33.20, SD 5.40; P<.001), comprehensiveness (mean 51.74, SD 6.47 vs mean 35.26, SD 6.66; P<.001), and safety (median 61, IQR 58-66 vs median 50, IQR 47-55.25; P<.001). Conclusions: GPT-4 could generate personalized self-management guidance for knee osteoarthritis with greater efficiency, accuracy, personalization, comprehensiveness, and safety than clinician-generated content, as assessed using standardized, guideline-aligned evaluation frameworks. These findings underscore the potential of large language models to support scalable, high-quality patient education in chronic disease management. The observed lexical complexity suggests the need to refine outputs for populations with limited health literacy. As an exploratory, single-center study, these results warrant confirmation in larger, multicenter cohorts with diverse demographic profiles. Future implementation should be guided by ethical and operational safeguards, including data privacy, transparency, and the delineation of clinical responsibility. Hybrid models integrating artificial intelligence–generated content with clinician oversight may offer a pragmatic path forward.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/67830

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In musculoskeletal care, ChatGPT-generated guidance for patients with knee osteoarthritis was found to be more comprehensive than clinician-written advice, though more difficult to read.

    “In musculoskeletal care, ChatGPT-generated guidance for patients with knee osteoarthritis was found to be more comprehensive than clinician-written advice, though more difficult to read (17).”

    From ChatGPT for diabetes education: potential, accuracy, and accessibility in patient support (2026) · cites this paper as [17]

    The Discussion’s Principal Findings support the comprehensiveness comparison, but contradict the readability claim: GPT-4 guidance had generally lower reading difficulty than clinician guidance. The Results’ Readability Evaluation likewise report better GPT-4 scores on three of four metrics; clinician-written material had a slight advantage only on the Coleman-Liau Index. Page numbers are not provided in our copy. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. [1]Katz JN Arant KR Loeser RF Diagnosis and treatment of hip and knee osteoarthritis: a review JAMA 2021 325 6 568 578 10.1001/jama.2020.22171 33560326 2776205 33560326 PMC8225295 · DOI
  2. [2]Deyle GD Allen CS Allison SC Gill NW Hando BR Petersen EJ Dusenberry DI Rhon DI Physical therapy versus glucocorticoid injection for osteoarthritis of the knee N Engl J Med 2020 382 15 1420 1429 10.1056/NEJMoa1905877 32268027 32268027 · DOI
  3. [3]Perruccio AV Young JJ Wilfong JM Denise Power J Canizares M Badley EM Osteoarthritis Cartilage 2024 32 2 159 165 10.1016/j.joca.2023.11.012 38035975 S1063-4584(23)00990-1 38035975 · DOI
  4. [4]GBD 2021 Osteoarthritis Collaborators Global, regional, and national burden of osteoarthritis, 1990-2020 and projections to 2050: a systematic analysis for the global burden of disease study 2021 Lancet Rheumatol 2023 5 9 e508 e522 10.1016/S2665-9913(23)00163-7 37675071 S2665-9913(23)00163-7 37675071 PMC10477960 · DOI
  5. [5]Goff AJ De Oliveira Silva D Merolli M Bell EC Crossley KM Barton CJ Patient education improves pain and function in people with knee osteoarthritis with better effects when combined with exercise therapy: a systematic review J Physiother 2021 67 3 177 189 10.1016/j.jphys.2021.06.011 34158270 S1836-9553(21)00054-0 34158270 · DOI
  6. [6]Duong V Oo WM Ding C Culvenor AG Hunter DJ Evaluation and treatment of knee pain: a review JAMA 2023 330 16 1568 1580 10.1001/jama.2023.19675 37874571 2811009 37874571 · DOI
  7. [7]Sharma L Osteoarthritis of the Knee N Engl J Med 2021 384 1 51 59 10.1056/NEJMcp1903768 33406330 33406330 · DOI
  8. [8]Moseng T Vliet Vlieland TPM Battista S Beckwée D Boyadzhieva V Conaghan PG Costa D Doherty M Finney AG Georgiev T Gobbo M Kennedy N Kjeken I Kroon FPB Lohmander LS Lund H Mallen CD Pavelka K Pitsillidou IA Rayman MP Tveter AT Vriezekolk JE Wiek D Zanoli G Østerås N EULAR recommendations for the non-pharmacological core management of hip and knee osteoarthritis: 2023 update Ann Rheum Dis 2024 83 6 730 740 10.1136/ard-2023-225041 38212040 S0003-4967(24)00129-8 38212040 PMC11103326 · DOI
  9. [9]Oosterhaven J Pell CD Schröder CD Popma H Spierenburg L Devillé WLJM Wittink H Health literacy and pain neuroscience education in an interdisciplinary pain management programme: a qualitative study of patient perspectives Pain Rep 2023 8 6 e1093 10.1097/PR9.0000000000001093 37868618 PAINREPORTS-D-23-0011 37868618 PMC10586826 · DOI
  10. [10]Martel-Pelletier J Barr AJ Cicuttini FM Conaghan PG Cooper C Goldring MB Goldring SR Jones G Teichtahl AJ Pelletier J Osteoarthritis Nat Rev Dis Primers 2016 2 16072 10.1038/nrdp.2016.72 27734845 nrdp201672 27734845 · DOI
  11. [11]Hunter DJ Bierma-Zeinstra S Osteoarthritis Lancet 2019 393 10182 1745 1759 10.1016/S0140-6736(19)30417-9 31034380 S0140-6736(19)30417-9 31034380 · DOI
  12. [12]Salazar-Méndez J Cuyul-Vásquez I Ponce-Fuentes F Guzmán-Muñoz E Núñez-Cortés R Huysmans E Lluch-Girbés E Viscay-Sanhueza N Fuentes J Pain neuroscience education for patients with chronic pain: A scoping review from teaching-learning strategies, educational level, and cultural perspective Patient Educ Couns 2024 123 108201 10.1016/j.pec.2024.108201 38387389 S0738-3991(24)00068-5 38387389 · DOI
  13. [13]Moglia A Georgiou K Cerveri P Mainardi L Satava RM Cuschieri A Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test Artif Intell Rev 2024 57 9 231 10.1007/s10462-024-10849-5 · DOI
  14. [14]Chen X Xie H Tao X Wang FL Leng M Lei B Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics Artif Intell Rev 2024 03 15 57 4 91 10.1007/s10462-024-10712-7 · DOI
  15. [15]Jayakumar P Moore MG Furlough KA Uhler LM Andrawis JP Koenig KM Aksan N Rathouz PJ Bozic KJ Comparison of an artificial intelligence-enabled patient decision aid vs educational material on decision quality, shared decision-making, patient experience, and functional outcomes in adults with knee osteoarthritis: a randomized clinical trial JAMA Netw Open 2021 4 2 e2037107 10.1001/jamanetworkopen.2020.37107 33599773 2776550 33599773 PMC7893500 · DOI
  16. [16]Akyon SH Akyon FC Camyar AS Hızlı F Sari T Hızlı Evaluating the capabilities of generative AI tools in understanding medical papers: qualitative study JMIR Med Inform 2024 12 e59258 10.2196/59258 39230947 v12i1e59258 39230947 PMC11411230 · DOI
  17. [17]Plagwitz L Neuhaus P Yildirim K Losch N Varghese J Büscher A Zero-shot LLMs for named entity recognition: targeting cardiac function indicators in German clinical texts Stud Health Technol Inform 2024 317 228 234 10.3233/SHTI240861 39234726 SHTI240861 39234726 · DOI
  18. [19]Liu J Shen H Chen K Li X Large language model produces high accurate diagnosis of cancer from end-motif profiles of cell-free DNA Brief Bioinform 2024 25 5 bbae430 10.1093/bib/bbae430 39222060 7747593 39222060 PMC11367762 · DOI
  19. [20]Ruiz Sarrias O Martínez Del Prado MP Sala Gonzalez Azcuna Sagarduy J Casado Cuesta P Figaredo Berjano C Galve-Calvo E López de San Vicente Hernández B López-Santillán M Nuño Escolástico M Sánchez Togneri L Sande Sardina L Pérez Hoyos MT Abad Villar MT Zabalza Zudaire M Sayar Beristain O Leveraging large language models for precision monitoring of chemotherapy-induced toxicities: a pilot study with expert comparisons and future directions Cancers (Basel) 2024 16 16 2830 10.3390/cancers16162830 39199603 cancers16162830 39199603 PMC11352281 · DOI
  20. [21]Yeo YH Samaan JS Ng WH Ting P Trivedi H Vipani A Ayoub W Yang JD Liran O Spiegel B Kuo A Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma Clin Mol Hepatol 2023 29 3 721 732 10.3350/cmh.2023.0089 36946005 cmh.2023.0089 36946005 PMC10366809 · DOI
  21. [22]Benary M Wang XD Schmidt M Soll D Hilfenhaus G Nassir M Sigler C Knödler M Keller U Beule D Keilholz U Leser U Rieke DT Leveraging large language models for decision support in personalized oncology JAMA Netw Open 2023 6 11 e2343689 10.1001/jamanetworkopen.2023.43689 37976064 2812097 37976064 PMC10656647 · DOI · Paper page
  22. [23]Sarraju A Bruemmer D Van Iterson E Cho L Rodriguez F Laffin L Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model JAMA 2023 329 10 842 844 10.1001/jama.2023.1044 36735264 2801244 36735264 PMC10015303 · DOI
  23. [24]Romano MF Shih LC Paschalidis IC Au R Kolachalama VB Large language models in neurology research and future practice Neurology 2023 101 23 1058 1067 10.1212/WNL.0000000000207967 37816646 WNL.0000000000207967 37816646 PMC10752640 · DOI
  24. [25]Beaulieu-Jones BR Berrigan MT Shah S Marwaha JS Lai S Brat GA Evaluating capabilities of large language models: performance of GPT-4 on surgical knowledge assessments Surgery 2024 175 4 936 942 10.1016/j.surg.2023.12.014 38246839 S0039-6060(23)00954-6 38246839 PMC10947829 · DOI · Paper page
  25. [28]Denecke K May R LLMHealthGroup Rivera Romero O Potential of large language models in health care: Delphi study J Med Internet Res 2024 26 e52399 10.2196/52399 38739445 v26i1e52399 38739445 PMC11130776 · DOI · Paper page
  26. [30]Flesch R A new readability yardstick J Appl Psychol 1948 32 3 221 233 10.1037/h0057532 18867058 18867058 · DOI · Paper page
  27. [31]Gunning R The Technique of Clear Writing 1952 New York, NY McGraw-Hill
  28. [32]Coleman M Liau TL A computer readability formula designed for machine scoring J Appl Psychol 1975 60 2 283 284 10.1037/h0076540 · DOI
  29. [34]Kolasinski SL Neogi T Hochberg MC Oatis C Guyatt G Block J Callahan L Copenhaver C Dodge C Felson D Gellar K Harvey WF Hawker G Herzig E Kwoh CK Nelson AE Samuels J Scanzello C White D Wise B Altman RD DiRenzo D Fontanarosa J Giradi G Ishimori M Misra D Shah AA Shmagel AK Thoma LM Turgunbaev M Turner AS Reston J 2019 American college of rheumatology/arthritis foundation guideline for the management of osteoarthritis of the hand, hip, and knee Arthritis Care Res (Hoboken) 2020 72 2 149 162 10.1002/acr.24131 31908149 31908149 PMC11488261 · DOI · Paper page
  30. [35]Bannuru RR Osani MC Vaysbrot EE Arden NK Bennell K Bierma-Zeinstra SMA Kraus VB Lohmander LS Abbott JH Bhandari M Blanco FJ Espinosa R Haugen IK Lin J Mandl LA Moilanen E Nakamura N Snyder-Mackler L Trojian T Underwood M McAlindon TE OARSI guidelines for the non-surgical management of knee, hip, and polyarticular osteoarthritis Osteoarthritis Cartilage 2019 27 11 1578 1589 10.1016/j.joca.2019.06.011 31278997 S1063-4584(19)31116-1 31278997 · DOI
  31. [36]Bruyère O Honvo G Veronese N Arden NK Branco J Curtis EM Al-Daghri NM Herrero-Beaumont G Martel-Pelletier J Pelletier J Rannou F Rizzoli R Roth R Uebelhart D Cooper C Reginster J An updated algorithm recommendation for the management of knee osteoarthritis from the European society for clinical and economic aspects of osteoporosis, osteoarthritis and musculoskeletal diseases (ESCEO) Semin Arthritis Rheum 2019 49 3 337 350 10.1016/j.semarthrit.2019.04.008 31126594 S0049-0172(19)30043-5 31126594 · DOI
  32. [37]Richmond J Hunter D Irrgang J Jones MH Snyder-Mackler L Van Durme D Rubin C Matzkin EG Marx RG Levy BA Watters WC Goldberg MJ Keith M Haralson RH Turkelson CM Wies JL Anderson S Boyer K Sluka P St Andre J McGowan R American Academy of Orthopaedic Surgeons American academy of orthopaedic surgeons clinical practice guideline on the treatment of osteoarthritis (OA) of the knee J Bone Joint Surg Am 2010 92 4 990 993 10.2106/JBJS.I.00982 20360527 92/4/990 20360527 · DOI
  33. [38]King LK Stanaitis I Hung V Koppikar S Waugh EJ Lipscombe L Hawker GA National institute of health and care excellence clinical criteria for the diagnosis of knee osteoarthritis: a prospective diagnostic accuracy study in individuals with type 2 diabetes Arthritis Care Res (Hoboken) 2024 10.1002/acr.25464 39542837 PMC12038217 39542837 · DOI
  34. [39]Nasa P Jain R Juneja D Delphi methodology in healthcare research: how to decide its appropriateness World J Methodol 2021 11 4 116 129 10.5662/wjm.v11.i4.116 34322364 34322364 PMC8299905 · DOI · Paper page
  35. [40]Goff AJ De Oliveira Silva D Ezzat AM Crossley KM Pazzinatto MF Barton CJ Lived Experience GroupHealth Professional Group Co-design of the web-based 'My Knee' education and self-management toolkit for people with knee osteoarthritis Digit Health 2023 9 20552076231163810 10.1177/20552076231163810 37009308 10.1177_20552076231163810 37009308 PMC10052584 · DOI
  36. [41]Mathur S Sutton J Personalized medicine could transform healthcare Biomed Rep 2017 7 1 3 5 10.3892/br.2017.922 28685051 BR-0-0-922 28685051 PMC5492710 · DOI
  37. [42]Kirkman M Sevdalis N Arora S Baker P Vincent C Ahmed M The outcomes of recent patient safety education interventions for trainee physicians and medical students: a systematic review BMJ Open 2015 5 5 e007705 10.1136/bmjopen-2015-007705 25995240 bmjopen-2015-007705 PMC4442206 25995240 · DOI
  38. [43]Li SM Li TL Guo R Chen P Du WS Kang SB Yan MZ Cheng WZ Effectiveness and safety of acupotomy for knee osteoarthritis: study protocol for a randomized controlled trial Trials 2021 22 1 824 10.1186/s13063-021-05786-5 34801083 10.1186/s13063-021-05786-5 34801083 PMC8606077 · DOI
  39. [44]Song H Xia Y Luo Z Liu H Song Y Zeng X Li T Zhong G Li J Chen M Zhang G Xiao B Evaluating the performance of different large language models on health consultation and patient education in urolithiasis J Med Syst 2023 47 1 125 10.1007/s10916-023-02021-3 37999899 10.1007/s10916-023-02021-3 37999899 · DOI
  40. [45]Robinson MA Belzberg M Thakker S Bibee K Merkel E MacFarlane DF Lim J Scott JF Deng M Lewin J Soleymani D Rosenfeld D Liu R Liu TYA Ng E Assessing the accuracy, usefulness, and readability of artificial-intelligence-generated responses to common dermatologic surgery questions for patient education: a double-blinded comparative study of ChatGPT and google bard J Am Acad Dermatol 2024 90 5 1078 1080 10.1016/j.jaad.2024.01.037 38296195 S0190-9622(24)00138-5 38296195 · DOI
  41. [46]Srinivasan N Samaan JS Rajeev ND Kanu MU Yeo YH Samakar K Large language models and bariatric surgery patient education: a comparative readability analysis of GPT-3.5, GPT-4, Bard, and online institutional resources Surg Endosc 2024 38 5 2522 2532 10.1007/s00464-024-10720-2 38472531 10.1007/s00464-024-10720-2 38472531 PMC11078810 · DOI
  42. [47]Alessandri-Bonetti M Liu HY Palmesano M Nguyen VT Egro FM Online patient education in body contouring: a comparison between google and chatGPT J Plast Reconstr Aesthet Surg 2023 87 390 402 10.1016/j.bjps.2023.10.091 37939643 S1748-6815(23)00642-3 37939643 · DOI
  43. [48]Wang J Shi R Le Q Shan K Chen Z Zhou X He Y Hong J Evaluating the effectiveness of large language models in patient education for conjunctivitis Br J Ophthalmol 2025 109 2 185 191 10.1136/bjo-2024-325599 39214677 bjo-2024-325599 39214677 · DOI
  44. [49]Dihan Q Chauhan MZ Eleiwa TK Brown AD Hassan AK Khodeiry MM Elsheikh RH Oke I Nihalani BR VanderVeen DK Sallam AB Elhusseiny AM Large language models: a new frontier in paediatric cataract patient education Br J Ophthalmol 2024 108 10 1470 1476 10.1136/bjo-2024-325252 39174290 bjo-2024-325252 39174290 · DOI
  45. [50]Huang Z Bianchi F Yuksekgonul M Montine TJ Zou J A visual-language foundation model for pathology image analysis using medical Twitter Nat Med 2023 29 9 2307 2316 10.1038/s41591-023-02504-3 37592105 10.1038/s41591-023-02504-3 37592105 · DOI · Paper page

Cited by

Sorted by citations

Authors
Du, K., Li, A., Zuo, Q.-H., Zhang, C.-Y., Guo, R., Chen, P., Du, W.-S., & Li, S.-M.
Affiliations
  • Beijing University of Chinese Medicine (CN)
  • Beijing Hospital of Traditional Chinese Medicine (CN)
Corresponding author
Li, Shu-Ming
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2025-05-07
Volume
27
Pages
e67830
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/67830
PMC
PMC12096024
OpenAlex
W4408999757
Cited by
24
References
6
Citations by year
2026: 17 · 2025: 7
FWCI
5.78
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
knee osteoarthritis, GPT-4, large language models, personalized patient education, readability, content personalization, patient education materials, orthopedic patients, patient-provider communication, misinformation risk, medication adherence, patient satisfaction, patient-reported outcomes, patient-reported quality of life
MeSH terms
Self-Management, Generative Artificial Intelligence, Aged, Artificial Intelligence, Double-Blind Method, Female, Humans, Male, Middle Aged, Patient Education as Topic, Osteoarthritis, Knee, Precision Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review