Performance of ChatGPT compared to clinical practice guidelines in making informed decisions for lumbosacral radicular pain: a cross-sectional study.
Abstract
OBJECTIVE: To compare the accuracy of an artificial intelligence chatbot to clinical practice guidelines (CPGs) recommendations for providing answers to complex clinical questions on lumbosacral radicular pain. DESIGN: Cross-sectional study. METHODS: We extracted recommendations from recent CPGs for diagnosing and treating lumbosacral radicular pain. Relative clinical questions were developed and queried to OpenAI’s ChatGPT (GPT-3.5). We compared ChatGPT answers to CPGs recommendations by assessing the (1) internal consistency of ChatGPT answers by measuring the percentage of text wording similarity when a clinical question was posed 3 times, (2) reliability between 2 independent reviewers in grading ChatGPT answers, and (3) accuracy of ChatGPT answers compared to CPGs recommendations. Reliability was estimated using Fleiss’ kappa (κ) coefficients, and accuracy by interobserver agreement as the frequency of the agreements among all judgments. RESULTS: We tested 9 clinical questions. The internal consistency of text ChatGPT answers was unacceptable across all 3 trials in all clinical questions (mean percentage of 49%, standard deviation of 15). Intrareliability (reviewer 1: κ = 0.90, standard error [SE] = 0.09; reviewer 2: κ = 0.90, SE = 0.10) and interreliability (κ = 0.85, SE = 0.15) between the 2 reviewers was “almost perfect.” Accuracy between ChatGPT answers and CPGs recommendations was slight, demonstrating agreement in 33% of recommendations. CONCLUSION: ChatGPT performed poorly in internal consistency and accuracy of the indications generated compared to clinical practice guideline recommendations for lumbosacral radicular pain. J Orthop Sports Phys Ther 2024;54(3):222-228. Epub 29 January 2024. doi:10.2519/jospt.2024.12151
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several chatbots have experienced inconsistent accuracy by clinical practice guideline standards.
“Currently, few studies use clinical practice guidelines to define successful performance, and several chatbots have experienced inconsistent accuracy by these standards.168,169”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [169]
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
What low back pain is and why we need to pay attention
Hartvigsen, J., Hancock, M. J., Kongsted, A., Louw, Q., Ferreira, M. L., Genevay, S., Hoy, D., Karppinen, J., Pransky, G., Sieper, J., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)30480-x · Cited by 4,652 · see citing works
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465 · see citing works
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9606-8 · Cited by 4,260 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
Clinical Practice Guidelines We Can Trust
Board on Health Care Services (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,102 · see citing works
Ferreira, M. L., de Luca, K. E., Haile, L. M., Steinmetz, J. D., Culbreth, G. T., Cross, M., Kopec, J. A., Ferreira, P. H., Blyth, F., Buchbinder, R., et al. (2023) · The Lancet Rheumatology · doi:10.1016/s2665-9913(23)00098-x · Cited by 2,113 · see citing works
Sarker, I. H. (2022) · SN Computer Science · doi:10.1007/s42979-022-01043-x · Cited by 1,208 · see citing works
Dave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204 · see citing works
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · see citing works
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840 · see citing works
The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines
Brouwers, M. C., Kerkvliet, K., & Spithoff, K. (2016) · BMJ · doi:10.1136/bmj.i1152 · Cited by 812 · see citing works
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 655 · see citing works
Konstantinou, K., & Dunn, K. M. (2008) · Spine · doi:10.1097/brs.0b013e318183a4a2 · Cited by 549 · see citing works
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
Understanding the meaning of accuracy, trueness and precision
Menditto, A., Patriarca, M., & Magnusson, B. (2006) · Accreditation and Quality Assurance · doi:10.1007/s00769-006-0191-z · Cited by 342 · see citing works
Accessing Artificial Intelligence for Clinical Decision-Making
Giordano, C., Brennan, M., Mohamed, B., Rashidi, P., Modave, F., & Tighe, P. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.645232 · Cited by 326 · see citing works
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282 · see citing works
Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT
Haver, H. L., Ambinder, E. B., Bahl, M., Oluyemi, E. T., Jeudy, J., & Yi, P. H. (2023) · Radiology · doi:10.1148/radiol.230424 · Cited by 247 · see citing works
Health Information–Seeking Behavior of Seniors Who Use the Internet: A Survey
Medlock, S., Eslami, S., Askari, M., Arts, D. L., Sent, D., de Rooij, S. E., & Abu-Hanna, A. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3749 · Cited by 237 · see citing works
Large language model (ChatGPT) as a support tool for breast tumor board
Sorin, V., Klang, E., Sklair‐Levy, M., Cohen, I., Zippel, D. B., Lahat, N. B., Konen, E., & Barash, Y. (2023) · npj Breast Cancer · doi:10.1038/s41523-023-00557-8 · Cited by 216 · see citing works
O'Connell, N. E., Cook, C. E., Wand, B. M., & Ward, S. P. (2016) · Best Practice & Research Clinical Rheumatology · doi:10.1016/j.berh.2017.05.001 · Cited by 193 · see citing works
Self-citationDiagnosis and treatment of sciatica
Jensen, R. K., Kongsted, A., Kjaer, P., & Koes, B. (2019) · BMJ · doi:10.1136/bmj.l6273 · Cited by 193 · see citing works
Lennartz, S., Dratsch, T., Zopfs, D., Persigehl, T., Maintz, D., Große Hokamp, N., & Pinto dos Santos, D. (2021) · Journal of Medical Internet Research · doi:10.2196/24221 · Cited by 153 · see citing works
Gattrell, W. T., Hungin, A. P. S., Price, A. I., Winchester, C., Tovey, D. I., Hughes, E. L., van Zuuren, E. J., Goldman, K., Logullo, P., Matheis, R. J., et al. (2022) · Research Integrity and Peer Review · doi:10.1186/s41073-022-00122-0 · Cited by 115 · see citing works
Chervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.05.151 · Cited by 87 · see citing works
Accuracy of Information Provided by ChatGPT Regarding Liver Cancer Surveillance and Diagnosis
Cao, J. J., Kwon, D. H.-M., Ghaziani, T., Kwo, P. Y., Tse, G., Kesselman, A. J., Kamaya, A., & Tse, J. R. (2023) · American Journal of Roentgenology · doi:10.2214/ajr.23.29493 · Cited by 75 · see citing works
Khorami, A. K., Oliveira, C. B., Maher, C. G., Bindels, P., Machado, G. C., Pinto, R. Z., Koes, B. W., & Chiarotto, A. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10112482 · Cited by 73 · see citing works
Rossettini, G., Cook, C. E., Palese, A., Pillastrini, P., & Turolla, A. (2023) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2023.12000 · Cited by 62 · see citing works
Self-citationAssessing the Accuracy and Clinical Utility of ChatGPT in Laboratory Medicine
Muñoz‐Zuluaga, C., Zhao, Z., Wang, F., Greenblatt, M. B., & Yang, H. S. (2023) · Clinical Chemistry · doi:10.1093/clinchem/hvad058 · Cited by 61 · see citing works
Can ChatGPT Accurately Answer a PICOT Question? Assessing AI Response to a Clinical Question
Branum, C., & Schiavenato, M. (2023) · Nurse Educator · doi:10.1097/nne.0000000000001436 · Cited by 55 · see citing works
Quality of ChatGPT Responses to Questions Related To Liver Transplantation
Endo, Y., Sasaki, K., Moazzam, Z., Lima, H. A., Schenk, A. D., Limkemann, A. J. P., Washburn, W. K., & Pawlik, T. M. (2023) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-023-05714-9 · Cited by 42 · see citing works
Appropriateness of Recommendations Provided by ChatGPT to Interventional Radiologists
Barat, M., Soyer, P. A., & Dohan, A. J. J. (2023) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231170133 · Cited by 36 · see citing works
NICE low back pain guidelines: opportunities and obstacles to change practice
O’Sullivan, K., O’Keeffe, M., & O’Sullivan, P. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-097810 · Cited by 29 · see citing works
Generative AI could revolutionize health care — but not if control is ceded to big tech
Toma, A., Senkaiahliyan, S., Lawler, P. R., Rubin, B., & Wang, B. (2023) · Nature · doi:10.1038/d41586-023-03803-y · Cited by 29 · see citing works
Nowell, J. L. (2009) · BMJ · doi:10.1136/bmj.b450 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
GastroBot: a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation
Zhou, Q., Liu, C., Duan, Y., Sun, K., Li, Y., Kan, H., Gu, Z., Shu, J., & Hu, J. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1392555 · Cited by 44 · see citing works
Rossettini, G., Bargeri, S., Cook, C., Guida, S., Palese, A., Rodeghiero, L., Pillastrini, P., Turolla, A., Castellini, G., & Gianola, S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1574287 · Cited by 31 · see citing works
Self-citationTortella, F., Palese, A., Turolla, A., Castellini, G., Pillastrini, P., Landuzzi, M. G., Cook, C., Galeoto, G., Giovannico, G., Rodeghiero, L., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07176-w · Cited by 28 · see citing works
“Moreover, while several studies have investigated AI Chatbots in clinical practice [47, 48], research on their application in physiotherapy education remains obscure [4].”
Self-citationSafran, E., & Yildirim, S. (2025) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-025-08650-8 · Cited by 21 · see citing works
- Authors
- Gianola, S., Bargeri, S., Castellini, G., Cook, C., Palese, A., Pillastrini, P., Salvalaggio, S., Turolla, A., & Rossettini, G.
- ORCID
- Venue
- Journal of Orthopaedic & Sports Physical Therapy
- Published
- 2024-03
- Volume
- 54
- Issue
- 3
- Pages
- 222-228
- ISSN
- 0190-6011 (ISSN-L)
- 1938-1344
- OpenAlex
- W4391317566
- Cited by
- 62
- References
- 38
- Citations by year
- 2026: 21 · 2025: 34 · 2024: 7
- FWCI
- 5.87
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Orthopaedic and Sports Physical Therapy (publishedVersion)
- PubMed (publishedVersion)
- Institutional Research Information System (University of Udine) (submittedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- ChatGPT, large language models, lumbosacral radicular pain, internal consistency, inter-rater reliability, guideline adherence, artificial intelligence in clinical decision-making
- MeSH terms
- Artificial Intelligence, Back Pain, Cross-Sectional Studies, Decision Making, Humans, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
