Assessing the performance of GPT-4 in the filed of osteoarthritis and orthopaedic case consultation
Open accessAbstract
Background: Large Language Models (LLMs) like GPT-4 demonstrate potential applications in diverse areas, including healthcare and patient education. This study evaluates GPT-4’s competency against osteoarthritis (OA) treatment guidelines from the United States and China and assesses its ability in diagnosing and treating orthopedic diseases. Methods: Data sources included OA management guidelines and orthopedic examination case questions. Queries were directed to GPT-4 based on these resources, and its responses were compared with the established guidelines and cases. The accuracy and completeness of GPT-4’s responses were evaluated using Likert scales, while case inquiries were stratified into four tiers of correctness and completeness. Results: GPT-4 exhibited strong performance in providing accurate and complete responses to OA management recommendations from both the American and Chinese guidelines, with high Likert scale scores for accuracy and completeness. It demonstrated proficiency in handling clinical cases, making accurate diagnoses, suggesting appropriate tests, and proposing treatment plans. Few errors were noted in specific complex cases. Conclusions: GPT-4 exhibits potential as an auxiliary tool in orthopedic clinical practice and patient education, demonstrating high accuracy and completeness in interpreting OA treatment guidelines and analyzing clinical cases. Further validation of its capabilities in real-world clinical scenarios is needed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cross, M., Smith, E. U. R., Hoy, D. G., Nolte, S., Ackerman, I. N., Fransen, M., Bridgett, L., Williams, S. R. M., Guillemin, F., Hill, C. L., et al. (2014) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2013-204763 · Cited by 3,510 · see citing works
Glyn-Jones, S., Palmer, A. J. R., Agricola, R., Price, A. J., Vincent, T. L., Weinans, H., & Carr, A. J. (2015) · The Lancet · doi:10.1016/s0140-6736(14)60802-3 · Cited by 3,151 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“Further, it has been employed as a preparatory tool for medical board examinations, thus evidencing its immense potential in education([6-7]).”
MethodDave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
Can artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 655 · see citing works
Non-surgical management of knee osteoarthritis: comparison of ESCEO and OARSI 2019 guidelines
Arden, N., Perry, T. A., Bannuru, R. R., Bruyère, O., Cooper, C. C., Haugen, I. K., Hochberg, M. C., McAlindon, T. E., Mobasheri, A., & Reginster, J. (2020) · Nature Reviews Rheumatology · doi:10.1038/s41584-020-00523-9 · Cited by 582 · see citing works
Brophy, R. H., & Fillingham, Y. A. (2022) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-21-01233 · Cited by 402 · see citing works
Zhang, H., Shao, Y. X., Yao, Z., Liu, L., Zhang, H., Yin, J., Xie, H., Li, K., Lai, P., Zeng, H., et al. (2022) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2021-221513 · Cited by 197 · see citing works
Temsah, O., Khan, S., Chaiah, Y., Senjab, A., Alhasan, K. A., Jamal, A. A., Aljamaan, F., Malki, K. H., Halwani, R., Al‐Tawfiq, J. A., et al. (2023) · Cureus · doi:10.7759/cureus.37281 · Cited by 182 · see citing works
Guidelines for the diagnosis and treatment of osteoarthritis in China (2019 edition)
Zhang, Z., Huang, C., Jiang, Q., Zheng, Y., Liu, Y., Liu, S., Chen, Y., Mei, Y., Ding, C., Chen, M., et al. (2020) · Annals of Translational Medicine · doi:10.21037/atm-20-4665 · Cited by 167 · see citing works
Evaluating GPT4 on Impressions Generation in Radiology Reports
Sun, Z., Ong, H., Kennedy, P., Tang, L., Chen, S., Elias, J. E., Lucas, E., Shih, G., & Peng, Y. (2023) · Radiology · doi:10.1148/radiol.231259 · Cited by 162 · see citing works
GPT-4 in Radiology: Improvements in Advanced Reasoning
Bhayana, R., Bleakney, R. R., & Krishna, S. (2023) · Radiology · doi:10.1148/radiol.230987 · Cited by 127 · see citing works
Bennell, K. L., Lawford, B. J., Keating, C., Brown, C. K., Kasza, J., Mackenzie, D., Metcalf, B., Kimp, A. J., Egerton, T., Spiers, L., et al. (2021) · Annals of Internal Medicine · doi:10.7326/m21-2388 · Cited by 106 · see citing works
Chen, N., Fong, D., & Wong, J. Y. H. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.50674 · Cited by 93 · see citing works
Ali, R., Tang, O. Y., Connolly, I. D., et al. (2023) · medRxiv · doi:10.1101/2023.04.06.23288265 · Cited by 79 · see citing works
Osteoarthritis in people over 16: diagnosis and management—updated summary of NICE guidance
Wood, G., Neilson, J., Cottrell, E., et al. (2023) · BMJ · doi:10.1136/bmj.p24 · Cited by 77 · see citing works
Artificial Intelligence in Sports Medicine: Could GPT-4 Make Human Doctors Obsolete?
Cheng, K., Guo, Q., He, Y., Lu, Y., Xie, R., Li, C., & Wu, H. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03213-1 · Cited by 59 · see citing works
Cheng, K., Li, Z., Li, C., Xie, R., Guo, Q., He, Y., & Wu, H. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03207-z · Cited by 50 · see citing works
Martel-Pelletier, J., Barr, A., Cicuttini, F., Conaghan, P., Cooper, C., Goldring, M. B., Goldring, S. R., Jones, G., Teichtahl, A., & Pelletier, J. (2016) · Nature Reviews Disease Primers · doi:10.1038/nrdp.2016.72 · Cited by 49 · see citing works
Kumah-Crystal, Y. A., Mankowitz, S. L., Embí, P. J., & Lehmann, C. U. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad104 · Cited by 45 · see citing works
OpenAI — explain why some countries are excluded from ChatGPT
Wang, S. H. (2023) · Nature · doi:10.1038/d41586-023-00553-9 · Cited by 27 · see citing works
Yoshiyasu, Y., Wu, F. M., Dhanda, A. K., Gorelik, D., Takashima, M., & Ahmed, O. G. (2023) · International Forum of Allergy & Rhinology · doi:10.1002/alr.23201 · Cited by 27 · see citing works
Daily briefing: What scientists think of GPT-4, the new AI chatbot
Graham, F. (2023) · Nature · doi:10.1038/d41586-023-00839-y · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 2017;30
- P. Lee, S. Bubeck, J. Petro. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. Reply. N Engl J Med. 2023;388(25):2400.
- Qing Lyu, Josh Tan, Michael E Zapadka, Janardhana Ponnatapura, Chuang Niu, Kyle J Myers, Ge Wang, Christopher T Whitlow. Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limitations, and potential. Visual Computing for Industry, Biomedicine, and Art. 2023;6(1):9.
- Sajan B Patel, Kyle Lam, Michael Liebrenz. ChatGPT: friend or foe. Lancet Digit. Health. 2023;5:e102.
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Large language models in medical and healthcare fields: applications, advances, and challenges
Wang, D., & Zhang, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10921-0 · Cited by 246 · see citing works
- Authors
- Li, J., Gao, X., Dou, T., Gao, Y., & Zhu, W.
- Affiliations
- First Hospital of China Medical University (CN)
- Fourth Affiliated Hospital of China Medical University (CN)
- Jinzhou Medical University (CN)
- Corresponding author
- Zhu, Wannan
- ORCID
- Gao, Yuyang
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-08-09
- OpenAlex
- W4385766957
- Cited by
- 4
- References
- 26
- Citations by year
- 2025: 2 · 2024: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- medRxiv (acceptedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Rehabilitation › Musculoskeletal Disorders and Rehabilitation
- Keywords
- GPT-4, large language models, osteoarthritis, orthopedic diagnosis, treatment guidelines, AI-assisted diagnosis, patient education, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
