Assessment of OpenEvidence responses to American Academy of Orthopaedic Surgeons clinical practice guidelines for anterior cruciate ligament reconstruction
Open accessAbstract
INTRODUCTION/OBJECTIVES: General-purpose large language models (LLMs) have substantial limitations, including fabricated references and inconsistent concordance with evidence-based clinical practice guidelines (CPGs). OpenEvidence (OE) (Cambridge, MA) is a retrieval-augmented generation (RAG) platform designed to provide accurate, evidence-based, citation-supported answers. This study assessed the concordance, quality, and citation characteristics of OE responses to the 2022 American Academy of Orthopaedic Surgeons (AAOS) CPG for anterior cruciate ligament reconstruction (ACLR). It was hypothesized that OE would deliver relevant, accurate, clear, complete, and evidence-based responses, with low rates of hallucination and irrelevant citations. METHODS: Fifteen prompts reflecting the 2022 AAOS ACLR CPG were curated and categorized into three domains: (1) preoperative evaluation and diagnosis, (2) surgical timing and technique, and (3) rehabilitation and prevention. Each prompt was entered into OE in a new session to minimize context bias. Two orthopedic surgeons rated response quality on a 5-point Likert scale for relevance, accuracy, clarity, completeness, and evidence-based content. Inter-rater reliability was assessed using intra-class correlation coefficients (ICCs). All citations were extracted and classified as relevant, irrelevant, or hallucinated, and journal sources and publication years were recorded. RESULTS: Inter-rater reliability was high across all domains (overall ICC, 0.90; 95% confidence interval, 0.82-0.96). Median scores exceeded 4.0 for all domains, with perfect scores for evidence-based (5.0 [5.0-5.0]) and the lowest scores for clarity (5.0 [4.0-5.0]). Across 15 prompts, OE generated 127 PubMed-indexed citations (mean, 8.5 per response; range 6-11) with zero hallucinated and only 2.4% irrelevant citations. Citations spanned 29 journals from 1997 to 2025. CONCLUSION: OE produced highly relevant, accurate, clear, complete, and evidence-based responses to AAOS ACLR CPG prompts, with trustworthy PubMed-indexed citations and no hallucinations. Its RAG architecture may offer a credible, citation-oriented adjunct to evidence-based decision support in orthopedic surgery. LEVEL OF EVIDENCE: Level V, descriptive study.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C. (2024) · Journal of Medical Internet Research · doi:10.2196/53164 · Cited by 413 · see citing works
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267 · see citing works
Jeong, M., Sohn, J., Sung, M., & Kang, J. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae238 · Cited by 132 · see citing works
Exploring the Potential of ChatGPT as a Supplementary Tool for Providing Orthopaedic Information
Kaarre, J., Feldt, R., Keeling, L. E., Dadoo, S., Zsidai, B., Hughes, J. D., Samuelsson, K., & Musahl, V. (2023) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-023-07529-2 · Cited by 117 · see citing works
ChatGPT and large language models in orthopedics: from education and surgery to research
Chatterjee, S., Bhattacharya, M., Pal, S., Lee, S., & Chakraborty, C. (2023) · Journal of Experimental Orthopaedics · doi:10.1186/s40634-023-00700-1 · Cited by 82 · see citing works
Hurt, R. T., Stephenson, C. R., Gilman, E. A., Aakre, C. A., Croghan, I. T., Mundi, M. S., Ghosh, K., & Edakkanambeth Varayil, J. (2025) · Journal of Primary Care & Community Health · doi:10.1177/21501319251332215 · Cited by 78 · see citing works
Brophy, R. H., & Lowry, K. J. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-22-01020 · Cited by 70 · see citing works
Fahy, S., Oehme, S., Milinkovic, D. D., Jung, T., & Bartek, B. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14010104 · Cited by 65 · see citing works
Self-citationWoo, J. J., Yang, A. J., Olsen, R. J., Hasan, S. S., Nawabi, D. H., Nwachukwu, B. U., Williams, R. J., & Ramkumar, P. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.10.042 · Cited by 64 · see citing works
Yang, J., Ardavanis, K. S., Slack, K. E., Fernando, N. D., Valle, C. J. D., & Hernandez, N. M. (2024) · The Journal of Arthroplasty · doi:10.1016/j.arth.2024.01.029 · Cited by 53 · see citing works
Johns, W. L., Martinazzi, B. J., Miltenberg, B. H., Nam, H. H., & Hammoud, S. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.01.017 · Cited by 52 · see citing works
Nwachukwu, B. U., Varady, N. H., Allen, A. A., Dines, J. S., Altchek, D. W., Williams, R. J., & Kunze, K. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.07.040 · Cited by 42 · see citing works
Gültekin, O., Inoue, J., Yılmaz, B., Çerçi, M. H., Kılınç, B. E., Yilmaz, H., Prill, R., & Kayaalp, M. E. (2025) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.12711 · Cited by 41 · see citing works
Dagher, T., Dwyer, E., Baker, H. P., Kalidoss, S., & Strelzow, J. (2024) · Clinical Orthopaedics and Related Research · doi:10.1097/corr.0000000000003234 · Cited by 40 · see citing works
Kunze, K. N., Nwachukwu, B. U., Cote, M. P., & Ramkumar, P. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.12.010 · Cited by 35 · see citing works
Quinn, M. S., Milner, J., Schmitt, P. R., Morrissey, P. J., Lemme, N. J., Marcaccio, S. E., DeFroda, S. F., Tabaddor, R. R., & Owens, B. D. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.09.020 · Cited by 24 · see citing works
Nian, P. P., Umesh, A., Jones, R. H., Adhiyaman, A., Williams, C. J., Goodbody, C. M., Heyer, J. H., & Doyle, S. M. (2024) · Journal of the Pediatric Orthopaedic Society of North America · doi:10.1016/j.jposna.2024.100135 · Cited by 14 · see citing works
Megafu, M. N., Guerrero, O., Yendluri, A., Parsons, B. O., Galatz, L. M., Li, X., Kelly, J. D. A., & Parisien, R. L. (2025) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2025.01.039 · Cited by 10 · see citing works
Transforming orthopaedics with AI: Insights from a custom ChatGPT on ESSKA osteotomy consensus
Sweed, T. A., Mabrouk, A., & Dawson, M. (2025) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.12653 · Cited by 8 · see citing works
Mahamid, A., Laver, L., Zahalka, S., Oettl, F. C., Behrbalk, E., Hirschmann, M. T., & Samuelsson, K. (2025) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70273 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Still being checked.
- Authors
- El Kayali, M. K., Neumann, E. M., Fahy, S., Bartek, B., Oehme, S., Getgood, A., Jung, T., & Milinkovic, D. D.
- Affiliations
- Charité – Universitätsmedizin Berlin (DE)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Qatar Orthopaedic and Sports Medicine Hospital (QA)
- Venue
- Journal of ISAKOS
- Publisher
- Elsevier BV
- Published
- 2026-08
- Volume
- 19
- Pages
- 101160
- ISSN
- 2059-7754 (ISSN-L)
- OpenAlex
- W7163986365
- Cited by
- 0
- References
- 24
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of ISAKOS Joint Disorders & Orthopaedic Sports Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- OpenEvidence, retrieval-augmented generation, large language models, anterior cruciate ligament reconstruction, American Academy of Orthopaedic Surgeons, hallucinations, citation accuracy, evidence-based decision support, orthopedic surgery, preoperative assessment, inter-rater reliability, clinical implementation of AI
- MeSH terms
- Anterior Cruciate Ligament Injuries, Orthopedic Surgeons, Large Language Models, Humans, Orthopedics, United States, Practice Guidelines as Topic, Evidence-Based Medicine, Anterior Cruciate Ligament Reconstruction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
