Development of a novel artificial intelligence clinical decision support tool for hand surgery: HandRAG
Open accessAbstract
Purpose: Hand surgery decision-making requires integration of complex anatomical understanding, diverse patient-specific factors, and nuanced operative techniques. While artificial intelligence (AI), large language models (LLMs), and retrieval-augmented generation (RAG) models have advanced significantly in various fields, no AI-driven clinical decision support systems currently exist for hand surgery. A novel retrieval-enhanced AI large language model specifically tailored for hand surgery was developed, capable of effectively utilizing peer-reviewed published hand surgery literature for clinical decision support in real-time at point of care. Methods: An AI clinical decision support system was developed integrating all available open-access 4510 peer-reviewed hand surgery publications from 2000 to 2024 identified through hand surgery-relevant keywords. Documents were processed using a hierarchical pipeline based on the RAPTOR methodology, which breaks down large texts into smaller segments to enhance accurate retrieval. The system was evaluated using 15 standardized clinical queries assessed using automated computational metrics for correctness and semantic similarity to source documents. Results: The AI system demonstrated consistent performance with an average G-Eval correctness score of 0.79, SEM with an average similarity score of 0.75 (range: 0.54-0.86) and average maximum similarity score of 0.80 (range: 0.56-0.91), predominantly at moderate confidence levels. Generated recommendations were contextually appropriate and reliably linked to relevant hand surgery literature, providing accurate and clinically meaningful guidance. Conclusion: The AI system, HandRAG, incorporating RAG and LLM approach offers potential benefits for evidence-based clinical decision support and education in hand surgery.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The orthopaedic RAG chatbot [7] obtained high accuracy (4.55/5), clarity (4.77/5), trust (4.23/5) scores, and significantly outperformed traditional template patient-information leaflets and rule-based FAQ systems.
“The orthopaedic RAG chatbot [7] obtained high accuracy (4.55/5), clarity (4.77/5), trust (4.23/5) scores, and significantly outperformed traditional template patient-information leaflets and rulebased FAQ systems.”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [7]
The Results and Discussion describe HandRAG as a hand-surgery RAG tool and report its performance, but the reported averages are G-Eval correctness 0.79 and SEM score 0.75—not the three scores claimed. The Methods (System evaluation) say this proof-of-concept used computational metrics rather than human evaluation; our copy does not report a comparison with patient-information leaflets or rule-based FAQ systems. (LLM, conf=0.88)
Experimental — accuracy not yet measuredTrust in [7] was enhanced due to generation of correct responses and guideline-linked responses, with clinicians rating clarity and trust greater than 4.2/5.
“Trust enhanced in [7] due to the generation of correct responses, and a guideline-linked response, with clinicians rating clarity and trust greater than 4.2/5.”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [7]
The Results report generally strong computational correctness scores, and the Discussion says HandRAG grounds responses in citable literature. But the Methods (System evaluation; page unavailable) explicitly say this proof-of-concept used computational metrics rather than evaluation by practicing surgeons. The paper therefore does not support the claimed clinician ratings above 4.2/5 or a measured increase in trust. (LLM, conf=0.88)
Experimental — accuracy not yet measuredExplanations in multi-agent RAG systems are directly connected to retrieved biomedical literature so clinicians can study sources behind diagnostic suggestions.
“These explanations were directly connected to retrieved biomedical literature so clinicians could study sources of information behind diagnostic suggestions.[7]”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [7]
The Methods (Integration of LLM to RAG system; page unavailable) say HandRAG generates answers with citations linking to retrieved source documents, and the Discussion describes its responses as grounded in citable hand-surgery literature. The paper does not describe HandRAG as a multi-agent system or establish that clinicians used its citations to examine the sources behind diagnostic suggestions; its examples chiefly concern clinical questions and treatment recommendations. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826 · see citing works
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101 · see citing works
Dimensionality reduction for visualizing single-cell data using UMAP
Becht, E., McInnes, L., Healy, J., Dutertre, C.-A., Kwok, I. W. H., Ng, L. G., Ginhoux, F., & Newell, E. W. (2019) · Nature Biotechnology · doi:10.1038/nbt.4314 · Cited by 5,358 · see citing works
Reynolds, D. A. (2009) · Encyclopedia of Biometrics · doi:10.1007/978-0-387-73003-5_196 · Cited by 2,062 · see citing works
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948 · see citing works
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · see citing works
Large language models and the perils of their hallucinations
Azamfirei, R., Kudchadkar, S. R., & Fackler, J. C. (2023) · Critical Care · doi:10.1186/s13054-023-04393-x · Cited by 352 · see citing works
Chingovska, I., Anjos, A., & Marcel, S. (2009) · Springer eBooks · doi:10.1007/978-3-642-27733-7 · Cited by 246 · see citing works
Ge, J., Sun, S., Owens, J., Galvez, V., Gologorskaya, O., Lai, J. C., Pletcher, M. J., & Lai, K. (2024) · Hepatology · doi:10.1097/hep.0000000000000834 · Cited by 189 · see citing works
Viroli, C., & McLachlan, G. J. (2017) · Statistics and Computing · doi:10.1007/s11222-017-9793-z · Cited by 167 · see citing works
Physician- and Large Language Model–Generated Hospital Discharge Summaries
Williams, C. Y. K., Subramanian, C. R., Ali, S. S., Apolinario, M., Askin, E., Barish, P. N., Cheng, M., Deardorff, W. J., Donthi, N., Ganeshan, S., et al. (2025) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2025.0821 · Cited by 127 · see citing works
Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening
Ünlü, O., Shin, J., Mailly, C. J., Oates, M., Tucci, M. R., Varugheese, M., Wagholikar, K. B., Wang, F., Scirica, B. M., Blood, A. J., et al. (2024) · NEJM AI · doi:10.1056/aioa2400181 · Cited by 119 · see citing works
Unlu, O., Shin, J., Mailly, C. J., Oates, M. F., Tucci, M. R., Varugheese, M., Wagholikar, K., Wang, F., Scirica, B. M., Blood, A. J., et al. (2024) · doi:10.1101/2024.02.08.24302376 · Cited by 117 · see citing works
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., & Manning, C. D. (2024) · arXiv · doi:10.48550/arxiv.2401.18059 · Cited by 36 · see citing works
Oeding, J. F., Williams, R. J., Camp, C. L., Sánchez‐Sotelo, J., Kelly, B. T., Nawabi, D. H., Karlsson, J., Pearle, A. D., Martin, R. K., Jang, S. J., et al. (2023) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-023-07338-7 · Cited by 33 · see citing works
Innovative Approaches to Safe Surgery: A Narrative Synthesis of Best Practices
Hussain, A. K., Kakakhel, M. K., Ashraf, M. F., Shahab, M., Ahmad, F., Luqman, F., Ahmad, M., Nour, A. M., Varrassi, G., & Kinger, S. (2023) · Cureus · doi:10.7759/cureus.49723 · Cited by 33 · see citing works
Özmen, B. B., & Mathur, P. (2025) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2025.03.053 · Cited by 32 · see citing works
SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual SimilarityPreprint
Aynetdinov, A., & Akbik, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.17072 · Cited by 31 · see citing works
Özmen, B. B., & Schwarz, G. S. (2024) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2024.04.054 · Cited by 27 · see citing works
Self-citationGenovese, A., Prabha, S., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Trabilsy, M., Tao, C., Aziz, K. T., Murray, P. M., & Forte, A. J. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf038 · Cited by 24 · see citing works
ChatGPT can yield valuable responses in the context of orthopaedic trauma surgery
Kaarre, J., FELDT, R. H., Zsidai, B., Senorski, E. H., Rydberg, E. M., Wolf, O., Mukka, S., Möller, M., & Samuelsson, K. (2024) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.12047 · Cited by 20 · see citing works
The artificial intelligence advantage: Supercharging exploratory data analysis
Oettl, F. C., Oeding, J. F., FELDT, R. H., Ley, C., Hirschmann, M. T., Samuelsson, K., & ESSKA Artificial Intelligence Working Group (2024) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.12389 · Cited by 20 · see citing works
Winkler, P. W., Zsidai, B., Senorski, E. H., Pruneski, J. A., Hirschmann, M. T., Ley, C., Tischer, T., Herbst, E., Pareek, A., Musahl, V., et al. (2025) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70247 · Cited by 20 · see citing works
Özmen, B. B. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf051 · Cited by 18 · see citing works
Kunze, K. N., Williams, R. J., Ranawat, A. S., Pearle, A. D., Kelly, B. T., KARLSSON, J. L., Martin, R. K., & Pareek, A. (2024) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.12018 · Cited by 15 · see citing works
Artamonov, A., Bachar-Avnieli, I., Klang, E., Lubovsky, O., Atoun, E., Bermant, A., & Rosinsky, P. J. (2024) · Arthroscopy Sports Medicine and Rehabilitation · doi:10.1016/j.asmr.2024.100923 · Cited by 14 · see citing works
Özmen, B. B., Berber, İ., & Schwarz, G. S. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf049 · Cited by 12 · see citing works
Self-citationBusch, F., Kaibel, L., Nguyen, H., Lemke, T., Ziegelmayer, S., Graf, M., Marka, A. W., Endrös, L., Prucker, P., Spitzl, D., et al. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01483-w · Cited by 10 · see citing works
Transforming orthopaedics with AI: Insights from a custom ChatGPT on ESSKA osteotomy consensus
Sweed, T. A., Mabrouk, A., & Dawson, M. (2025) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.12653 · Cited by 8 · see citing works
UMAP for Dimensionality Reduction in Sleep Stage Classification Using EEG Data
Deng, Y., Albidah, H., Cheng, H., Dallal, A. H., Yin, J., & Mao, Z. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782097 · Cited by 8 · see citing works
Özmen, B. B., & Schwarz, G. S. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf098 · Cited by 3 · see citing works
Self-citationMandatory Surgeon Skills for Care of the Mutilated Hand
Woo, D., Granruth, C. B., Gundlach, B., & Levin, L. S. (2024) · Journal of Hand Surgery Global Online · doi:10.1016/j.jhsg.2024.07.007 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Explainable artificial intelligence: enhancing decision-making in plastic surgery
Özmen, B. B., Almeida, V. F. A., Mathur, P., & Schwarz, G. S. (2025) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2025.07.012 · Cited by 6 · see citing works
Self-citationMugambiwa, T., & Ndlovu, B. (2026) · Journal of Applied Informatics and Computing · doi:10.30871/jaic.v10i1.11900 · Cited by 4 · see citing works
Gorgos, P., Ternell, K. H., Hammarstrand, C., Wallmon, A., Brogren, E., Björkman, A., & Horváth, A. (2025) · Hand surgery & rehabilitation · doi:10.1016/j.hansur.2025.102530 · Cited by 3 · see citing works
Bonifaz, J. B., Landeta-López, P., Guevara‐Vega, C., & Quiña‐Mera, A. (2026) · Applied Sciences · doi:10.3390/app16073147 · Cited by 2 · see citing works
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Li, C., Akhtar, Z., Kwak, M., Ji, Y., Zhang, H., Obi, T., Ren, Y., Wu, X., Sivarajkumar, S., Lehmann, H. P., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25933 · Cited by 1 · see citing works
Evidence-based AI clinical decision support system for acute burn care and complex reconstruction
Özmen, B. B., Singh, N., Berber, I., Shah, K., Pinsky, E., Duraes, E. F. R., & Schwarz, G. S. (2026) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2026.03.012 · Cited by 1 · see citing works
Self-citationResponse to: Beyond PlasticSurgeryGPT: The Imperative for Reasoning-Enhanced LLMs in Plastic Surgery
Özmen, B. B., Berber, İ., & Schwarz, G. S. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf164
Self-citationÖzmen, B. B., Almeida, V. F. A., Berber, I., Ha, J. Y., Duraes, E. F. R., Isakov, R., Bernard, S. L., Djohan, R. S., & Schwarz, G. S. (2026) · Plastic Surgery · doi:10.1177/22925503261446323
Self-citationKara, S., & Karcıoğlu, A. A. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12279-6
España-Lopera, J. C., Núñez‐Valdéz, E. R., & García-Díaz, V. (2026) · International Conference on Federated Learning and Intelligent Computing Systems (FLICS) · doi:10.1109/flics70075.2026.11621915
Patel, A., Khand, Y., Vallamchetla, S. K., Li, P., Tao, C., & Cheung, J. (2026) · Journal of Medical Internet Research · doi:10.2196/104092
Risks and Ethical Concerns of Artificial Intelligence in Plastic Surgery
Özmen, B. B., & Schwarz, G. S. (2026) · Clinics in Plastic Surgery · doi:10.1016/j.cps.2026.09.004
Self-citation
- Authors
- Ozmen, B. B., Singh, N., Shah, K., Berber, I., Singh, D., Pinsky, E., Rampazzo, A., & Schwarz, G. S.
- Affiliations
- Boston University (US)
- Cleveland Clinic (US)
- Venue
- Journal of Hand and Microsurgery
- Publisher
- Elsevier BV
- Published
- 2025-07
- Volume
- 17
- Issue
- 4
- Pages
- 100293
- ISSN
- 0974-3227 (ISSN-L)
- PMC
- PMC12210289
- OpenAlex
- W4411196645
- Cited by
- 11
- References
- 33
- Citations by year
- 2026: 8 · 2025: 3
- FWCI
- 2.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Hand and Microsurgery (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- hand surgery, clinical decision support, retrieval-augmented generation, large language models, evidence-based medicine, semantic similarity, hand surgery education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
