The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index; 21 of the 26 sources its claims cite were checked for support.
Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts
Open accessAbstract
BACKGROUND AND OBJECTIVES: Large Language Models (LLMs) are increasingly used by healthcare professionals and patients for medical information synthesis. Unfortunately, their tendency to generate fabricated information (“hallucinations”) poses a significant risk. This study quantifies and compares citation accuracy of 5 prominent LLMs for common neurosurgical spine topics. METHODS: Five LLMs (ChatGPT, Gemini, Claude, Microsoft Copilot, and OpenEvidence) were evaluated. Ten clinician-level questions derived from North American Spine Society guidelines were posed using a prompt instructing the LLM to act as an “ experienced spine surgeon .” Ten corresponding patient-level questions were posed using a “ patient ” persona prompt. Both prompts requested a concise answer and 3 to 5 peer-reviewed references. Each citation was manually verified and scored as 2 (accurate), 1 (real but misrepresented), or 0 (fabricated). RESULTS: Performance varied significantly across models and question types. OpenEvidence, which directly indexes PubMed and requires user verification as a healthcare professional, achieved perfect accuracy. Among general purpose LLMs, Claude demonstrated the highest accuracy (average score 1.67, 78.0% accurate citations). Gemini had the lowest accuracy (average score 0.97) and the highest rate of fabrications (26.2%). All general purpose LLMs performed worse on patient-level queries. ChatGPT's accuracy score dropped from 1.83 to 1.08 for patient questions while its fabrication rate increased from 2.1% to 20.0%. Error profiles differed by model. Copilot frequently misrepresented article metadata (55.1% score-1 citations) and cited non–peer-reviewed websites (31.6%), particularly for patient questions. CONCLUSION: General purpose LLMs exhibit substantial and variable citation errors when queried on spine surgery topics. Accuracy is substantially lower for patient-facing prompts, which often yield fabricated references or nonscientific sources. While specialized, access-restricted platforms such as OpenEvidence provide high accuracy, the performance of widely accessible models highlights the need for careful verification of all LLM-generated medical information to ensure patient safety.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This study quantified citation accuracy and hallucinations of five large language models (LLMs) for spine surgery questions using clinician-level versus patient-level prompts. General-purpose LLMs showed highly variable citation errors, with patient-level prompts substantially worsening accuracy and increasing fabricated references, while OpenEvidence achieved perfect citation accuracy.
Key findings
- Across 481 citations, OpenEvidence had perfect accuracy (mean score 2.00; 120/120, 100% accurate). Among general-purpose LLMs, Claude was highest (mean 1.67; 78.0% accurate), followed by ChatGPT (1.45; 56.1%), Copilot (1.18; 31.6%), and Gemini lowest (0.97; 23.1% accurate) with the highest fabrication rate (26.2%).
- Patient-level prompts significantly reduced accuracy for general-purpose LLMs: ChatGPT fell from 1.83 to 1.08 (P < .001), with fabrications rising from 2.1% to 20.0%; Gemini fell from 1.14 to 0.77 (P = .027), with fabrications rising from 2.0% to 20.0%. Claude showed a non-significant trend (1.78 vs 1.56, P = .258), and Copilot showed no significant difference (P = .494).
- Error profiles differed by model: Copilot had the highest misrepresented-citation rate (55.1%), mostly “Other Metadata Errors” (74.1%), and cited non–peer-reviewed websites in 31.6% of total responses (48.0% for patient vs 14.6% for clinician queries). ChatGPT and Gemini score-1 errors were most often incorrect PMIDs/DOIs (87.5% and 60.6%, respectively), while Claude had the lowest score-1 rate (11.0%).
Methods
- Five LLMs were evaluated: ChatGPT 5.0, Gemini 2.5 Flash, Claude Sonnet 4.5, Microsoft Copilot, and OpenEvidence 2.7.12, all queried in October 2025 via default public web interfaces. Twenty prompts were used: 10 clinician-level questions derived from North American Spine Society guidelines and 10 corresponding patient-level layperson questions; each requested a concise answer and 3–5 peer-reviewed references, with each query run in a separate chat session.
- Citations were manually verified by three independent medical student reviewers using PubMed and publisher websites and scored as 2 (accurate), 1 (real but misrepresented), or 0 (fabricated). Inter-rater reliability was high (95.4% agreement; Cohen’s κ = 0.918), with discordant scores resolved by consensus with the senior author.
- Mean accuracy scores were calculated per model and compared for clinician versus patient prompts using two-tailed Mann-Whitney U tests between models, one-tailed Mann-Whitney U tests between prompt types, and χ2 tests for error-type distributions; significance was P < .05, and no statistical testing was performed for OpenEvidence because it achieved a perfect score.
Limitations
- The paper states that the study was conducted at a single time point, models are constantly updated, the 20 questions are not exhaustive, each query was sampled only once, and temperature or other modeling parameters were not recorded or altered. It also states that it did not evaluate the factual accuracy of the text answers themselves, only the quality of supporting citations.
- The text makes evident that only four general-purpose LLMs were compared against one specialized, access-restricted platform, and all models were queried through default public interfaces without laboratory control of retrieval, so retrieval behavior and availability may vary by platform configuration and over time.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
31 Own result · 10 Partly supported · 10 Own method · 9 Own interpretation · 3 Supported · 3 Not checked yet · 3 No basis given · 1 Supported so far · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In particular, prompt framing and persona assignment (e.g., instructing the model to respond as if to a clinician vs a patient) can meaningfully shift LLM outputs and measured performance, including hallucination rate and factual accuracy.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [2] Maity S Saikia MJ. Large language models in healthcare and medical applications: a review. Bioengineering (Basel). 2025; 12(6): 631. 40564447 PMC12189880 sourceExperimental — accuracy not yet measured
Section 4.3 (page unavailable) says that framing prompts can guide LLM outputs toward greater accuracy and improve task performance. It does not address clinician-versus-patient persona assignment or show that prompting changes hallucination rates; Section 7.2 discusses hallucinations as a general limitation, not as an outcome of prompt framing. (LLM, conf=0.84)
- Partly supportedcites [15] Liu J Liu F Wang C Liu S. Prompt engineering in clinical practice: tutorial for clinicians. J Med Internet Res. 2025; 27: e72644. 40955776 PMC12439060 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says small changes in prompt wording, structure, or tone can markedly change output relevance and accuracy, supporting the prompt-framing and factual-accuracy portion. The ‘Evaluate and Refine the Output’ section recommends checking for hallucinations, but our copy does not establish that assigning a clinician-versus-patient persona changes hallucination rates or other measured performance. (LLM, conf=0.88)
- Partly supportedcites [16] Esmaeilzadeh P. Ethical implications of using general-purpose LLMs in clinical settings: a comparative analysis of prompt engineering strategies and their impact on patient safety. BMC Med Inform Decis Mak. 2025; 25(1): 342. 41024005 PMC12481957 sourceExperimental — accuracy not yet measured
The Results report that prompt strategies changed measured clinical performance, and the Discussion reports improved diagnostic accuracy with meta-cognitive prompting (page unavailable). But the Limitations say hallucinations were not systematically evaluated, while the Discussion calls for future comparisons of patient-facing and clinician-facing contexts. The paper therefore does not establish the claimed effects of persona assignment on hallucination rate or factual accuracy. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For example, Pressman et al9 note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [9] Pressman SM Borna S Gomez-Cabello CA Haider SA Haider CR Forte AJ. Clinical and surgical applications of large language models: a systematic review. J Clin Med. 2024; 13(11): 3041. 38892752 PMC11172607 sourceExperimental — accuracy not yet measured
Section 4.5 reports that ChatGPT gave futile treatment recommendations in a salivary-gland clinic study and underperformed at triaging dermatologic lesions for Mohs surgery. It also warns that patients may encounter inaccurate, potentially harmful information. It does not report an instance of an LLM failing to identify a medical emergency, or establish that the cited treatment recommendations were given directly to patients. Page numbers are unavailable in our copy. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The utility of these models is limited by a significant and well-documented flaw: hallucinations.8”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Supportedcites [8] Shah R Schwab JH. Large language models in spine surgery: a promising technology. HSS J. 2025; 21(3): 15563316251340696. PMC12125007 40454291 source
The Discussion identifies hallucinations—incorrect or misleading outputs—as a challenge to LLM use that requires accuracy checks. The Patient Education section also reports a 4%–7% hallucination rate in LLM-generated imaging reports, supporting the claim that this is a documented limitation. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Within spine surgery, the literature has largely centered on procedural and technical use cases such as diagnostics, image processing, and prognostic prediction, rather than evaluating the reliability of LLM-generated “evidence” itself.6”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [6] Shi L Wang H Shea GKH. The application of artificial intelligence in spine surgery: a scoping review. J Am Acad Orthop Surg Glob Res Rev. 2025; 9(4): e24.00405. PMC11999406 40239218 sourceExperimental — accuracy not yet measured
The Abstract Results report that the 105 included studies most commonly addressed prognostication, diagnosis, and medical image processing, supporting the claim about the literature reviewed. The Methods limit that review to selected articles published from 2020 to March 2024; our copy does not establish that the wider spine-surgery literature has neglected the reliability of LLM-generated evidence. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Fabricated or hallucinated citations can create a false evidence base that is difficult for a busy clinician, much less a member of the lay public, to verify.8”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [8] Shah R Schwab JH. Large language models in spine surgery: a promising technology. HSS J. 2025; 21(3): 15563316251340696. PMC12125007 40454291 sourceExperimental — accuracy not yet measured
The Clinical Practice section (page unavailable) reports that GPT-4 generated incorrect citations in half of its responses to neurosurgical questions, and the Discussion warns that hallucinations can produce misleading information. The paper does not say that these citations create a false evidence base or that busy clinicians or members of the public have difficulty verifying them. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It is well-established that how an LLM is prompted (“prompt engineering”) significantly affects the accuracy and ethical behavior of its output.2,15,16”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [2] Maity S Saikia MJ. Large language models in healthcare and medical applications: a review. Bioengineering (Basel). 2025; 12(6): 631. 40564447 PMC12189880 sourceExperimental — accuracy not yet measured
Section 4.3 says carefully crafted prompts guide LLMs toward accurate, clinically relevant outputs and can constrain responses to align with medical best practices. Section 7.3 discusses ethical concerns, but does not say that prompt engineering significantly affects the ethical behavior of outputs. No page numbers are available in our copy. (LLM, conf=0.84)
- Supportedcites [15] Liu J Liu F Wang C Liu S. Prompt engineering in clinical practice: tutorial for clinicians. J Med Internet Res. 2025; 27: e72644. 40955776 PMC12439060 source
The Introduction says minor changes in prompt wording, structure, or tone can markedly change output accuracy, and that poorly designed prompts can produce biased outputs. The Key Principles section also identifies ethical considerations as a principle of prompt design for accurate and safe outputs. (LLM, conf=0.90)
- Supportedcites [16] Esmaeilzadeh P. Ethical implications of using general-purpose LLMs in clinical settings: a comparative analysis of prompt engineering strategies and their impact on patient safety. BMC Med Inform Decis Mak. 2025; 25(1): 342. 41024005 PMC12481957 source
The Results (Domain-Specific model performance; page unavailable) report that diagnostic accuracy differed by prompting strategy and that ethical reasoning varied significantly by prompt strategy. These findings support the claim in the clinical settings studied. (LLM, conf=0.91)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“These errors are fundamental to the design of LLMs, which are optimized for fluency and coherence rather than factual information.11”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Supportedcites [11] Bélisle-Pipon JC. Why we need to be careful with LLMs in medicine. Front Med (Lausanne). 2024; 11: 1495582. 39697212 PMC11652181 source
The Discussion says LLMs generate fluent, coherent text but that their responses are not necessarily grounded in verified facts. It explains that their next-word prediction mechanism is indifferent to truth and that they are trained to predict patterns, not produce factual outputs. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while Claude (11.0%) and ChatGPT (11.2%) had lowest fabrication rates.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Between-Model Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It had the highest rate of misrepresented (Score 1) citations (55.1%), and these were overwhelmingly “Other Metadata Errors” (e.g., incorrect authors, date, or title), which accounted for 74.1%.”
- Also stated in: § Error Composition · plain
“Copilot | 54 | 3 (5.6) | 11 (20.4) | 40 (74.1) | 31 (57.4)”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Error Composition
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Error Composition · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“no statistical performance difference was found between Copilot and Gemini (U = 2668.00, P = .080).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Between-Model Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 15
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This risk is amplified by the tendency for LLMs to present these fabrications with an authoritative tone, mixing them with factual information, in a way that makes errors difficult to detect, presenting a potential threat to patient safety.7,11”
- Also stated in: § Clinical Implications · boosted
“This is particularly dangerous because LLMs deliver these fabrications with an authoritative tone, mixing them seamlessly with otherwise correct information, making errors extremely difficult for a layperson to detect.11”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Supportedcites [7] Kouzy R Bitterman DS. Improving patient engagement: is there a role for large language models?. Int J Radiat Oncol Biol Phys. 2024; 120(3): 639- 641. 39326949 PMC12235655 source
In “Accessible LLMs as a Parallel to Patient Engagement” (page not provided), the paper says LLMs can mix correct and incorrect information while answering authoritatively, making errors difficult to detect. “Scalable Engagement” also warns that hallucinations can have serious consequences in healthcare. (LLM, conf=0.95)
- Partly supportedcites [11] Bélisle-Pipon JC. Why we need to be careful with LLMs in medicine. Front Med (Lausanne). 2024; 11: 1495582. 39697212 PMC11652181 sourceExperimental — accuracy not yet measured
The Introduction describes plausible-sounding inaccuracies, and the Discussion says fluent, coherent responses can conceal that they are not grounded in verified facts, posing risks in medical care (page numbers unavailable). This supports the claim that convincing presentation can make errors hard to recognize, but our copy does not specifically say that LLMs mix fabrications with factual information. (LLM, conf=0.84)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Clinical Implications · boosted
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“LLMs are being explored for their utility in spine surgery research and patient education, offering to simplify complex medical concepts for patients trying to understand their own conditions, improving healthcare literacy.7,8”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [7] Kouzy R Bitterman DS. Improving patient engagement: is there a role for large language models?. Int J Radiat Oncol Biol Phys. 2024; 120(3): 639- 641. 39326949 PMC12235655 sourceExperimental — accuracy not yet measured
The “Accessible LLMs as a Parallel to Patient Engagement” section discusses patients using LLMs to ask about their care, summarize information, and formulate questions for clinicians. The paper focuses on radiation oncology and patient engagement, however; it does not discuss spine surgery research or establish that LLMs improve healthcare literacy by simplifying complex concepts. (LLM, conf=0.86)
- Supportedcites [8] Shah R Schwab JH. Large language models in spine surgery: a promising technology. HSS J. 2025; 21(3): 15563316251340696. PMC12125007 40454291 source
The Discussion says LLMs show potential to enhance patient education and streamline spine-surgery research, including by explaining conditions and treatment options to patients. The Patient Education section also describes their use to simplify medical jargon into understandable language. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large Language Models (LLMs) are increasingly used by healthcare professionals and patients for medical information synthesis.”
- Also stated in: § CONCLUSION: · plain
“The rapid adoption of generative artificial intelligence (AI) and large language models (LLMs) has transformed much of daily life, with healthcare emerging as a primary field for its application.1,2”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > BACKGROUND AND OBJECTIVES:
What the sources say
- Supportedcites [1] Meng X Yan X Zhang K The application of large language models in medicine: a scoping review. iScience. 2024; 27(5): 109713. 38746668 PMC11091685 source
The Introduction describes LLMs as rapidly integrating into healthcare. In Results (“The integration of LLMs in medical writing”), the review describes researchers using them to synthesize medical findings; in Results (“Elevating medical communication”), it describes LLMs providing patients with understandable information about conditions and treatments. (LLM, conf=0.88)
- Partly supportedcites [2] Maity S Saikia MJ. Large language models in healthcare and medical applications: a review. Bioengineering (Basel). 2025; 12(6): 631. 40564447 PMC12189880 sourceExperimental — accuracy not yet measured
Section 5.4 says LLMs analyze and summarize medical literature for healthcare professionals. Section 5.3 describes patient-facing assistants that provide basic health information and explain medical terminology, but it does not establish that patients increasingly use LLMs for medical information synthesis specifically. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § CONCLUSION: · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“This phenomenon has been documented in the fields of oncology, medical genetics, and orthopedic surgery but remains under-investigated in the field of neurosurgery.17-19”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Supportedcites [17] Chen YC Lee SH Sheu H Enhancing responses from large language models with role-playing prompts: a comparative study on answering frequently asked questions about total knee arthroplasty. BMC Med Inform Decis Mak. 2025; 25(1): 196. 40410756 PMC12102839 source
The Discussion (page unavailable) reports that role-playing prompts improved LLM responses to total knee arthroplasty questions. Within the grouped citations, this paper supports the orthopedic-surgery example; it is not the cited evidence for the other specialties or the neurosurgery research gap. (LLM, conf=0.91)
- Not checked yetcites [18] Yoon SM Lyu J Djunadi TA Navigating artificial intelligence (AI) accuracy: a meta-analysis of hallucination incidence in large language model (LLM) responses to oncology questions. J Clin Oncol. 2025; 43(16_suppl): e13686.
- Not checked — no full text of the sourcecites [19] Flaharty KA Hu P Hanchard SL Evaluating large language models on medical, lay-language, and self-reported descriptions of genetic conditions. Am J Hum Genet. 2024; 111(9): 1819- 1833. 39146935 PMC11393706 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Beaulieu-Jones et al10 describe instances, where models provide accurate information but applied in the wrong context, posing a particular risk for clinicians who may be skimming the response.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Partly supportedcites [10] Beaulieu-Jones BR Berrigan MT Shah S Marwaha JS Lai SL Brat GA. Evaluating capabilities of large language models: performance of GPT-4 on surgical knowledge assessments. Surgery. 2024; 175(4): 936- 942. 38246839 PMC10947829 sourceExperimental — accuracy not yet measured
Table 1 (labeled Conclusion; page unavailable) describes answers based on accurate clinical information applied inaccurately to the question’s context, including a treatment recommendation that overlooks medically refractory symptoms. The Discussion warns that such errors could affect clinical care, but our copy does not describe a particular risk for clinicians who skim responses. (LLM, conf=0.86)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A patient seeking to be well-informed about their spinal condition could be misled by fabricated citations or directed to nonscientific sources.7”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Clinical Implications
What the sources say
- Partly supportedcites [7] Kouzy R Bitterman DS. Improving patient engagement: is there a role for large language models?. Int J Radiat Oncol Biol Phys. 2024; 120(3): 639- 641. 39326949 PMC12235655 sourceExperimental — accuracy not yet measured
The “Accessible LLMs as a Parallel to Patient Engagement” section says LLMs can give patients authoritative-sounding but incorrect answers and relay internet misinformation. The “Scalable Engagement” section also warns about hallucinations. Neither section specifically discusses fabricated citations, directing patients to nonscientific sources, or spinal conditions. (LLM, conf=0.85)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“High rates of citation fabrication have been quantified in other fields, such as a study on systematic reviews in the field of orthopedic surgery conducted by Chelli et al,12 which found high hallucination rates in ChatGPT.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
What the sources say
- Supportedcites [12] Chelli M Descamps J Lavoué V, et al. Hallucination rates and reference accuracy of ChatGPT and bard for systematic reviews: comparative analysis. J Med Internet Res. 2024; 26: e53164. 38776130 PMC11153973 source
The Results report hallucination rates of 39.6% for GPT-3.5 and 28.6% for GPT-4. The Discussion identifies the study as testing references for systematic reviews of rotator cuff disease, an orthopedic condition, and describes the resulting rate of misleading or hallucinated references as exceeding 25%. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, exceeding 5 citations after reprompting was uncommon among general purpose models. ChatGPT did so only once, returning 6 citations, and no other general purpose model made this error.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Model-Specific Behaviors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“By contrast, OpenEvidence most frequently returned citation lists longer than 5, consistent with its retrieval-augmented PubMed-linked workflow.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Model-Specific Behaviors
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude emerged as the superior model overall, outperforming ChatGPT (U = 3943.50, P = .018), Gemini (U = 1553.00, P < .001), and Copilot (U = 2853.50, P < .001).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Between-Model Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude showed a trend toward lower scores for patient prompts (1.78 vs 1.56), although this did not reach statistical significance (U = 1156.00, P = .258).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Clinician vs Patient Prompts
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT followed with an average citation score of 1.45 (CI 1.31-1.59) (56.1% accurate citations).”
- Also stated in: § Between-Model Performance · plain
“ChatGPT | 98 | 1.45 [1.31-1.59] | 55 (56.1) | 32 (32.7) | 11 (11.2)”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Between-Model Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Between-Model Performance · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A total of 481 citations were collected and analyzed, 361 of which were from the 4 general purpose LLMs. The remaining 120 responses, provided by the specialized OpenEvidence model, achieved perfect accuracy (Score 2).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Gemini had the lowest accuracy (average score 0.97) and the highest rate of fabrications (26.2%).”
- Also stated in: § Between-Model Performance · boosted
“Gemini demonstrated the highest rate of fabricated (Score 0) citations at 26.2% of the reported citations”
- Also stated in: § Between-Model Performance · plain
“Gemini | 65 | 0.97 [0.79-1.14] | 15 (23.1) | 33 (50.8) | 17 (26.2)”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > RESULTS:
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Between-Model Performance · boosted
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Because the main source of responses with greater than 5 citations achieved perfect accuracy, the citation-count protocol is unlikely to have materially inflated error rates or altered comparative conclusions in this data set.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Model-Specific Behaviors
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“persona-driven prompts can substantially reduce the risk of receiving fabricated information.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Clinical Implications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“User-adjustable hyperparameters (e.g., temperature) were not uniformly available across platforms and were not recorded.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > METHODS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This study was conducted at a single point in time, and these models are constantly being updated.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Limitations and Future Directions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The models' performance in October 2025 is relevant to current and future use.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To prevent cross-contamination or memory carry-over from previous queries, each question was run in a new, separate chat session.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > METHODS > Prompt Personas and Queries
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The phenomenon of hallucinations, or the generation of false, misleading, or fabricated information presented as fact, poses a critical danger in a medical context.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > CONCLUSION:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Three independent medical student reviewers performed the citations verification.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > METHODS > Grading of Generated Citations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This study validates the practice of “prompt engineering,” demonstrating that clinician-level prompts elicit more accurate responses”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > CONCLUSIONS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, this study did not evaluate the factual accuracy of the text answers themselves, only the quality of supporting citations.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Limitations and Future Directions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each generated citation was manually verified by the authors using PubMed and publisher websites to confirm its existence and the accuracy of its metadata (authors, title, year, journal, PMID/DOI).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > METHODS > Grading of Generated Citations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Manual verification using PubMed and publisher websites can reliably determine whether a citation is real and whether its metadata are correct.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any discordant scores were reviewed jointly and resolved by consensus with the senior author.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > METHODS > Grading of Generated Citations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“OpenEvidence, which directly indexes PubMed and requires user verification as a healthcare professional, achieved perfect accuracy.”
- Also stated in: § CONCLUSION: · plain
“one specialized medical platform, which requires user verification as a healthcare professional.20”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > RESULTS:
What the sources say
- Not checked yetcites [20] About | OpenEvidence. Accessed November 3, 2025. https://www.openevidence.com/about
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § CONCLUSION: · plain
In the paper
“OpenEvidence, which directly indexes PubMed and requires user verification as a healthcare professional, achieved perfect accuracy.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > Abstract > RESULTS:
What the sources say
- Not checked yetcites [20] About | OpenEvidence. Accessed November 3, 2025. https://www.openevidence.com/about
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, each query was only sampled one time, and modeling parameters such as temperature, a parameter that controls the “randomness” or “creativity” of an AI model's output, were not explicitly recorded or altered.”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > DISCUSSION > Limitations and Future Directions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: A single query per question is sufficient to characterize each model's citation accuracy.
The paper relies on this without support.
The default, publicly available web interfaces and internet search capabilities of the LLMs reflect how typical clinicians and patients use them.
“Accordingly, we report model performance as encountered by typical clinician and patient users, recognizing that retrieval behavior and availability may vary by platform configuration and over time.”
The paper relies on this without support.
A single query per question is sufficient to characterize each model's citation accuracy.
“In addition, each query was only sampled one time, and modeling parameters such as temperature, a parameter that controls the “randomness” or “creativity” of an AI model's output, were not explicitly recorded or altered.”
The paper relies on this without support.
The 20-question bank adequately represents common spine surgery topics and typical patient inquiries.
“Second, the list of 20 questions is not exhaustive but nonetheless represents some of the most commonly performed spine procedures.”
The paper relies on this without support.
The clinician and patient prompts differ primarily in persona and not in other factors that could affect citation accuracy.
“Two standardized, persona-driven prompts were used for respective question categories.”
The paper relies on this without support.
Manual verification using PubMed and publisher websites can reliably determine whether a citation is real and whether its metadata are correct.
“Each generated citation was manually verified by the authors using PubMed and publisher websites to confirm its existence and the accuracy of its metadata (authors, title, year, journal, PMID/DOI).”
The paper relies on this without support.
The models' performance in October 2025 is relevant to current and future use.
“This study was conducted at a single point in time, and these models are constantly being updated.”
The paper relies on this without support.
The North American Spine Society guidelines are an appropriate source for clinician-level questions.
“Questions were sourced directly from clinical guidelines published by the North American Spine Society (NASS), encompassing common procedures, including anterior cervical discectomy and fusion, lumbar laminectomy, transforaminal lumbar interbody fusion, and lumbar microdiscectomy”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Clinician vs Patient Prompts
“A critical finding was significant drop in performance of all general purpose LLMs when responding to patient-level prompts compared with clinician-level prompts (Figure 2).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Clinician vs Patient Prompts
“Claude showed a trend toward lower scores for patient prompts (1.78 vs 1.56), although this did not reach statistical significance (U = 1156.00, P = .258). Copilot showed no significant difference in mean accuracy scores between patient-level and clinician-level prompts (U = 1198.00, P = .494).”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Error Composition
“Gemini (20.0% vs 2.0%).”
§ Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts > RESULTS > Between-Model Performance
“Gemini demonstrated the highest rate of fabricated (Score 0) citations at 26.2% of the reported citations”
Citations
Cites
Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C. (2024) · Journal of Medical Internet Research · doi:10.2196/53164 · Cited by 602 · cited 1×
“8 High rates of citation fabrication have been quanti fi ed in other fi elds, such as a study on systematic reviews in the fi eld of orthopedic surgery conducted by Chelli et al, 12 which found high hallucination rates in ChatGPT.”
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414
Large Language Models in Healthcare and Medical Applications: A Review
Maity, S., & Saikia, M. J. (2025) · Bioengineering · doi:10.3390/bioengineering12060631 · Cited by 249 · cited 1×
“2,15,16 In particular, prompt framing and persona assignment (e.g., instructing the model to respond as if to a clinician vs a patient) can meaningfully shift LLM outputs and measured performance, including hallucination rate and factual accuracy.”
Chen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112
Clinical and Surgical Applications of Large Language Models: A Systematic Review
Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13113041 · Cited by 107 · cited 1×
“For example, Pressman et al 9 note instances where LLMs provided incorrect or unsafe medical advice to patients, such as suggesting inappropriate treatments or failing to identify emergencies.”
Artificial intelligence in healthcare (Review)
Alhejaily, A. (2024) · Biomedical Reports · doi:10.3892/br.2024.1889 · Cited by 95
Why we need to be careful with LLMs in medicine
Bélisle-Pipon, J.-C. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1495582 · Cited by 91 · cited 3×
“7,11 The focus on citation generation is therefore essential.”
Beaulieu-Jones, B. R., Berrigan, M. T., Shah, S., Marwaha, J. S., Lai, S.-L., & Brat, G. A. (2024) · Surgery · doi:10.1016/j.surg.2023.12.014 · Cited by 83 · cited 2×
“Beaulieu-Jones et al 10 describe instances, where models provide accurate information but applied in the wrong context, posing a particular risk for clinicians who may be skimming the response.”
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
Flaharty, K. A., Hu, P., Hanchard, S. L., Ripper, M. E., Duong, D., Waikel, R. L., & Solomon, B. D. (2024) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2024.07.011 · Cited by 26 · cited 1×
“17-19 Accordingly, it remains unknown whether clinician-level vs patient-level persona prompting differentially affects the tendency for models to hallucinate references when queried for neurosurgical topics.”
Esmaeilzadeh, P. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03182-6 · Cited by 26 · cited 1×
“2,15,16 In particular, prompt framing and persona assignment (e.g., instructing the model to respond as if to a clinician vs a patient) can meaningfully shift LLM outputs and measured performance, including hallucination rate and factual accuracy.”
Chen, Y.-C., Lee, S.-H., Sheu, H., Lin, S.-H., Hu, C.-C., Fu, S.-C., Yang, C.-P., & Lin, Y.-C. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03024-5 · Cited by 23
Revolutionizing spine surgery with emerging AI–FEA integration
Franceschini, C., Ahmadi, M., Zhang, X., Wu, K. L., Lin, M., Weston, R., Rodio, A., Tang, Y., Engeberg, E. D., Pires, G., et al. (2025) · Journal of Robotic Surgery · doi:10.1007/s11701-025-02772-w · Cited by 21
Artificial Intelligence in Planning for Spine Surgery
Ali, I. S., Bakaes, Y., MacLeod, J., Lee, T. Y., Cho, S., & Hsu, W. K. (2025) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-025-09992-5 · Cited by 18
The Application of Artificial Intelligence in Spine Surgery: A Scoping Review
Shi, L., Wang, H., & Shea, G. K.-H. (2025) · JAAOS: Global Research and Reviews · doi:10.5435/jaaosglobal-d-24-00405 · Cited by 9
AtlasGPT: a language model grounded in neurosurgery with domain-specific data and document retrieval
Ali, R., Abdulrazeq, H., Patil, A., Cheatham, M., Connolly, I. D., Tang, O. Y., Doberstein, C., Riccelli, T. E., Huang, K. T., Shankar, G. M., et al. (2025) · Journal of neurosurgery · doi:10.3171/2024.12.jns241607 · Cited by 8 · cited 1×
“10,13,14 Within spine surgery, the literature has largely centered on procedural and technical use cases such as diagnostics, image processing, and prognostic prediction, rather than evaluating the reliability of LLM-generated “ evidence ” itself.”
Yoon, S. M., Lyu, J., Djunadi, T. A., Song, J., Kim, H. S., Min, R. S., Sakellakis, M., & Chae, Y. K. (2025) · Journal of Clinical Oncology · doi:10.1200/jco.2025.43.16_suppl.e13686 · Cited by 8
Large Language Models in Spine Surgery: A Promising Technology
Shah, R., & Schwab, J. H. (2025) · HSS Journal®: The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1177/15563316251340696 · Cited by 5 · cited 1×
“8 High rates of citation fabrication have been quanti fi ed in other fi elds, such as a study on systematic reviews in the fi eld of orthopedic surgery conducted by Chelli et al, 12 which found high hallucination rates in ChatGPT.”
Improving Patient Engagement: Is There a Role for Large Language Models?
Kouzy, R., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.06.001 · Cited by 3 · cited 3×
“7,11 The focus on citation generation is therefore essential.”
Comparative diagnostic capability of large language models in neurosurgery
Warrier, A. K., Levin-Carrion, Y., Shah, S. B., Som, T., Bosland, A., Wardell, J., Ha, C. J., Singh, R., Pando, A., & Liu, J. K. (2026) · Journal of neurosurgery · doi:10.3171/2025.9.jns25846 · Cited by 1
Cited by
Sorted by citations
The Potential of Artificial Intelligence in Dental Trauma
Akitomo, T., Hamada, M., Kusaka, S., Zhao, J., & Nomura, R. (2026) · Oral Diseases · doi:10.1111/odi.70529
Metadata
- Authors
- McLaughlin, N. D., Srinivas, A. N., Lowe, Z. F., Botterbush, K. S., Patel, M. S., & Avila, M. J.
- Affiliations
- Saint Louis University (US)
- Corresponding author
- Avila, Mauricio J.
- Venue
- Neurosurgery Practice
- Published
- 2026-06
- Volume
- 7
- Issue
- 3
- Pages
- e000244-e000244
- ISSN
- 2834-4383 (ISSN-L)
- PMC
- PMC13225591
- OpenAlex
- W7162806514
- Cited by
- 1
- References
- 18
- Citations by year
- 2026: 1
- FWCI
- 1.99
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Neurosurgery Open (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, hallucinations, spinal surgery, citation accuracy, reference verification, OpenEvidence, ChatGPT, Claude, Gemini, Copilot, health misinformation, patient safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
