Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Gemini 2.5 Pro has achieved ≥85-90% accuracy on national pediatric surgery and obstetrics-gynecology exams, well above local passing thresholds of approximately 60% for each assessment.

“In specialty board examinations, Gemini 2.5 Pro has achieved ≥ 85-90% accuracy on national pediatric surgery and obstetrics-gynecology exams, well above local passing thresholds, which were approximately 60% for each assessment (Wielochowska et al. 2025).”

• cites Wielochowska et al. 2025 sourcePartly supported

  The Results report 96.63% accuracy on the obstetrics–gynecology specialization exam, above the 60% success threshold specified in the Introduction. The paper does not report a pediatric surgery exam result or its passing threshold, so it supports only the obstetrics–gynecology portion of the claim. (LLM, conf=0.90)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In medicine, large language models have demonstrated strong performance on complex reasoning tasks and standardized examinations, suggesting a potential role in clinical care and medical education.

“In medicine, these models have demonstrated strong performance on complex reasoning tasks and standardized examinations, suggesting a potential role in clinical care and medical education (Singhal et al. 2023).”

• cites Singhal et al. 2023 sourcePartly supported

  The Model development and evaluation section reports state-of-the-art performance on USMLE-style questions, and the Methods describe medical questions involving complex multi-step reasoning. The Main and Discussion sections describe potential clinical applications while stressing that substantial limitations remain. The paper does not explicitly connect these results to a potential role in medical education. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

Gemini 2.5 Pro fared relatively better on Taxonomy 3 (clinical reasoning) questions, consistent with reports that newer Gemini models often excel on complex and integrative problem solving tasks.

“Gemini 2.5 Pro fared relatively better on Taxonomy 3 (clinical reasoning) questions, consistent with reports than newer Gemini models often excel on complex and integrative problem solving tasks (Wielochowska et al. 2025;Sheikhalishahi et al. 2025;Boczkowski et al. 2025).”

• cites Wielochowska et al. 2025 sourcePartly supported

  The Results report high accuracy on clinical cases (91.67%), which supports the narrower point that Gemini 2.5 Pro performed well on clinical tasks. But the paper does not use Taxonomy 3 or establish that newer Gemini models often excel on complex, integrative tasks; its clinical-case accuracy was numerically lower than theoretical-question accuracy (97.19%), with no significant difference (p=0.313). (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Sheikhalishahi et al. 2025 sourcePartly supported

  The Discussion (page unavailable) reports that Gemini 2.5 Flash outperformed the compared ChatGPT models on Persian internal-medicine board questions and describes an advantage in text-based clinical reasoning. It does not establish that newer Gemini models often excel across complex and integrative problem-solving tasks. The cited study tested Flash, not Gemini 2.5 Pro, and did not analyze Taxonomy 3 questions. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Boczkowski et al. 2025 sourcePartly supported

  The Discussion (page unavailable) says Gemini 2.5 Pro’s high pediatric-surgery exam score may indicate a predisposition for processing complex medical data. It does not establish that newer Gemini models often excel at complex, integrative problem-solving tasks. Its clinical-reasoning comparison found no significant advantage over theoretical questions; the cited paper does not assess the citing authors’ Taxonomy 3 category. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Early research demonstrated that GPT-3.5 could attain passing-level performance on licensing examinations such as the USMLE without specialty-specific training.

“Early research demonstrated that GPT-3.5 could attain passinglevel performance on licensing examinations such as the United States Medical Licensing Examination (USMLE) without specialty-specific training (Kung et al. 2023).”

• cites Kung et al. 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Several studies have evaluated large language model performance on high-stakes examinations, including the USMLE, demonstrating passing-level performance without specific fine-tuning.

“Several studies have evaluated LLM performance on high-stakes examinations, including the United States Medical Licensing Examination (USMLE), demonstrating passing-level performance without specific fine-tuning (Brin et al. 2023;Bicknell et al. 2024;Kung et al. 2023).”

• cites Brin et al. 2023 sourcePartly supported

  The Discussion (page unavailable) says previous studies found that LLMs could pass the USMLE, and the Results report high accuracy on this paper’s USMLE-type questions. But this study tested only 80 selected soft-skills questions, not the full examination, and our copy does not establish the claim’s ‘without specific fine-tuning’ qualifier. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Bicknell et al. 2024 sourcePartly supported

  The Introduction’s “Previous Assessments of LLM Accuracy in Medical Contexts” reports that ChatGPT achieved passing scores on USMLE Step exams in a prior study. This paper’s own Methods describe testing ChatGPT through ordinary chat sessions, but the copy does not explicitly establish the claimed absence of specific fine-tuning; its Study Limitations also say its questions were USMLE-style, not actual exam questions. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Kung et al. 2023 sourceSupported

Classified as probably not a substantial claim.

Retrieval-augmented large language models with targeted retrievals and agentic workflows can provide more evidence-grounded and clinically actionable responses to real-world clinical questions than base models alone.

“Recent work has shown that these LLMs, augmented with targeted retrievals and agentic workflows, can provide more evidence-grounded and therefore clinically actionable responses to real-world clinical questions than base models can alone (Low et al. 2025).”

• cites Low et al. 2025 sourceSupported

May combine two claims.

Comparative evaluations of ChatGPT-5 and Gemini 2.5 on internal medicine subspecialty board examinations demonstrate higher overall accuracy for Gemini, while relative performance varies across subspecialties rather than remaining consistent across domains.

“More recent comparative evaluations of ChatGPT-5 and Gemini 2.5 on internal medicine subspecialty board examinations demonstrate higher overall accuracy for Gemini, while also showing that relative performance varies across subspecialties rather than remaining consistent across domains (Sheikhalishahi et al. 2025).”

• cites Sheikhalishahi et al. 2025 sourceSupported

May combine two claims.

Subsequent evaluations of GPT-4 and GPT-4 Omni documented substantially higher accuracy, frequently exceeding 88-90% correct responses, and outperforming earlier large language models on tasks requiring both factual recall and clinical reasoning.

“Subsequent evaluations of GPT-4 and GPT-4 Omni documented substantially higher accuracy, frequently exceeding 88-90% correct responses, and, most notably, outperforming earlier LLMs on tasks requiring both factual recall and clinical reasoning (Bicknell et al. 2024;Chen et al. 2024).”

• cites Bicknell et al. 2024 sourcePartly supported

  The Results report that GPT-4o scored 90.4% overall and outperformed GPT-4 and GPT-3.5; the diagnostics and management results also show improvements over earlier models. However, GPT-4 scored 81.1% overall, so the suggestion that both newer models frequently exceeded 88–90% is not supported by the overall results. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Chen et al. 2024 sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In medical education settings, OpenEvidence and similar platforms have been reported to assist students integrate guideline-based reasoning and efficiently locate relevant literature during clinical rotations.

“In medical education settings, OpenEvidence and similar platforms have been reported to assist students integrate guideline-based reasoning and efficiently locate relevant literature during clinical rotations (Patel et al. 2025).”

• cites Patel et al. 2025 sourceSupported

Classified as probably not a substantial claim. May combine two claims.

OpenEvidence is a clinician-facing large language model that grounds responses in cited, peer-reviewed medical sources, which may improve transparency and reduce hallucinations but does not eliminate them entirely.

“OpenEvidence (OpenEvidence, Inc.) is one such platform; it is a clinician-facing LLM that grounds responses in cited, peer-reviewed medical sources, which may improve transparency and reduce hallucinations but does not eliminate them entirely (Hurt et al. 2025).”

• cites Hurt et al. 2025 sourcePartly supported

  The Methods describe OpenEvidence as an AI-powered platform designed to assist clinicians, and the Conclusions say its responses offered support through citations of peer-reviewed sources. The paper does not establish that those citations improve transparency or reduce hallucinations. Its Introduction discusses hallucinations as a limitation of ChatGPT, not as an outcome assessed for OpenEvidence. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Performance varied by orthopaedic subspecialty but showed no statistically significant differences between models (p > 0.05 for all pairwise comparisons).

“Performance varied by orthopaedic subspecialty but showed no statistically significant differences between models (p > 0.05 for all pairwise comparisons).”

May combine two claims.

OpenEvidence's incorrect responses showed a qualitatively different pattern: explanations were either absent or consisted of brief, citation-grounded statements that described correct clinical principles yet arrived at the wrong answer choice, suggesting errors more consistent with answer-selection limitations than reasoning or knowledge failures.

“OpenEvidence's incorrect responses showed a qualitatively different pattern: explanations were either absent or consisted of brief, citation-grounded statements that described correct clinical principles yet arrived at the wrong answer choice, suggesting errors more consistent with answer-selection limitations than reasoning or knowledge failures.”

May combine two claims.

Gemini 2.5 Pro achieved the highest overall accuracy at 80.0% (212/265), followed by ChatGPT-5 at 78.1% (207/265).

“Gemini 2.5 Pro achieved the highest overall accuracy at 80.0% (212/265), followed by ChatGPT-5 at 78.1% (207/ 265).”

May combine two claims.

The largest numerical discrepancy was observed in Oncology, where Gemini 2.5 Pro achieved higher accuracy than ChatGPT-5 by 22.7 percentage points (77.3% vs 54.5%), though this did not reach statistical significance.

“The largest numerical discrepancy was observed in Oncology, where Gemini 2.5 Pro achieved higher accuracy than ChatGPT-5 by 22.7 percentage points (77.3% vs. 54.5%),though this did not reach statistical significance.”

May combine two claims.

In the shared text-only subset, Gemini 2.5 Pro and OpenEvidence each achieved 84.7% accuracy, while ChatGPT-5 slightly trailed at 81.1%.

“In this shared subset, where all three models could be directly compared, Gemini 2.5 Pro and OpenEvidence each achieved 84.7% accuracy, while ChatGPT-5 Plus slightly trailed both at 81.1%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A total of 265 multiple-choice questions were included in the final dataset.

“A total of 265 multiple-choice questions were included in the final dataset.”

May combine two claims.

On text-only questions, all three models demonstrated highest accuracy in Adult Spine, with OpenEvidence performing in the same range as the multimodal models.

“On text only questions, all three models demonstrated highest accuracy in Adult Spine, with OpenEvidence performing in the same range to the multimodal models (Figure 2).”

May combine two claims.

Within the text-only subset, OpenEvidence achieved 100% accuracy on Foot and Ankle questions, whereas ChatGPT-5 Plus scored 77.3% followed by Gemini 2.5 Pro scored at 68.2%.

“Within the text-only subset, OpenEvidence achieved 100% accuracy on Foot and Ankle questions, whereas ChatGPT-5 Plus scored 77.3% followed by Gemini 2.5 Pro scored at 68.2% here (Table 3, Figure 3).”

May combine two claims.

For T2 interpretation questions, Gemini 2.5 Pro showed numerically higher accuracy than ChatGPT-5 (76.3% vs 68.4%; p = 0.44), though this was not statistically significant.

“In contrast, Gemini 2.5 Pro showed numerically higher accuracy than ChatGPT-5 on questions requiring interpretation of images or data (T2) (76.3% vs. 68.4%;p = 0.44), though this data was not statistically significant.”

May combine two claims.

On the subset of 111 text-only questions, Gemini 2.5 Pro and OpenEvidence achieved identical accuracy (84.7%, 94/111), outperforming ChatGPT-5 (81.1%, 90/111).

“On the subset of 111 text-only questions where all three models could be directly compared, Gemini 2.5 Pro and OpenEvidence achieved identical accuracy (84.7%, 94/111), outperforming ChatGPT-5 (81.1%, 90/111).”

May combine two claims.

ChatGPT-5, Gemini 2.5 Pro, and OpenEvidence demonstrated high accuracy on the 2014 OITE, with performance at or above published senior resident benchmarks.

“In this study, ChatGPT-5, Gemini 2.5 Pro, and OpenEvidence demonstrated high accuracy on the 2014 OITE, with performance at or above published senior resident benchmarks.”

Classified as probably not a substantial claim.

Questions were stratified into text-only (n=111) and image-based (n=154) subsets.

“Questions were further stratified into text-only (n=111) and image-based (n=154) subsets to accommodate the different input capabilities of the AI models.”

May combine two claims.

For factual recall (T1) questions, ChatGPT-5 achieved 88.1% accuracy compared to Gemini 2.5 Pro 84.7% (p = 0.59) and OpenEvidence 86.4% (p = 0.87).

“For factual recall (T1) questions, Chat-GPT-5 achieved slightly higher accuracy (88.1%) compared to Gemini 2.5 Pro (84.7%; p = 0.59) and OpenEvidence (86.4%; p = 0.87).”

May combine two claims.

OpenEvidence's strong performance in our text-only OITE questions is consistent with these findings and suggests that retrieval-based reasoning can perform as well as or better than general multimodal models on exam-style clinical questions, particularly when questions rely on standards of care.

“OpenEvidence's strong performance in our text-only OITE questions is consistent with these findings and suggests that retrieval-based reasoning can perform as well as or better than general multimodal models on exam-style clinical questions, particularly when questions rely on standards of care (Table 2).”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Comparing performance across different model architectures may help orthopaedic trainees and educators decide which tools are appropriate for evidence-based studying and rotation-level clinical reasoning, and identify where supervision is most needed.

“Rather than identifying a single 'best' model, comparing performance across different model architectures may help orthopaedic trainees and educators decide which tools appropriate for evidencebased studying and rotation-level clinical reasoning, and identify where supervision is most needed given known limitations (Xu et al. 2025).”

• cites Xu et al. 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

OpenEvidence's Taxonomy 3 performance was comparable to that of Gemini 2.5 Pro despite its lack of multimodal input, suggesting that explicit evidence retrieval can support reasoning on clinical scenarios involving multiple steps.

“OpenEvidence's Taxonomy 3 performance was comparable to that of Gemini 2.5 Pro despite its lack of multimodal input, suggesting that explicit evidence retrieval can support reasoning on clinical scenarios involving multiple steps (Table 1 and 2).”

May combine two claims.

The Orthopaedic In-Training Examination serves as the primary standardized assessment for orthopaedic residents in the United States and a predictor of performance on the American Board of Orthopaedic Surgery Part 1 certifying examination.

“The Orthopaedic In-Training Examination (OITE), administered annually by the American Academy of Orthopaedic Surgeons (AAOS), serves as the primary standardized assessment for orthopaedic residents in the United States and a predictor of performance on the American Board of Orthopaedic Surgey (ABOS) Part 1 certifying examination (Swanson et al. 2013).”

• cites Swanson et al. 2013 sourceNot checked — no full text of the source

May combine two claims.

ChatGPT-5 Plus performed best on Taxonomy 1 (recall-oriented) items, aligning with its broad exposure to factual medical content and its strength in direct knowledge retrieval.

“ChatGPT-5 Plus performed best on Taxonomy 1 (recall-oriented) items, aligning with its broad exposure to factual medical content and its strength in direct knowledge retrieval.”

May combine two claims.

The absolute differences were small (a 3.6 percentage point difference between ChatGPT-5 and the other two models) and did not reach statistical significance (p = 0.71), indicating that no model consistently outperformed the others on purely text-based items.

“The absolute differences were small (a 3.6 percentage point difference between ChatGPT-5 and the other two models), and they did not reach statistical significance (p = 0.71), indicating that no model consistently outperformed the others on purely text-based items (Table 2).”

May combine two claims.

Subspecialty analyses were limited by small question counts per domain, ranging from 16 questions in Hand to 49 in Trauma, which reduces statistical power and limits the reliability of domain-level comparisons.

“Subspecialty analyses were further limited by small question counts per domain, ranging from 16 questions in Hand to 49 in Trauma, which reduces statistical power and limits the reliability of domain-level comparisons.”

Classified as probably not a substantial claim.

Large language model performance remains imperfect, particularly for questions that require nuanced clinical judgment, integration of multiple pieces of information, or differentiation between closely related management options.

“Despite these high accuracies reported across prior studies, LLM performance remains imperfect, particularly for questions that require nuanced clinical judgment, integration of multiple pieces of information, or differentiation between closely related management options.”

Classified as probably not a substantial claim. May combine two claims.

OpenEvidence currently supports text-only input, so it was evaluated solely on the text-only question subset (n=111).

“As OpenEvidence currently supports text-only input, it was evaluated solely on the text-only question subset (n=111).”

May combine two claims.

While the T2 image-based difference did not reach statistical significance, it suggests potential variation in visual reasoning capability between model architectures.

“While this difference did not reach statistical significance, it suggests potential variation in visual reasoning capability between model architectures.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In several cases, ChatGPT-5 appeared to apply more contemporary clinical guidelines that diverge from the 2014 examination key, which may reflect the model's training on more recent literature.

“In several cases, the model appeared to apply more contemporary clinical guidelines that diverge from the 2014 examination key, which may itself reflect the model's training on more recent literature.”

May combine two claims.

A model answering correctly by 2014 standards could theoretically provide outdated guidance by contemporary standards, and performance on this exam may not predict accuracy on more recent OITE administrations.

“As a result, a model answering correctly by 2014 standards could theoretically provide outdated guidance by contemporary standards, and performance on this exam may not predict accuracy on more recent OITE administrations.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Prior work has examined older model generations or focused on general medical licensing tests, but no previous study has assessed these newer architectures, head-to-head, on a specialty-specific orthopaedic assessment.

“Prior work has examined older model generations or focused on general medical licensing tests, but no previous study has assessed these newer architectures, head-to-head, on a specialty-specific orthopaedic assessment.”

Classified as probably not a substantial claim.

The comparable performance of OpenEvidence and general-purpose models on text-based questions, combined with its qualitatively distinct error profile, suggests that retrieval-grounded platforms may offer a more transparent and verifiable starting point for evidence-based self-study.

“The comparable performance of OpenEvidence and general-purpose models on text-based questions, combined with its qualitatively distinct error profile, suggests that retrieval-grounded platforms may offer a more transparent and verifiable starting point for evidence-based self-study -particularly for subspecialties with well-established clin-”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paid versions of these models were used, so performance may differ for users who utilize the free versions, which may rely on older or smaller models.

“We also used the paid versions of these models, so performance may differ for users who utilize the free versions, which may rely on older or smaller models.”

May combine two claims.

These factors may influence external validity and limit conclusions about performance on newer exams, updated model releases, or current clinical standards.

“These factors may influence external validity and limit conclusions made about performance on newer exams, updated model releases, or current clinical standards.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Resident benchmarks were taken from prior literature, and newer model versions such as ChatGPT 5.2 and Gemini 3.0 were not used in the study.

“In addition, resident benchmarks were taken from prior literature, and newer model versions such as ChatGPT 5.2 and Gemini 3.0 were not used in our study.”

May combine two claims.

General-purpose large language models from different model families have converged to a similarly high performance level on structured, multiple-choice assessments, with variability concentrated more in subspecialty-specific reasoning style than in raw score.

“This data suggests that general-purpose LLMs from different model families have converged to a similarly high performance level on structured, multiple-choice assessments, with variability concentrated more in subspecialty-specific reasoning style than in raw score.”

Classified as probably not a substantial claim. May combine two claims.

The overlap likely reflects convergence on well-established clinical knowledge, but shared exposure to publicly available exam content during model training cannot be excluded as contributing to agreement on specific items.

“While this overlap likely reflects convergence on well-established clinical knowledge, it cannot be excluded that shared exposure to publicly available exam content during model training contributed to agreement on specific items, a possibility that underscores the need for contamination-resistant evaluation designs in future work.”

May combine two claims.

OpenEvidence does not allow for image interpretation, which only allows comparison of all three models using text-only questions.

“OpenEvidence also does not allow for image interpretation, which only allows us to compare all three models using text only questions.”

May combine two claims.

This study is the first to directly compare the performance of three contemporary frontier large language models on the OITE, and the first to evaluate a retrieval-augmented system on this exam format.

“This study is the first to directly compare the performance of three contemporary frontier LLMs (ChatGPT 5 Plus, Gemini 2.5 Pro, and Open Evidence) on the OITE, and the first to evaluate a retrieval-augmented system (OpenEvidence) on this exam format.”

May combine two claims.

Across the text-only questions, all three models agreed on the correct answer for 72.1% of items, and only 5.4% were missed by all models, demonstrating substantial overlap in their strengths.

“Across the text-only questions, all three models agreed on the correct answer for 72.1% of items, and only 5.4% were missed by all models, demonstrating substantial overlap in their strengths (Figure 4).”

May combine two claims.

This pattern is consistent with OpenEvidence contributing the highest number of unique correct responses (n=7), suggesting that when it diverged from the other models it did so on the basis of evidence-grounded reasoning rather than pattern-matched responses.

“This pattern is consistent with OpenEvidence contributing the highest number of unique correct responses among the three models (n=7, Figure 4), suggesting that when it diverged from the other models it did so on the basis of evidencegrounded reasoning rather than pattern-matched responses.”

Probably not a substantial claim: may not pass S4 (based).

Prior exposure of large language models to the 2014 OITE during model training cannot be excluded.

“Additionally, prior exposure of LLMs to the 2014 OITE during model training cannot be excluded.”

May combine two claims.

While overall accuracy was similar across models, differences in performance on specific subspecialties and cognitive tasks reflect variation in how each model handles distinct types of clinical knowledge.

“These results indicate that while overall accuracy was similar across models, differences in performance on specific subspecialties within orthopaedic surgery and cognitive tasks reflect variation in how each model handles distinct types of clinical knowledge.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Examination accuracy reflects a narrow dimension of medical knowledge and should not be equated with the clinical reasoning or real-world competency of senior trainees.

“However, examination accuracy reflects a narrow dimension of medical knowledge and should not be equated with the clinical reasoning or real-world competency of senior trainees.”

Classified as probably not a substantial claim. May combine two claims.

ChatGPT-5 and Gemini 2.5 Pro may offer broader utility across question types including image and video-based items, but their tendency toward confident errors warrants active verification of model outputs rather than passive acceptance.

“ChatGPT-5 and Gemini 2.5 Pro may offer broader utility across question types including image and video-based items, but their tendency toward confident errors warrants active verification of model outputs rather than passive acceptance.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

These limitations likely reflect the probabilistic nature of current models and their reliance on pattern recognition rather than true causal clinical reasoning.

“These limitations likely reflect the probabilistic nature of current models and their reliance on pattern recognition rather than true causal clinical reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Classifications were done by two reviewers S.P and S.H and disagreements were resolved by a third review, A.K.

“Classifications were done by two reviewers S.P and S.H and disagreements were resolved by a third review, A.K.”

May combine two claims.

OpenEvidence performed comparably to the multimodal models despite lacking visual input capabilities, excelling particularly in text-based reasoning (T3) questions (82.7%).

“Notably, OpenEvidence performed comparably to the multimodal models despite lacking visual input capabilities, excelling particularly in text-based reasoning (T3) questions (82.7%).”

Classified as probably not a substantial claim.

A standardized prompt structure was used for all interactions to ensure consistency and to minimize variability.

“A standardized prompt structure was used for all interactions to ensure consistency and to minimize variability (Figure 1).”

Classified as probably not a substantial claim. May combine two claims.

Direct three-way comparison across all models is limited to the shared text-only subset, while ChatGPT-5 and Gemini 2.5 Pro were additionally evaluated across the full 265-question dataset.

“Consequently, direct three-way comparison across all models is limited to this shared subset, while ChatGPT-5 and Gemini 2.5 Pro were additionally evaluated across the full 265-question dataset including image and video-based items.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study was exempt from Institutional Review Board oversight because it involved publicly available, de-identified educational materials and no human subjects.

“As this study involved publicly available, de-identified educational materials and no human subjects, it was exempt from Institutional Review Board (IRB) oversight.”

Classified as probably not a substantial claim.

Questions were classified by cognitive taxonomy based on Buckwalter et al: Taxonomy 1 recall and recognition of facts, Taxonomy 2 interpretation of data, images, or diagnostic studies, and Taxonomy 3 complex clinical reasoning, problem-solving, and management decisions.

“Additionally, questions were classified by cognitive taxonomy based on established medical education frameworks defined by Buckwalter et al: Taxonomy 1 (T1): Recall and recognition of facts, Taxonomy 2 (T2): Interpretation of data, images, or diagnostic studies, and Taxonomy 3 (T3): Complex clinical reasoning, problem-solving, and management decisions (Buckwalter et al. 1981).”

• cites Buckwalter et al. 1981 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The chat context was reset after every question to prevent information leakage or memory bias between items.

“The chat context was reset after every question to prevent information leakage or memory bias between items.”