Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations

Classified as probably not a substantial claim. May combine two claims.

Another study compared OpenEvidence responses to real clinical dilemmas, including a blinded comparison with human-generated responses, and found comparable subjective ratings of reviewer satisfaction.

“Another study compared OpenEvidence responses to real clinical dilemmas, including a blinded comparison with human-generated responses, and found comparable subjective ratings of reviewer satisfaction.15”

• cites [15] Ben Shitrit, 2026, Real-world human-LLM Interactions-Prospective blinded versus unblinded expert physician assessments of LLM responses to complex medical dilemmas, PLOS Digit Health sourcePartly supported

  The Methods (Study design) confirm that physicians assessed OpenEvidence responses to real clinical dilemmas alongside manually written human responses in a blinded stage. But the comparable overall satisfaction ratings in Results (Table 1) compare blinded and unblinded LLM responses, not OpenEvidence with human responses. The Discussion says blinded reviewers rated human responses higher on reliability, new information, and clarity, so the claimed comparability with human responses is not established by those passages. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

One study reported that specialists preferred LLM responses compared with generalist physician responses in a pilot study using real-world medical questions.

“One study reported how specialists preferred LLM responses compared with generalist physician responses in a pilot study using real-world medical questions.10”

• cites [10] Singhal, 2025, Toward expert-level medical question answering with large language models, Nat Med sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

More specialized LLM tools, such as OpenEvidence, have reported high accuracy and performance on the real clinical questions submitted by physicians.

“More specialized LLM tools, such as OpenEvidence, have reported high accuracy and performance on the real clinical questions submitted by physicians.14”

• cites [14] Vishwanath, 2026, General-purpose large language models outperform specialized clinical AI tools on medical benchmarks, Nat Med sourcePartly supported

  The paper evaluates OpenEvidence on 100 real physician queries using blinded clinician ratings (RCQ evaluation; Fig. 2), so the setting and performance assessment match. But it does not report an accuracy percentage for those queries, and it places OpenEvidence below the general-purpose LLMs in clinician-rated performance. Its 89.6% accuracy figure comes from MedQA questions, not the real clinical queries. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Absolute scores in this study were lower than those reported in the original eSQUARE validation study.

“Notably, absolute scores in this study were lower than those reported in the original eSQUARE validation study.25”

• cites [25] Tran, 2022, eConsult specialist quality of response (eSQUARE): a novel tool to measure specialist correspondence via electronic consultation, J Telemed Telecare sourceSupported

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.

May combine two claims.

OpenEvidence, a context-specific LLM for medical applications, can provide clinical decision support and synthesize peer-reviewed literature and clinical guidelines for health care clinicians.

“OpenEvidence, a context-specific LLM for medical applications, can provide clinical decision support and synthesize peer-reviewed literature and clinical guidelines for health care clinicians.12”

• cites [12] Hurt, 2025, The Use of an artificial intelligence platform OpenEvidence to augment clinical decision-making for primary care physicians, J Prim Care Community Health sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

eConsults can lead to decreased health care utilization and cost savings.

“eConsults can also lead to decreased health care utilization and cost savings.8,9”

• cites [8] Whittington, 2021, Cost savings associated with electronic specialty consultations, Am J Manag Care sourceNot checked — no full text of the source
• cites [9] Keely, 2013, Utilization, Benefits, and Impact of an e-consultation service across diverse specialties and primary care providers, Telemed J E Health sourcePartly supported

  The Results report that a planned face-to-face referral was avoided in 43% of submitted cases, supporting the claim that eConsults can reduce specialist-service use. The paper does not report cost savings or a cost analysis, so that part of the claim is unsupported. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

The eSQUARE instrument used is an 11-item validated tool designed to appraise eConsult quality across General, Management, and Communication domains, with strong validity evidence.

“The eSQUARE instrument is an 11-item validated tool designed to appraise eConsult quality across 3 domains, General, Management, and Communication, along with a single-item global rating.25”

• Also stated in: § eSQUARE Instrument and Comparison of Specialist and AI eConsults · boosted

  “The instrument reports strong validity evidence, including content validity, internal structure, and evidence of relations with other variables.”

• cites [25] Tran, 2022, eConsult specialist quality of response (eSQUARE): a novel tool to measure specialist correspondence via electronic consultation, J Telemed Telecare sourcePartly supported

  Results > Tool development describes 10 assessment items plus a separate global rating scale, not an 11-item tool with an additional global rating. The Discussion reports initial validity evidence and high reliability for assessing specialist eConsult replies. The passages do not identify General, Management, and Communication as three domains. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

MASLD impacts an estimated 25%-35% of adults worldwide.

“MASLD impacts an estimated 25%-35% of adults worldwide, and because most patients with MASLD are managed by PCCs or general internists, the condition represents a common source of eConsult requests.20,21”

• cites [20] Younossi, 2016, Global epidemiology of nonalcoholic fatty liver disease—meta-analytic assessment of prevalence, incidence, and outcomes, Hepatology sourceNot checked — no full text of the source
• cites [21] Kanwal, 2021, Clinical care pathway for the risk stratification and management of patients with nonalcoholic fatty liver disease, Gastroenterology sourcePartly supported

  The paper’s Abstract reports that approximately 37% of US adults have NAFLD, supporting the broad point that the condition is common. It does not give a worldwide estimate of 25%–35%; both the population and the stated range differ. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

GI eConsults are an established model with substantial evidence supporting their accuracy and clinical utility.

“In contrast, GI eConsults are an established model with substantial evidence supporting their accuracy and clinical utility.30”

• cites [30] Anderson, 2021, Implications of electronic consultations for clinician communication and relationships: a qualitative study, Med Care sourcePartly supported

  The Abstract says e-consults are increasingly used across health systems and facilitate access to specialty expertise, supporting the general point that they are an established, clinically useful model. The Results (Documentation of Communication) report clinicians’ perceptions of more reliable advice, but the paper does not establish the accuracy of GI eConsults or substantial GI-specific evidence for their clinical utility. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

eConsults increase access to specialty care in rural and underserved areas, leading to faster specialist access and improved satisfaction among primary care clinicians.

“They increase access to specialty care in rural and underserved areas, leading to faster specialist access and improved satisfaction among PCCs.7”

• cites [7] Lee, 2020, Electronic Consultations (eConsults) for safe and equitable coordination of virtual outpatient specialty care, Appl Clin Inform sourcePartly supported

  The main text (page unavailable) reports faster specialist input and reduced waits for patients facing access barriers, including those in safety-net settings. It identifies rural patients as facing barriers, but does not specifically show improved access for rural patients. It also reports higher satisfaction only among clinicians who believe in eConsults’ mission, not a general increase in primary care clinician satisfaction caused by eConsults. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous studies have shown that LLM responses may receive higher ratings than physician responses for patient-facing communication.

“Previous studies have shown that LLM responses may receive higher ratings than physician responses for patient-facing communication, although length, organization, and perceived empathy may have influenced reviewer preferences.13”

• cites [13] Ayers, 2023, Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum, JAMA Intern Med sourceSupported

Classified as probably not a substantial claim.

Large language models like OpenAI's GPT-5 and knowledge-specific systems such as OpenEvidence have reported the ability to generate patient-specific, evidence-based recommendations.

“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”

• cites [10] Singhal, 2025, Toward expert-level medical question answering with large language models, Nat Med sourcePartly supported

  The Abstract and “Chain of retrieval” section report that Med-PaLM 2 answered real-world bedside consultation questions and used external medical information to improve grounding. They do not report this ability for GPT-5 or OpenEvidence, or establish that either system generates patient-specific, evidence-based recommendations. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

The original eSQUARE validation study reported that the instrument could distinguish among eConsults categorized as low, moderate, or high quality, but it did not establish validated score thresholds or a minimally important difference for these values.

“The original eSQUARE validation study reported that the instrument could distinguish among eConsults categorized as low, moderate, or high quality. However, it did not establish validated score thresholds or a minimally important difference for these values.”

• cites [25] Tran, 2022, eConsult specialist quality of response (eSQUARE): a novel tool to measure specialist correspondence via electronic consultation, J Telemed Telecare sourceSupported

Classified as probably not a substantial claim.

eConsults enable asynchronous communication between primary care clinicians and specialists without in-person visits.

“eConsults enable asynchronous communication between PCCs and specialists without in-person visits.5”

• cites [5] Thind, 2025, Evaluating the impact of an eConsult platform on specialty care access for Medicaid patients, J Am Board Fam Med sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Prior studies on clinical documentation suggest that structured formats improve readability and may be perceived as higher quality.

“Prior studies on clinical documentation suggest that structured formats improve readability and may be perceived as higher quality.27,28”

• cites [27] Ebbers, 2022, The impact of structured and standardized documentation on documentation quality: a multicenter, retrospective study, J Med Syst sourceSupported
• cites [28] Klappe, 2023, Correctly structured problem lists lead to better and faster clinical decision-making in electronic health records compared to non-curated problem lists: a single-blinded crossover randomized controlled trial, Int J Med Inform sourceNot checked — no full text of the source

May combine two claims.

OpenEvidence-generated eConsults received higher ratings of perceived quality and informational adequacy than gastroenterologist-authored responses for MASLD, although clinically relevant errors were identified.

“OpenEvidence-generated eConsults received higher ratings of perceived quality and informational adequacy than gastroenterologist-authored responses for MASLD, although clinically relevant errors were identified.”

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.
• AssumptionRests on an assumption: Removing citations from OpenEvidence-generated eConsults before blinded review was sufficient to prevent reviewers from identifying the source and did not alter the perceived quality of the responses.

May combine two claims.

OpenEvidence-generated eConsults were substantially longer and more consistently structured than specialist-authored eConsults, and reviewers more often judged them to contain an appropriate amount of information.

“OpenEvidence-generated eConsults were substantially longer and more consistently structured than specialist-authored eConsults, and reviewers more often judged them to contain an appropriate amount of information.”

May combine two claims.

Individual reviewers reported consistent findings across all domains and global quality ratings, with OpenEvidence-generated eConsults receiving higher scores than GIH eConsults.

“Individual reviewers reported consistent findings across all domains and global quality ratings, with OpenEvidence-generated eConsults receiving higher scores than GIH eConsults.”

May combine two claims.

OpenEvidence-generated eConsults had a mean of 639 words, a median of 644 words, a minimum of 279 words, and a maximum of 848 words.

“OpenEvidence-generated eConsults were longer, with a mean of 639 words, a median of 644 words, a minimum of 279 words, and a maximum of 848 words.”

May combine two claims.

OpenEvidence-generated responses had higher word counts, with the 2.5th percentile of the distribution at ∼450 words, and reported a more modest increase in eSQUARE total score across their word count range.

“OpenEvidence-generated responses had higher word counts, with the 2.5th percentile of the distribution at ∼450 words, and reported a more modest increase in eSQUARE total score across their word count range (Supplemental Figure 2, available online at https://www.mcpdigitalhealth.org/).”

May combine two claims.

The median word count was 297 (Q1, Q3: 215-356) for GIH responses and 668 (619-734) for OpenEvidence-generated responses, with minimal overlap between distributions.

“The median (Q1, Q3) word count was 297 (215-356) words for GIH responses and 668 (619-734) words for OpenEvidence-generated responses, with minimal overlap between the distributions (Supplemental Figure 1, available online at https://www.mcpdigitalhealth.org/).”

May combine two claims.

GIH eConsults had a mean of 318 words, a median of 295 words, a minimum of 77 words, and a maximum of 1571 words.

“GIH eConsults had a mean of 318 words, a median of 295 words, a minimum of 77 words, and a maximum of 1571 words.”

May combine two claims.

Errors were identified in 13 of 125 OpenEvidence-generated eConsults (10.4%), including 12 minor errors and 1 potentially major error (0.8%).

“Errors were identified in 13 of 125 OpenEvidence-generated eConsults (10.4%), including 12 responses with minor errors and 1 response with an error classified as potentially major (0.8%).”

May combine two claims.

Although there was heterogeneity in absolute scores among reviewers, the overall pattern of findings was consistent across all reviewers.

“Although there was heterogeneity in absolute scores among reviewers, the overall pattern of findings was consistent across all reviewers.”

May combine two claims.

Each reviewer identified statistically significant differences between OpenEvidence-generated and GIH eConsults across all domains and global quality ratings, although there was heterogeneity in absolute scores.

“Although there was heterogeneity in absolute scores, each reviewer identified statistically significant differences between OpenEvidence-generated and GIH eConsults across all domains and global quality ratings (Table 1).”

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.
• AssumptionRests on an assumption: Removing citations from OpenEvidence-generated eConsults before blinded review was sufficient to prevent reviewers from identifying the source and did not alter the perceived quality of the responses.

May combine two claims.

OpenEvidence-generated eConsults consistently followed a structured format, including headers, bullet points, and numbered problem lists, whereas GIH eConsults more commonly consisted of longer narrative text, often without separation between clinical assessment and management recommendations.

“OpenEvidence-generated eConsults consistently followed a structured format, including headers, bullet points, and numbered problem lists. In contrast, GIH eConsults more commonly consisted of longer narrative text, often without separation between clinical assessment and management recommendations.”

May combine two claims.

Among GIH responses, scores increased with increasing length among shorter responses but plateaued at approximately 350-400 words, whereas OpenEvidence responses were substantially longer overall and reported only a modest upward trend across their observed range.

“Among GIH responses, scores increased with increasing length among shorter responses but plateaued at ∼350-400 words, whereas OpenEvidence responses were substantially longer overall and reported only a modest upward trend across their observed range.”

Probably not a substantial claim: may not pass S3 (load-bearing).

As a retrospective single-center study at a tertiary referral center, findings may not generalize to community practice settings or to health systems with different eConsult workflows.

“First, as a retrospective single-center study at a tertiary referral center, findings may not generalize to community practice settings or to health systems with different eConsult workflows.”

Classified as probably not a substantial claim.

The findings provide a rationale for using OpenEvidence as a drafting or decision support tool with specialist oversight.

“Rather, they provide a rationale using OpenEvidence as a drafting or decision support tool with specialist oversight.”

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.

Probably not a substantial claim: may not pass S2 (checkable).

OpenEvidence-generated recommendations should be considered decision support rather than autonomous clinical care.

“At the same time, the detection of clinically relevant errors, including one potentially major error, highlights that OpenEvidence-generated recommendations should be considered decision support rather than autonomous clinical care.”

May combine two claims.

The study was not powered to estimate uncommon but potentially serious errors, so the reported error frequencies should not be interpreted as generalizable estimates of platform safety.

“In addition, the study was not powered to estimate uncommon but potentially serious errors. As such, the reported error frequencies should not be interpreted as generalizable estimates of platform safety.”

Classified as probably not a substantial claim.

Reviewers were blinded to the consult type (AI vs GIH specialist) during the review process.

“Reviewers were blinded to the consult type (AI vs GIH specialist) during the review process.”

Classified as probably not a substantial claim.

Both AI and GIH eConsults were reformatted into text documents to reduce stylistic cues.

“Both AI and GIH eConsults were reformatted into text documents to reduce stylistic cues.”

May combine two claims.

Reviewers differed in their overall scoring stringency while the direction of the comparison remained consistent across reviewers.

“This suggests that reviewers differed in their overall scoring stringency while the direction of the comparison remained consistent across reviewers.”

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.

Probably not a substantial claim: may not pass S3 (load-bearing).

The study was not designed to assess downstream utilization, referral avoidance, time to resolution, or patient-level safety outcomes.

“The study was also not designed to assess downstream utilization, referral avoidance, time to resolution, or patient-level safety outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Platform-internal system instructions and model settings were proprietary and were not visible to the investigators.

“Platform-internal system instructions and model settings were proprietary and were not visible to the investigators.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

If implemented, OpenEvidence-assisted workflows may be most valuable as a mechanism to improve consistency, readability, and completeness of consultation while still requiring clinician review for correctness and appropriateness to the individual patient.

“If implemented, OpenEvidence-assisted workflows may therefore be most valuable as a mechanism to improve consistency, readability, and completeness of consultation while still requiring clinician review for correctness and appropriateness to the individual patient.”

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.

Probably not a substantial claim: may not pass S3 (load-bearing).

All eConsults and referral notes were deidentified following safe harbor procedures.

“All eConsults and referral notes were deidentified following safe harbor procedures.26”

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

eSQUARE assesses physician reviewers' perceptions of the quality of specialist correspondence; higher total or domain scores do not establish greater diagnostic accuracy, superior management decisions, or superior underlying clinical reasoning.

“Fifth, eSQUARE assesses physician reviewers’ perceptions of the quality of specialist correspondence. Higher total or domain scores do not establish greater diagnostic accuracy, superior management decisions, or superior underlying clinical reasoning.”

Classified as probably not a substantial claim. May combine two claims.

To establish response process validity, all reviewers jointly evaluated 2 mock eConsults using the eSQUARE instrument and a study-specific item assessing the amount of information, and discussed item interpretation to ensure consistency.

“To establish response process validity, all reviewers jointly evaluated 2 mock eConsults using the eSQUARE instrument and a study-specific item assessing the amount of information. Reviewers discussed item interpretation to ensure consistency.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No identifiable patient data was retained during analysis.

“No identifiable patient data was retained during analysis.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This may enhance clarity and perceived usefulness, particularly when evaluating eConsults using a quality assessment instrument, such as eSQUARE.

“This may enhance clarity and perceived usefulness, particularly when evaluating eConsults using a quality assessment instrument, such as eSQUARE.”

• AssumptionRests on an assumption: The eSQUARE instrument is valid for assessing the quality of OpenEvidence-generated eConsults, not only specialist-authored eConsults.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The eSQUARE instrument does not independently establish diagnostic accuracy or assess the quality of underlying clinical reasoning.

“Although the instrument includes a management domain, it does not independently establish diagnostic accuracy or assess the quality of underlying clinical reasoning.”

May combine two claims.

The error review was performed by a single study-author hepatologist, was not independently duplicated, and did not include a formal adjudication process.

“The review was performed by a single study-author hepatologist, was not independently duplicated, and did not include a formal adjudication process.”

Probably not a substantial claim: may not pass S4 (based).

Data regarding the performance and safety of context-specific platforms such as OpenEvidence within eConsult workflows remain limited.

“Data regarding the performance and safety of context-specific platforms such as OpenEvidence within eConsult workflows remain limited.”

Classified as probably not a substantial claim.

The expert review was designed as an exploratory characterization of errors in OpenEvidence-generated responses rather than as a formally powered or independent safety assessment.

“The expert review was designed as an exploratory characterization of errors in the OpenEvidence-generated responses rather than as a formally powered or independent safety assessment.”

Classified as probably not a substantial claim.

The GIH and OpenEvidence-generated eConsults were randomized before review.

“The GIH eConsults and OpenEvidence-generated eConsults were randomized before review.”

Classified as probably not a substantial claim. May combine two claims.

Because the original referral order was unavailable, the referral question was extracted from the gastroenterologist-authored eConsult when explicitly documented; therefore, the absence of a recorded question did not necessarily indicate that the original referral lacked a discrete clinical question.

“Because the original referral order was unavailable, the referral question was extracted from the gastroenterologist-authored eConsult when explicitly documented. Therefore, the absence of a recorded question did not necessarily indicate that the original referral lacked a discrete clinical question.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study evaluated only OpenEvidence, a proprietary platform, using standardized prompts at a single point in time; performance may differ across platforms, model versions, specialties, clinical conditions, input formats, and prompting strategies, so findings should be interpreted as specific to OpenEvidence and not generalized to all LLM-generated eConsults.

“Third, this study evaluated only OpenEvidence, a proprietary platform, using standardized prompts at a single point in time. Performance may differ across platforms, model versions, specialties, clinical conditions, input formats, and prompting strategies. These findings should be interpreted as specific to OpenEvidence and should not be generalized to all LLM-generated eConsults.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The additional item assessing the amount of information has not been validated specifically for OpenEvidence-generated eConsults.

“Fourth, the additional item assessing the amount of information has not been validated specifically for OpenEvidence-generated eConsults.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Evolving standards of care remain a potential source of bias, and future work should incorporate temporally matched comparisons to further mitigate this concern.

“Regardless, evolving standards of care remain a potential source of bias, and future work should incorporate temporally matched comparisons to further mitigate this concern.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This study was reviewed and approved by the Mayo Clinic institutional review board and deemed to be of minimal risk.

“This study was reviewed and approved by the Mayo Clinic institutional review board and deemed to be of minimal risk.”

• Also stated in: § Ethics Statement · plain

  “This study was reviewed and approved by the Mayo Clinic Institutional Review Board and deemed to be of minimal risk.”

Classified as probably not a substantial claim. May combine two claims.

The present study did not evaluate a combined human-plus-AI workflow and therefore cannot determine whether OpenEvidence assistance improves final response quality, reduces specialist time or cognitive burden, or decreases or introduces clinical errors.

“However, the present study did not evaluate this combined workflow and therefore cannot determine whether OpenEvidence assistance improves final response quality, reduces specialist time or cognitive burden, or decreases or introduces clinical errors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All reviewers in the study jointly evaluated 2 mock eConsults and were instructed to avoid ceiling effects by applying the full Likert scale thoughtfully rather than defaulting to higher scores.

“All reviewers in our study jointly evaluated 2 mock eConsults and were instructed to avoid ceiling effects by applying the full Likert scale thoughtfully rather than defaulting to higher scores.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The specific underlying OpenEvidence model version was not displayed by the platform and therefore could not be recorded.

“The specific underlying model version was not displayed by the platform and therefore could not be recorded.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because the study focused on MASLD, a relatively protocol-driven consultation question with well-described noninvasive risk stratification and management pathways, performance may differ for less protocol-driven consult questions.

“Second, we focused on MASLD, a relatively protocol-driven consultation question with well-described noninvasive risk stratification and management pathways. Performance may differ for less protocol-driven consult questions.”

May combine two claims.

Although OpenEvidence generates cited recommendations, citations were removed before blinded review to reduce source-identifying cues; therefore, the study did not assess citation validity or the risk of hallucinated sources.

“Sixth, although OpenEvidence generates cited recommendations, we removed citations before blinded review to reduce source-identifying cues; therefore, we did not assess citation validity or the risk of hallucinated sources.”

May combine two claims.

GIH eConsults were generated from 2020 to 2024, during a period of evolving terminology and management for MASLD, while OpenEvidence-generated eConsults were produced at a single time point and may have reflected more current guideline language.

“Fourth, GIH eConsults were generated from 2020 to 2024, during a period of evolving terminology and management for MASLD. OpenEvidence-generated eConsults were produced at a single time point and may have reflected more current guideline language.”

Classified as probably not a substantial claim.

Five GIM physicians independently evaluated all eConsults for quality using the eSQUARE instrument.

“Five physicians (5 GIM physicians) independently evaluated all eConsults for quality using the eSQUARE instrument.”