Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; 9 of the 15 sources its claims cite were checked for support.

Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial

Open access

Abstract

Prospective, randomized controlled trial to determine whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources...

Overview

CoChat Summary

## TL;DR

In this randomized controlled trial of 92 physicians, access to GPT-4 via ChatGPT Plus improved performance on open-ended management reasoning cases compared with conventional resources alone. The LLM group also spent more time per case, and GPT-4 alone scored similarly to physicians using GPT-4.

## Key findings

  • Physicians using the LLM scored higher than those using conventional resources alone: 43.0% vs 35.7%, mean difference 6.5% (95% CI 2.7 to 10.2), p<0.001.
  • Improvements were seen in multiple domains: management decisions (40.5% vs 33.4%, difference 6.1%, 95% CI 2.5 to 9.7, p=0.001), diagnostic decisions (56.8% vs 45.8%, difference 12.1%, 95% CI 3.1 to 21.0, p=0.009), and context-specific questions (42.4% vs 34.9%, difference 6.2%, 95% CI 2.4 to 9.9, p=0.002).
  • GPT-4 users spent more time per case: 801.5 seconds vs 690.2 seconds, mean difference 119.3 seconds (95% CI 17.4 to 221.2), p=0.022.
  • GPT-4 alone performed comparably to physicians using GPT-4: 43.7% vs 43.0%, difference 0.9% (reported as −0.9% in the abstract), p=0.80.

## Methods

  • Prospective, randomized, single-blind controlled trial conducted from 30 November 2023 to 21 April 2024 across Stanford University, Beth Israel Deaconess Medical Center, and the University of Virginia, with physicians from across the United States.
  • 92 practicing attending physicians and residents in internal medicine, family medicine, or emergency medicine were randomized to GPT-4 via ChatGPT Plus plus conventional resources or conventional resources alone; a separate LLM-only comparison arm was also analyzed.
  • Participants worked through five expert-developed clinical case vignettes with open-ended management questions; outcomes were scored with Delphi-developed rubrics. Analyses were at the case level, clustered by participant, using generalized mixed-effects models.

## Limitations

  • The paper says the cases were clinical vignettes, not actual patient cases.
  • The paper says validity evidence for the rubric outside this study had not been gathered, and that only potentially correct answers were given credit while wrong answers were not penalized.
  • The paper says the case set was intentionally limited to five cases expected within a one-hour session, so a wider variety of cases could show different outcomes.
  • Evident from the text: the study used basic LLM training and technical support only, so results may depend on a relatively minimal implementation of LLM use in practice.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2024.08.05.24311485

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1: Study Flow Diagram 92 practicing attending physicians and residents with training in internal medicine, family medicine, or emergency medicine. Five expert-developed cases were presented, with scoring rubrics created through a Delphi process. Physicians were randomized to use either GPT-4 via ChatGPT Plus in addition to conventional resources (e.g., UpToDate, Google), or conventional resources alone. The primary outcome was difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case.
Figure 1: Study Flow Diagram 92 practicing attending physicians and residents with training in internal medicine, family medicine, or emergency medicine. Five expert-developed cases were presented, with scoring rubrics created through a Delphi process. Physicians were randomized to use either GPT-4 via ChatGPT Plus in addition to conventional resources (e.g., UpToDate, Google), or conventional resources alone. The primary outcome was difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case.CC BY-ND · PubMed Central
Figure 2. Comparisons of the Primary Outcome by Physicians with LLM and with Conventional Resources Only (Total score standardized to 0–100)
Figure 2. Comparisons of the Primary Outcome by Physicians with LLM and with Conventional Resources Only (Total score standardized to 0–100)CC BY-ND · PubMed Central

Verification

Claims the paper makes

5 Valid · 0 Not valid · 1 Not checked · 21 Own results and interpretations

9 more claims

Create a free account to see all 27 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 7
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 28 references

Cited by

See all 13

Metadata

Authors
Goh, E., Gallo, R., Strong, E., Weng, Y., Kerman, H., Freed, J., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., & Rodman, A.
Affiliations
  • Stanford University (US)
  • VA Palo Alto Health Care System (US)
  • Stanford Medicine (US)
  • Beth Israel Deaconess Medical Center (US)
  • University of Minnesota (US)
  • University of Virginia (US)
  • Microsoft (United States) (US)
  • Kaiser Permanente (US)
Corresponding author
Chen, Jonathan H
Venue
medRxiv
Publisher
openRxiv
Published
2024-08-07
OpenAlex
W4401436581
Cited by
13
References
28
Citations by year
2026: 4 · 2025: 8 · 2024: 1
Type
posted content
Language
en
Open access
Locations
  • medRxiv (acceptedVersion, cc-by-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nd)
Funders
National Institute on Drug Abuse (UG1DA015815)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, GPT-4, ChatGPT Plus, clinical decision support, physician decision-making, clinical case vignettes, diagnostic reasoning, physician performance, internal medicine, general practice, emergency medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. https://app.cochat.ai/papers/10.1101/2024.08.05.24311485
The paper
Goh, E., Gallo, R., Strong, E., Weng, Y., Kerman, H., Freed, J., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., & Rodman, A. (2024). Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. medRxiv. https://doi.org/10.1101/2024.08.05.24311485

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.