The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; 9 of the 15 sources its claims cite were checked for support.
Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial
Open accessAbstract
Prospective, randomized controlled trial to determine whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources...
Overview
CoChat Summary
## TL;DR
In this randomized controlled trial of 92 physicians, access to GPT-4 via ChatGPT Plus improved performance on open-ended management reasoning cases compared with conventional resources alone. The LLM group also spent more time per case, and GPT-4 alone scored similarly to physicians using GPT-4.
## Key findings
- Physicians using the LLM scored higher than those using conventional resources alone: 43.0% vs 35.7%, mean difference 6.5% (95% CI 2.7 to 10.2), p<0.001.
- Improvements were seen in multiple domains: management decisions (40.5% vs 33.4%, difference 6.1%, 95% CI 2.5 to 9.7, p=0.001), diagnostic decisions (56.8% vs 45.8%, difference 12.1%, 95% CI 3.1 to 21.0, p=0.009), and context-specific questions (42.4% vs 34.9%, difference 6.2%, 95% CI 2.4 to 9.9, p=0.002).
- GPT-4 users spent more time per case: 801.5 seconds vs 690.2 seconds, mean difference 119.3 seconds (95% CI 17.4 to 221.2), p=0.022.
- GPT-4 alone performed comparably to physicians using GPT-4: 43.7% vs 43.0%, difference 0.9% (reported as −0.9% in the abstract), p=0.80.
## Methods
- Prospective, randomized, single-blind controlled trial conducted from 30 November 2023 to 21 April 2024 across Stanford University, Beth Israel Deaconess Medical Center, and the University of Virginia, with physicians from across the United States.
- 92 practicing attending physicians and residents in internal medicine, family medicine, or emergency medicine were randomized to GPT-4 via ChatGPT Plus plus conventional resources or conventional resources alone; a separate LLM-only comparison arm was also analyzed.
- Participants worked through five expert-developed clinical case vignettes with open-ended management questions; outcomes were scored with Delphi-developed rubrics. Analyses were at the case level, clustered by participant, using generalized mixed-effects models.
## Limitations
- The paper says the cases were clinical vignettes, not actual patient cases.
- The paper says validity evidence for the rubric outside this study had not been gathered, and that only potentially correct answers were given credit while wrong answers were not penalized.
- The paper says the case set was intentionally limited to five cases expected within a one-hour session, so a wider variety of cases could show different outcomes.
- Evident from the text: the study used basic LLM training and technical support only, so results may depend on a relatively minimal implementation of LLM use in practice.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
5 Valid · 0 Not valid · 1 Not checked · 21 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“LLMs have not yet shown to perform similarly in management reasoning, which encompasses decision making around treatment, testing, patient preferences, social determinants of health, and cost-conscious care, all while managing risk.6–8”
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Introduction
What the sources say
- Not checked — no full text of the sourcecites [6] source
- Partly supportedcites [7] sourceExperimental — accuracy not yet measured
The Abstract and Conclusion report that GPT-4 produced largely accurate orthopedic treatment recommendations but did not fully consider patients’ situations or treatment urgency; the Discussion says it cannot replace professional medical management. This supports a limitation in management reasoning, but the pilot study did not assess the full range of testing, patient preferences, social determinants, and cost-conscious care described in the citing sentence. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [8] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to studies showing increased empathetic communication phrasing from LLMs to patient queries, this study indicates that LLMs may influence physicians to better consider human factors in their management reasoning.26–28”
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Discussion In this randomized controlled trial, the availability of an LLM improved physician management reasoning compared to conventional resources only, with comparable scores between physicians randomized to use AI and the AI alone. This suggests a future use for LLMs as a helpful adjunct to clinician judgment. From a cognitive psychology perspective, it is surprising that an LLM would enhance management reasoning. The abilities of LLMs to make diagnostic decisions likely results from their underlying token prediction architecture and its similarities to how physicians cluster and activate semantic illness scripts in making diagnoses.24 Management scripts, on the other hand, are highly contextual and individualized, and include many factors outside of the biomedical encounter. Thus, the best decision for a patient in a given situation may be different than another patient with the exact same condition in a different context. For example, the appropriate management of an incidentally discovered 2.0 centimeter upper lobe lung nodule in a hospitalized inpatient might be immediate biopsy in a patient unlikely to follow up; scheduled outpatient biopsy in a health system capable of organizing and ensuring continuity; outpatient PET scan in a patient reticent to undergo an invasive procedure; or serial imaging in a patient with limited life expectancy. The knowledge that such a large nodule in the upper lobe has a high chance of representing malignancy is only the first step in formulating a follow up plan – patient preferences, knowledge of the healthcare system, and the patient’s social situation are equally important factors. The group using the LLM also spent more time solving cases, another surprising finding, and in contrast to findings in diagnostic reasoning.4,5 Interacting with the LLM may have served as a “time out” to drive better decisions and consider the patient’s context. For example, physicians using the LLM often considered patient and other contextual factors in management decisions, as well as exhibiting apparent empathy to other providers and patients in difficult situations. We suspect that some of these emergent abilities come from the fine-tuning process called reinforcement learning through human feedback (RLHF), in which empathetic and patient-centered responses are rated as favorable by humans.25 Similar to studies showing increased empathetic communication phrasing from LLMs to patient queries, this study indicates that LLMs may influence physicians to better consider human factors in their management reasoning.26–28 Improved humanistic and patient-centered behaviors of clinicians when they collaborate with an LLM is an important and even reassuring finding even at the expense of taking more time. This study has multiple limitations. First, all our cases are clinical vignettes, based on, but not actual patient cases. While our scoring rubrics show substantial inter-rater reliability, validity evidence for this rubric has not been gathered outside this study. Only potentially correct answers were given credit, while wrong answers were not penalized. LLM-based outputs are often significantly longer than human responses, and it is thus possible that physicians in the AI group scored better merely through verbosity alone.
What the sources say
- Supportedcites [26] source
The citation supports the comparison to studies of empathetic LLM responses to patient queries, not the citing authors’ finding about physicians’ management reasoning. The Discussion reports that chatbot responses to patient questions were rated significantly higher for empathy than physician responses. (LLM, conf=0.91)
- Partly supportedcites [27] sourceExperimental — accuracy not yet measured
The Results (Likelihood to Recommend; page not indicated) report that physicians valued the empathy in AI-drafted patient replies, and the Discussion says such drafts may reduce cognitive burden. The paper does not establish that the drafts led physicians to better consider human factors in their clinical management reasoning; its reply examples show physicians making their own substantive clinical changes. (LLM, conf=0.82)
- Could not verifycites [28] sourceExperimental — accuracy not yet measured
The study’s discussion (page unavailable in our copy) says LLM assistance changed physicians’ responses and might lead them to adopt an LLM’s assessment. It does not assess whether physicians better considered human factors in their management reasoning; its mention of “human factors” concerns how LLMs should be evaluated. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and general management knowledge (29.4% versus 26.5%, difference: 3.3, 95% CI: −1.3% to 7.9%, p = 0.2)”
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Results > Question domain subgroups
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Physicians randomized to use GPT-4 spent 111.3 seconds more on each case (801.5 seconds versus 690.2, difference: 119.3 seconds, 95% CI: 17.4 to 221.2, p = 0.022).”
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Results > Time
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While we were unable to detect a difference between the two groups in factual recall (62.9% versus 53.8%, difference: 9.6%, 95% CI: −3.1% to 22.3%, p = 0.14)”
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Results > Question domain subgroups
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
9 more claims
Create a free account to see all 27 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 9
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Abstract > Results:
“GPT-4 users spent more time per case (mean difference 119.3 seconds, 95% CI 17.4–221.2, p=0.02).”
§ Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial > Results > Time
“Physicians randomized to use GPT-4 spent 111.3 seconds more on each case (801.5 seconds versus 690.2, difference: 119.3 seconds, 95% CI: 17.4 to 221.2, p = 0.022).”
Citations
Cites
Qualitative Research: Consensus methods for medical and health services research
Jones, J., & Hunter, D. (1995) · BMJ · doi:10.1136/bmj.311.7001.376 · Cited by 3,471
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Ledley, R. S., & Lusted, L. B. (1959) · Science · doi:10.1126/science.130.3366.9 · Cited by 991
Educational Strategies to Promote Clinical Diagnostic Reasoning
Bowen, J. L. (2006) · New England Journal of Medicine · doi:10.1056/nejmra054782 · Cited by 931 · cited 1×
“11 Current frameworks in management reasoning include context-dependent concepts such as shared decision making, dynamic relationships and competing priorities between medical systems and individuals, the physician-patient relationship, and time constraints inherent in modern clinical encounters.”
BackgroundPrompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · cited 1×
“23 Each prompt was run five times, and the results from the five runs were included for blinded grading alongside the human outputs before any unblinding or data analysis.”
MethodKhan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.818634 · Cited by 490 · cited 1×
“17,18”
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449
Self-citationTowards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 281 · cited 1×
“14 One of the theoretical strengths of LLMs is their ability to serve as a cooperation partner, augmenting human cognition.”
Background
Cited by
Roles and potential of Large language models in healthcare: A comprehensive review
Lin, C., & Kuo, C. (2025) · Biomedical Journal · doi:10.1016/j.bj.2025.100868 · Cited by 140
“While studies like Goh et al. demonstrate improved decision-making quality through physician-LLM collaboration [50], the actual value lies in how these tools support the adaptive capacity of healthcare professionals.”
Application of large language models in disease diagnosis and treatment
Yang, X., Li, T., Su, Q., Liu, Y., Kang, C., Lyu, Y., Zhao, L., Nie, Y., & Pan, Y. (2024) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003456 · Cited by 100
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28
Blease, C., Kharko, A., Sanchez, C. G., Alderman, J. E., Kumpunen, S., Sundemo, D. S., & Torous, J. (2025) · Journal of Medical Internet Research · doi:10.2196/74428 · Cited by 35
Kerman, H. M., Siden, R., Cool, J. A., Hom, J., Goh, E., Ahuja, N. K., Shieh, L. Y., Heidenreich, P. A., Yang, D., Rodman, A. M., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-025-10145-0 · Cited by 5
Self-citation
Metadata
- Authors
- Goh, E., Gallo, R., Strong, E., Weng, Y., Kerman, H., Freed, J., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., & Rodman, A.
- Affiliations
- Stanford University (US)
- VA Palo Alto Health Care System (US)
- Stanford Medicine (US)
- Beth Israel Deaconess Medical Center (US)
- University of Minnesota (US)
- University of Virginia (US)
- Microsoft (United States) (US)
- Kaiser Permanente (US)
- Corresponding author
- Chen, Jonathan H
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-08-07
- PMC
- PMC11326321
- OpenAlex
- W4401436581
- Cited by
- 13
- References
- 28
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by-nd)
- Funders
- National Institute on Drug Abuse (UG1DA015815)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, GPT-4, ChatGPT Plus, clinical decision support, physician decision-making, clinical case vignettes, diagnostic reasoning, physician performance, internal medicine, general practice, emergency medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. https://app.cochat.ai/papers/10.1101/2024.08.05.24311485
- The paper
- Goh, E., Gallo, R., Strong, E., Weng, Y., Kerman, H., Freed, J., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., & Rodman, A. (2024). Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. medRxiv. https://doi.org/10.1101/2024.08.05.24311485
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.