Physicians’ Diagnostic Accuracy, Confidence, and Resource Requests: A Vignette Study
Abstract
IMPORTANCE: Little is known about the relationship between physicians' diagnostic accuracy and their confidence in that accuracy. OBJECTIVE: To evaluate how physicians' diagnostic calibration, defined as the relationship between diagnostic accuracy and confidence in that accuracy, changes with evolution of the diagnostic process and with increasing diagnostic difficulty of clinical case vignettes. DESIGN, SETTING, AND PARTICIPANTS: We recruited general internists from an online physician community and asked them to diagnose 4 previously validated case vignettes of variable difficulty (2 easier; 2 more difficult). Cases were presented in a web-based format and divided into 4 sequential phases simulating diagnosis evolution: history, physical examination, general diagnostic testing data, and definitive diagnostic testing. After each phase, physicians recorded 1 to 3 differential diagnoses and corresponding judgments of confidence. Before being presented with definitive diagnostic data, physicians were asked to identify additional resources they would require to diagnose each case (ie, additional tests, second opinions, curbside consultations, referrals, and reference materials). MAIN OUTCOMES AND MEASURES: Diagnostic accuracy (scored as 0 or 1), confidence in diagnostic accuracy (on a scale of 0-10), diagnostic calibration, and whether additional resources were requested (no or yes). RESULTS: A total of 118 physicians with broad geographical representation within the United States correctly diagnosed 55.3% of easier and 5.8% of more difficult cases (P < .001). Despite a large difference in diagnostic accuracy between easier and more difficult cases, the difference in confidence was relatively small (7.2 vs 6.4 out of 10, for easier and more difficult cases, respectively) (P < .001) and likely clinically insignificant. Overall, diagnostic calibration was worse for more difficult cases (P < .001) and characterized by overconfidence in accuracy. Higher confidence was related to decreased requests for additional diagnostic tests (P = .01); higher case difficulty was related to more requests for additional reference materials (P = .01). CONCLUSIONS AND RELEVANCE: Our study suggests that physicians' level of confidence may be relatively insensitive to both diagnostic accuracy and case difficulty. This mismatch might prevent physicians from reexamining difficult cases where their diagnosis may be incorrect.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The successful extraction of two critical task parameters (endoscopists' own accuracy and the ai's accuracy) was likely at the core of the endoscopists' ability to select and follow the best ai advice.
“The successful extraction of two critical task parameters was likely at the core of this ability: endoscopists could intuitively but reliably predict for each lesion both their accuracy (not obvious37,38) and the accuracy of the ai (not obvious39).”
From Experimental evidence of effective human–AI collaboration in medical decision-making (2022) · cites this paper as [38]
Cites
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477 · see citing works
A cognitive perspective on medical expertise
Schmidt, H. G., Norman, G. R., & Boshuizen, H. P. A. (1990) · Academic Medicine · doi:10.1097/00001888-199010000-00001 · Cited by 1,135 · see citing works
Overconfidence as a Cause of Diagnostic Error in Medicine
Berner, E. S., & Graber, M. L. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.01.001 · Cited by 1,034 · see citing works
Diagnostic Error in Medicine: Analysis of 583 Physician-Reported Errors
Schiff, G. D. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.333 · Cited by 638 · see citing works
Types and Origins of Diagnostic Errors in Primary Care Settings
Singh, H., Giardina, T. D., Meyer, A. N. D., Forjuoh, S. N., Reis, M. D., & Thomas, E. J. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.2777 · Cited by 559 · see citing works
Self-citationTwo methods of reducing overconfidence
Arkes, H. R., Christensen, C., Lai, C., & Blumer, C. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90049-5 · Cited by 277 · see citing works
Overconfidence in Clinical Decision Making
Croskerry, P., & Norman, G. R. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.02.001 · Cited by 263 · see citing works
Diagnostic Errors in Medicine: A Case of Neglect
Graber, M. (2005) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(05)31015-4 · Cited by 259 · see citing works
Why Doctors Don't Disclose Uncertainty
Katz, J. (1984) · The Hastings Center Report · doi:10.2307/3560848 · Cited by 221 · see citing works
Enhancement of Clinicians' Diagnostic Reasoning by Computer-Based Consultation
Friedman, C. P., Elstein, A. S., Wolf, F. M., Murphy, G. C., Franz, T. M., Heckerling, P. S., Fine, P. L., Miller, T. M., & Abraham, V. (1999) · JAMA · doi:10.1001/jama.282.19.1851 · Cited by 219 · see citing works
Friedman, C. P., Gatti, G. G., Franz, T. M., Murphy, G. C., Wolf, F. M., Heckerling, P. S., Fine, P. L., Miller, T. M., & Elstein, A. S. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.30145.x · Cited by 187 · see citing works
Diagnostic errors and reflective practice in medicine
Mamede, S., Schmidt, H. G., & Rikers, R. M. J. P. (2006) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2006.00638.x · Cited by 160 · see citing works
Minimizing Diagnostic Error: The Importance of Follow-up and Feedback
Schiff, G. D. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.02.004 · Cited by 157 · see citing works
Understanding diagnostic errors in medicine: a lesson from aviation
Singh, H. (2006) · Quality and Safety in Health Care · doi:10.1136/qshc.2005.016444 · Cited by 156 · see citing works
Self-citationConfidence and accuracy in deductive reasoning
Shynkaruk, J. M., & Thompson, V. A. (2006) · Memory & Cognition · doi:10.3758/bf03193584 · Cited by 145 · see citing works
Confidence-Accuracy Calibration in Absolute and Relative Face Recognition Judgments.
Weber, N., & Brewer, N. (2004) · Journal of Experimental Psychology Applied · doi:10.1037/1076-898x.10.3.156 · Cited by 126 · see citing works
The association between physician risk tolerance and imaging use in abdominal pain
Pines, J. M., Hollander, J. E., Isserman, J. A., Chen, E. H., Dean, A. J., Shofer, F. S., & Mills, A. M. (2009) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2008.04.031 · Cited by 111 · see citing works
Should we confirm our clinical diagnostic certainty by autopsies?
Podbregar, M., Voga, G., Krivec, B., Skale, R., Parežnik, R., & Gabršček, L. (2001) · Intensive Care Medicine · doi:10.1007/s00134-001-1129-x · Cited by 87 · see citing works
Davis, D. P., Campbell, C., Poste, J. C., et al. (2005) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2005.02.008 · Cited by 52 · see citing works
Are Clinicians Correct When They Believe They Are Correct? Implications for Medical Decision Support
Charles, F., Guido, G., Arthur, E., Timothy, F., Gwendolyn, M., & Fredric, W. (2001) · Studies in Health Technology and Informatics · doi:10.3233/978-1-60750-928-8-454 · Cited by 41 · see citing works
Effects of a Decision Support System on the Diagnostic Accuracy of Users: A Preliminary Report
Elstein, A. S., Friedman, C. P., Wolf, F. M., Murphy, G., Miller, J., Fine, P. L., Heckerling, P. S., Miller, T., Sisson, J. C., Barlas, S., et al. (1996) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1996.97084515 · Cited by 36 · see citing works
Yazbek, J., Ameye, L., Testa, A. C., et al. (2009) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.7335 · Cited by 28 · see citing works
Miller, T. M., Franz, T. M., Murphy, G. C., Heckerling, P. S., Wolf, F. M., Gatti, G., Friedman, C. P., Fine, P. L., & Elstein, A. S. (2020) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/bc12-ke43 · Cited by 17 · see citing works
Murphy, G. C., Friedman, C. P., Elstein, A. S., et al. (1996) · PubMed · Cited by 14 · see citing works
The Relationship between Diagnostic Accuracy and Confidence in Medical Students.
Mann, D. (1993) · Cited by 13 · see citing works
Diagnostic Error: Is Overconfidence the Problem?
Graber, M. L. (2008) · Cited by 11 · see citing works
Wolf, F. M., Friedman, C. P., Elstein, A. S., et al. (1997) · PubMed · Cited by 9 · see citing works
Ultrasound operators' confidence influences diagnosis of ovarian tumors - a study in China.
Haiyan, H., & Min, D. (2011) · PubMed · Cited by 3 · see citing works
Lindberg, D. A. B. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.02.007 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Cognitive biases associated with medical decisions: a systematic review
Saposnik, G., Redelmeier, D., Ruff, C. C., & Tobler, P. N. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0377-1 · Cited by 989 · see citing works
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · see citing works
The Global Burden of Diagnostic Errors in Primary Care
Singh, H., Schiff, G. D., Graber, M. L., Onakpoya, I., & Thompson, M. J. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-005401 · Cited by 395 · see citing works
Self-citationExperimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 235 · see citing works
Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0096 · Cited by 154 · see citing works
Grewal, J. K., Tessier-Cloutier, B., Jones, M., Gakkhar, S., Ma, Y., Moore, R., Mungall, A. J., Zhao, Y., Taylor, M. D., Gelmon, K., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.2597 · Cited by 90 · see citing works
Thinking forensics: Cognitive science for forensic practitioners
Edmond, G., Towler, A., Growns, B., Ribeiro, G., Found, B., White, D., Ballantyne, K., Searston, R. A., Thompson, M. B., Tangen, J. M., et al. (2017) · Science & Justice · doi:10.1016/j.scijus.2016.11.005 · Cited by 95 · see citing works
Naguib, M., Brull, S. J., Hunter, J. M., Kopman, A. F., Fülesdi, B., Johnson, K. B., & Arkes, H. R. (2019) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000003714 · Cited by 82 · see citing works
The accuracy of German citizens’ confidence in their climate change knowledge
Fischer, H., Amelung, D., & Said, N. (2019) · Nature Climate Change · doi:10.1038/s41558-019-0563-0 · Cited by 78 · see citing works
Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key
Zwaan, L., & Hautz, W. E. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-009078 · Cited by 63 · see citing works
Developing Checklists to Prevent Diagnostic Error in Emergency Room Settings
Graber, M. L., Sorensen, A. V., Biswas, J., Modi, V., Wackett, A., Johnson, S., Lenfestey, N., Meyer, A. N. D., & Singh, H. (2014) · Diagnosis · doi:10.1515/dx-2014-0019 · Cited by 49 · see citing works
Self-citationChecklists to Prevent Diagnostic Errors: A Pilot Randomized Controlled Trial
Ely, J. W., & Graber, M. A. (2015) · Diagnosis · doi:10.1515/dx-2015-0008 · Cited by 37 · see citing works
Maude, J. (2014) · Diagnosis · doi:10.1515/dx-2013-0009 · Cited by 29 · see citing works
Improving Medical Residents’ Self-Assessment of Their Diagnostic Accuracy: Does Feedback Help?
Kuhn, J., van den Berg, P., Mamede, S., Zwaan, L., Bindels, P., & van Gog, T. (2022) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10080-9 · Cited by 21 · see citing works
The economics of diagnostic safety
Slawomirski, L., Kelly, D., de Bienassis, K., Kallas, K.-A., & Klazinga, N. (2025) · OECD Health Working Papers · doi:10.1787/fc61057a-en · Cited by 20 · see citing works
- Authors
- Meyer, A. N. D., Payne, V. L., Meeks, D. W., Rao, R., & Singh, H.
- Affiliations
- Michael E. DeBakey VA Medical Center (US)
- Venue
- JAMA Internal Medicine
- Publisher
- American Medical Association (AMA)
- Published
- 2013-11-25
- Volume
- 173
- Issue
- 21
- Pages
- 1952
- ISSN
- 2168-6106 (ISSN-L)
- OpenAlex
- W1972825759
- Cited by
- 268
- References
- 29
- Citations by year
- 2026: 30 · 2025: 14 · 2024: 22 · 2023: 23 · 2022: 25 · 2021: 32 · 2020: 25 · 2019: 30 · 2018: 21 · 2017: 9 · 2016: 14 · 2015: 7 · 2014: 15
- FWCI
- 10.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JAMA Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Public Health Service (T32HP10031)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Health Sciences › Health Professions › General Health Professions › Primary Care and Health Outcomes
- Keywords
- diagnostic calibration, diagnostic confidence, diagnostic accuracy, overconfidence, case difficulty, diagnostic process, resource requests, diagnostic testing, second opinion, curbside consultation, general internists, case vignettes
- MeSH terms
- Adult, Clinical Competence, Diagnosis, Differential, Diagnostic Errors, Female, Humans, Internal Medicine, Male, Middle Aged, Practice Patterns, Physicians', Physicians, Referral and Consultation, United States, Diagnostic Techniques and Procedures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
