Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key
Open accessAbstract
Picture yourself in a casino sitting at the roulette table. Will you put your money on either black or red, or will you go for a riskier choice and place it on just a single number? But before you can make up your mind, the elderly, obese smoker to your right reaches for his chest and collapses. ‘What is wrong with him?’, cries his accompanying wife, looking at you in shock. In this issue, Lawton and colleagues ask whether more experienced clinicians are better able to tolerate uncertainty and manage risks.1 Doctors working in three emergency departments (ED) read four clinical vignettes each accompanied by four quite distinct options for management, all of which might be deemed clinically acceptable. Participants used 5-point Likert scales to indicate their agreement with each of the management plans. By design, the four options offered for each vignette included two management plans one would characterise as ‘risk averse’—requiring further tests and/or admission of the patient to hospital. The other two management plans may be seen as less risk averse, usually involving referring a patient back to the general practitioner or offering reassurance to the patient. In addition to rating their agreement with each of the possible management plans for the four vignettes, participants reported their experience (number of years worked in an ED setting) and answered a 15-item ‘Physicians Reactions to Uncertainty’ scale. Analysing the responses from 90 doctors, who had worked in EDs for between 5 weeks and 21 years, Lawton and colleagues report a medium to large correlation between experience and tolerance of risk. More experienced clinicians made less risk averse decisions. We would like to provide three conceptual considerations potentially relevant to interpret these results from Lawton and colleagues and identify possible follow-up research. Roulette serves as a classic example for …
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The ability to recognize knowledge limitations enables clinicians to dynamically shift between System 1 and System 2 cognitive strategies.
“Crucially, the ability to recognize knowledge limitations enables clinicians to dynamically shift between these cognitive strategies33.”
From Large Language Models lack essential metacognition for reliable medical reasoning (2025) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cognitive Psychology and Its Implications
Anderson, J. R. (2014) · Cited by 4,121 · see citing works
Gigerenzer, G., & Gaissmaier, W. (2011) · Annual Review of Psychology · doi:10.1146/annurev-psych-120709-145346 · Cited by 4,037 · see citing works
Deliberate practice and acquisition of expert performance: a general overview
Anders Ericsson, K. (2008) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2008.00227.x · Cited by 1,827 · see citing works
Baron, J. A. (2006) · Cambridge University Press eBooks · doi:10.1017/9781009263672 · Cited by 1,434 · see citing works
Overconfidence as a Cause of Diagnostic Error in Medicine
Berner, E. S., & Graber, M. L. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.01.001 · Cited by 1,034 · see citing works
Oppenheimer, D. M. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.014 · Cited by 686 · see citing works
Diagnostic error and clinical reasoning
Norman, G. R., & Eva, K. W. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03507.x · Cited by 500 · see citing works
Kachalia, A. B., Gandhi, T. K., Puopolo, A. L., Yoon, C. S., Thomas, E. J., Griffey, R. T., Brennan, T. A., & Studdert, D. M. (2006) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2006.06.035 · Cited by 495 · see citing works
“I'll never play professional football” and other fallacies of self-assessment
Eva, K. W., & Regehr, G. (2008) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.150 · Cited by 452 · see citing works
Defining and Measuring Diagnostic Uncertainty in Medicine: A Systematic Review
Bhise, V., Rajan, S. S., Sittig, D. F., Morgan, R. O., Chaudhary, P., & Singh, H. (2017) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4164-1 · Cited by 308 · see citing works
A certain art of uncertainty: case presentation and the development of professional identity
Lingard, L., Garwood, K., Schryer, C. F., & Spafford, M. M. (2003) · Social Science & Medicine · doi:10.1016/s0277-9536(02)00057-6 · Cited by 270 · see citing works
Physicians’ Diagnostic Accuracy, Confidence, and Resource Requests: A Vignette Study
Meyer, A. N. D., Payne, V. L., Meeks, D. W., Rao, R., & Singh, H. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.10081 · Cited by 268 · see citing works
Rates and Characteristics of Paid Malpractice Claims Among US Physicians by Specialty, 1992-2014
Schaffer, A. C., Jena, A. B., Seabury, S. A., Singh, H., Chalasani, V., & Kachalia, A. B. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.0311 · Cited by 227 · see citing works
Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review
Lambe, K. A., O'Reilly, G., Kelly, B. D., & Curristan, S. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004417 · Cited by 215 · cited 1× · see citing works
“Such metacognition enables us to recognise how confident we are in a decision, 16 thus, for instance, recognising when to slow down when we should 17–19 and to over-rule first, intuitive impressions with more deliberate reasoning.”
Brown, T. W., McCarthy, M. L., Kelen, G. D., & Levy, F. (2010) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2010.00729.x · Cited by 213 · see citing works
Friedman, C. P., Gatti, G. G., Franz, T. M., Murphy, G. C., Wolf, F. M., Heckerling, P. S., Fine, P. L., Miller, T. M., & Elstein, A. S. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.30145.x · Cited by 187 · see citing works
Hatala, R. M., Brooks, L. R., & Norman, G. R. (2003) · Advances in Health Sciences Education · doi:10.1023/a:1022687404380 · Cited by 167 · see citing works
Ilgen, J. S., Eva, K. W., de Bruin, A. B. H., Cook, D. A., & Regehr, G. (2018) · Advances in Health Sciences Education · doi:10.1007/s10459-018-9859-5 · Cited by 142 · see citing works
Moulton, C.-A. E., Regehr, G., Lingard, L. A., Merritt, C. E., & MacRae, H. (2010) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-010-1178-y · Cited by 121 · see citing works
Pusic, M. V., Pecaric, M. R., & Boutis, K. (2011) · Academic Medicine · doi:10.1097/acm.0b013e3182178c3c · Cited by 120 · see citing works
Monitoring and regulation of learning in medical education: The need for predictive cues
de Bruin, A. B. H., Dunlosky, J., & Cavalcanti, R. B. (2017) · Medical Education · doi:10.1111/medu.13267 · Cited by 112 · see citing works
Lawton, R., Robinson, O. C., Harrison, R., et al. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008390 · Cited by 98 · see citing works
Assessing Diagnostic Reasoning: A Consensus Statement Summarizing Theory, Practice, and Future Needs
Ilgen, J. S., Humbert, A. J., Kuhn, G. J., Hansen, M. L., Norman, G. R., Eva, K. W., Charlin, B., & Sherbino, J. (2012) · Academic Emergency Medicine · doi:10.1111/acem.12034 · Cited by 79 · see citing works
Omron, R., Kotwal, S., Garibaldi, B. T., & Newman‐Toker, D. E. (2018) · AEM Education and Training · doi:10.1002/aet2.10119 · Cited by 41 · see citing works
Putting the puzzle together: the role of ‘problem definition’ in complex clinical judgement
Cristancho, S., Lingard, L. A., Forbes, T. L., Ott, M. C., & Novick, R. J. (2016) · Medical Education · doi:10.1111/medu.13210 · Cited by 38 · see citing works
Calibrating how doctors think and seek information to minimise errors in diagnosis
Meyer, A. N. D., & Singh, H. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-006071 · Cited by 33 · see citing works
Accuracy of self‐monitoring: does experience, ability or case difficulty matter?
Hautz, W. E., Schubert, S., Schauber, S. K., Kunina‐Habenicht, O., Hautz, S. C., Kämmer, J. E., & Eva, K. W. (2019) · Medical Education · doi:10.1111/medu.13801 · Cited by 29 · see citing works
Self-citationImplementation of diagnostic pauses in the ambulatory setting
Huang, G. C., Kriegel, G., Wheaton, C., Sternberg, S. B., Sands, K., Richards, J. B., Johnston, K. T., & Aronson, M. D. (2018) · BMJ Quality & Safety · doi:10.1136/bmjqs-2017-007192 · Cited by 22 · see citing works
The Effect of Performance Standards and Medical Experience on Diagnostic Calibration Accuracy
Nederhand, M. L., Tabbers, H. K., Splinter, T. A. W., & Rikers, R. M. J. P. (2018) · Health Professions Education · doi:10.1016/j.hpe.2017.12.008 · Cited by 19 · see citing works
Hautz, W. E. (2018) · Medical Education · doi:10.1111/medu.13602 · Cited by 17 · see citing works
Self-citationClinical practice, deliberate practice, and “big data”
Norman, G. R. (2018) · Advances in Health Sciences Education · doi:10.1007/s10459-018-9856-8 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Gigerenzer G, Todd PM. Simple Heuristics That Make Us Smart. Oxford: Oxford Univ Press, 2001.
- Meyer AND, Payne VL, Meeks DW. Confidence, and resource requests: a vignette study. JAMA Intern Med 2013;173:1952-8.
Cited by
Sorted by citations
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
“Crucially, the ability to recognize knowledge limitations enables clinicians to dynamically shift between these cognitive strategies 33 .”
Improving Medical Residents’ Self-Assessment of Their Diagnostic Accuracy: Does Feedback Help?
Kuhn, J., van den Berg, P., Mamede, S., Zwaan, L., Bindels, P., & van Gog, T. (2022) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10080-9 · Cited by 21 · see citing works
Self-citation
- Authors
- Zwaan, L., & Hautz, W. E.
- Affiliations
- Erasmus MC (NL)
- University of Bern (CH)
- Corresponding author
- Zwaan, Laura
- Venue
- BMJ Quality & Safety
- Publisher
- BMJ
- Published
- 2019-02-06
- Volume
- 28
- Issue
- 5
- Pages
- 352-355
- ISSN
- 1475-3898 (ISSN-L)
- 2044-5415
- 2044-5423
- OpenAlex
- W2915056872
- Cited by
- 63
- References
- 31
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 13 · 2023: 8 · 2022: 9 · 2021: 10 · 2020: 7
- FWCI
- 6.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- BMJ Quality & Safety (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- BORIS (University Library Bern) (submittedVersion, cc-by)
- HighWire Press Open Archive (submittedVersion)
- Open Access CRIS of the University of Bern
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (451-16-032)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Keywords
- clinical uncertainty, risk tolerance, diagnostic accuracy, calibration, physician experience, emergency department clinicians, risk aversion, clinical decision-making, diagnostic confidence
- MeSH terms
- Calibration, Emergency Service, Hospital, England, State Medicine, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
