Improving Medical Residents’ Self-Assessment of Their Diagnostic Accuracy: Does Feedback Help?
Open accessAbstract
When physicians do not estimate their diagnostic accuracy correctly, i.e. show inaccurate diagnostic calibration, diagnostic errors or overtesting can occur. A previous study showed that physicians’ diagnostic calibration for easy cases improved, after they received feedback on their previous diagnoses. We investigated whether diagnostic calibration would also improve from this feedback when cases were more difficult. Sixty-nine general-practice residents were randomly assigned to one of two conditions. In the feedback condition, they diagnosed a case, rated their confidence in their diagnosis, their invested mental effort, and case complexity, and then were shown the correct diagnosis (feedback). This was repeated for 12 cases. Participants in the control condition did the same without receiving feedback. We analysed calibration in terms of (1) absolute accuracy (absolute difference between diagnostic accuracy and confidence), and (2) bias (confidence minus diagnostic calibration). There was no difference between the conditions in the measurements of calibration (absolute accuracy, p = .204; bias, p = .176). Post-hoc analyses showed that on correctly diagnosed cases (on which participants are either accurate or underconfident), calibration in the feedback condition was less accurate than in the control condition, p = .013. This study shows that feedback on diagnostic performance did not improve physicians’ calibration for more difficult cases. One explanation could be that participants were confronted with their mistakes and thereafter lowered their confidence ratings even if cases were diagnosed correctly. This shows how difficult it is to improve diagnostic calibration, which is important to prevent diagnostic errors or maltreatment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Underconfident clinicians are likely to seek external information to validate and ensure their diagnostic accuracy, resulting in over testing and delays in decision making.
“underconfident clinicians are likely to seek external information to validate and ensure their diagnostic accuracy [106,107], resulting in over testing and delays in decision making [108].”
From Toward Safer Diagnoses: A SEIPS-Based Narrative Review of Diagnostic Errors (2026) · cites this paper as [108]
The Introduction says underconfidence in a correct diagnosis could lead to unnecessary further testing and lengthen the diagnostic process. The Discussion also mentions seeking more information or consulting a colleague, but the paper does not establish that underconfident clinicians are likely to do so to validate their diagnoses. Its account of overtesting and delay is a possibility, not a measured outcome. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349 · cited 1× · see citing works
“Each diagnosis was rated by two general practitioners with an ‘excellent’ interrater reliability, ICC = 0.96 (Cicchetti, 1994).”
MethodInfluentialZimmerman, B. J. (2008) · American Educational Research Journal · doi:10.3102/0002831207312909 · Cited by 3,735 · see citing works
Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006) · JAMA · doi:10.1001/jama.296.9.1094 · Cited by 2,375 · see citing works
Diagnostic error in internal medicine
Graber, M. L., Franklin, N., & Gordon, R. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.13.1493 · Cited by 1,549 · see citing works
Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning
Koriat, A. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.4.349 · Cited by 1,363 · see citing works
Overconfidence as a Cause of Diagnostic Error in Medicine
Berner, E. S., & Graber, M. L. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.01.001 · Cited by 1,034 · see citing works
Self-Assessment in the Health Professions: A Reformulation and Research Agenda
Eva, K. W., & Regehr, G. (2005) · Academic Medicine · doi:10.1097/00001888-200510001-00015 · Cited by 992 · see citing works
Slowing Down When You Should: A New Model of Expert Judgment
Moulton, C.-A. E., Regehr, G., Mylopoulos, M., & MacRae, H. (2007) · Academic Medicine · doi:10.1097/acm.0b013e3181405a76 · Cited by 315 · see citing works
Cardiac surgery report cards: comprehensive review and statistical critique
Shahian, D. M., Normand, S. T., Torchiana, D. F., Lewis, S. M., Pastore, J. O., Kuntz, R. E., & Dreyer, P. I. (2001) · The Annals of Thoracic Surgery · doi:10.1016/s0003-4975(01)03222-2 · Cited by 280 · see citing works
Physicians’ Diagnostic Accuracy, Confidence, and Resource Requests: A Vignette Study
Meyer, A. N. D., Payne, V. L., Meeks, D. W., Rao, R., & Singh, H. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.10081 · Cited by 268 · see citing works
Labuhn, A. S., Zimmerman, B. J., & Hasselhorn, M. (2010) · Metacognition and Learning · doi:10.1007/s11409-010-9056-2 · Cited by 240 · see citing works
Hacker, D. J., Bol, L., & Bahbahani, K. (2008) · Metacognition and Learning · doi:10.1007/s11409-008-9021-5 · Cited by 213 · see citing works
Friedman, C. P., Gatti, G. G., Franz, T. M., Murphy, G. C., Wolf, F. M., Heckerling, P. S., Fine, P. L., Miller, T. M., & Elstein, A. S. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.30145.x · Cited by 187 · see citing works
Minimizing Diagnostic Error: The Importance of Follow-up and Feedback
Schiff, G. D. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.02.004 · Cited by 157 · see citing works
Impact of a quarterly report card on colonoscopy quality measures
Kahi, C. J., Ballard, D. D., Shah, A. S., Mears, R., & Johnson, C. S. (2013) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2013.01.012 · Cited by 128 · see citing works
Improving students’ self-evaluation of learning for key concepts in textbook materials
Rawson, K. A., & Dunlosky, J. T. (2007) · The European Journal of Cognitive Psychology · doi:10.1080/09541440701326022 · Cited by 119 · see citing works
Monitoring and regulation of learning in medical education: The need for predictive cues
de Bruin, A. B. H., Dunlosky, J., & Cavalcanti, R. B. (2017) · Medical Education · doi:10.1111/medu.13267 · Cited by 112 · see citing works
Constraints on the Calibration of Performance
Schraw, G. J., Potenza, M. T., & Nebelsick-Gullet, L. (1993) · Contemporary Educational Psychology · doi:10.1006/ceps.1993.1034 · Cited by 109 · see citing works
Lipko, A. R., Dunlosky, J. T., Hartwig, M. K., Rawson, K. A., Swan, K. P., & Cook, D. L. (2009) · Journal of Experimental Psychology Applied · doi:10.1037/a0017599 · Cited by 74 · see citing works
Improving college students’ evaluation of text learning using idea-unit standards
Dunlosky, J. T., Hartwig, M. K., Rawson, K. A., & Lipko, A. R. (2010) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2010.502239 · Cited by 72 · see citing works
The Path to Diagnostic Excellence Includes Feedback to Calibrate How Clinicians Think
Meyer, A. N. D., & Singh, H. (2019) · JAMA · doi:10.1001/jama.2019.0113 · Cited by 68 · see citing works
Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key
Zwaan, L., & Hautz, W. E. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-009078 · Cited by 63 · see citing works
Self-citationImproving Students’ Metacomprehension Accuracy
Griffin, T. D., Mielicki, M. K., & Wiley, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108235631.025 · Cited by 61 · see citing works
Effects of self-assessment feedback on self-assessment and task-selection accuracy
Raaijmakers, S. F., Baars, M., PAAS, F. G. W. C., van Merriënboer, J. J. G., & van Gog, T. (2019) · Metacognition and Learning · doi:10.1007/s11409-019-09189-5 · Cited by 59 · see citing works
Calibration and Self-Regulated Learning
Hacker, D. J., & Bol, L. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108235631.026 · Cited by 42 · see citing works
Omron, R., Kotwal, S., Garibaldi, B. T., & Newman‐Toker, D. E. (2018) · AEM Education and Training · doi:10.1002/aet2.10119 · Cited by 41 · see citing works
Can Providing Rubrics for Writing Tasks Improve Developing Writers' Calibration Accuracy?
Hawthorne, K. A., Bol, L., & Pribesh, S. L. (2017) · The Journal of Experimental Education · doi:10.1080/00220973.2017.1299081 · Cited by 36 · see citing works
Stenov, V., Wind, G., Skinner, T. C., Reventlow, S. D., & Hempler, N. F. (2017) · BMC Medical Education · doi:10.1186/s12909-017-1003-3 · Cited by 33 · see citing works
Filho, G. B. D. C., Moura, A. S., Brandão, P. R., Schmidt, H. G., & Mamede, S. (2019) · Perspectives on Medical Education · doi:10.1007/s40037-019-0522-5 · Cited by 30 · see citing works
Self-citationMeyer, A. N. D., Upadhyay, D. K., Collins, C. A., Fitzpatrick, M. H., Kobylinski, M., Bansal, A. B., Torretti, D., & Singh, H. (2020) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/j.jcjq.2020.08.014 · Cited by 25 · see citing works
Beyond right or wrong: More effective feedback for formative multiple-choice tests
Ryan, A., Judd, T. S., Swanson, D. B., Larsen, D. P., Elliott, S. L., Tzanetos, K., Tzanetos, K., & Kulasegaram, K. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00606-z · Cited by 24 · see citing works
Nederhand, M. L., Tabbers, H. K., & Rikers, R. M. J. P. (2019) · Applied Cognitive Psychology · doi:10.1002/acp.3548 · Cited by 20 · see citing works
The Effect of Performance Standards and Medical Experience on Diagnostic Calibration Accuracy
Nederhand, M. L., Tabbers, H. K., Splinter, T. A. W., & Rikers, R. M. J. P. (2018) · Health Professions Education · doi:10.1016/j.hpe.2017.12.008 · Cited by 19 · see citing works
Tweed, M., Stein, S., WILKINSON, T. J., Purdie, G., & Smith, J. K. (2017) · BMC Medical Education · doi:10.1186/s12909-017-0942-z · Cited by 13 · see citing works
Defining and tracking medical student self-monitoring using multiple-choice question item certainty
Tweed, M., Purdie, G., & WILKINSON, T. J. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02250-x · Cited by 10 · see citing works
Kuhn, J., van den Berg, P., Mamede, S., Zwaan, L., Diemers, A. D., Bindels, P., & van Gog, T. (2020) · Health Professions Education · doi:10.1016/j.hpe.2020.07.004 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
1 of the 39 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Lukama, L., Aldous, C. M., Michelo, C., & Kalinda, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281686 · Cited by 36 · see citing works
Meta-analysis of Interventions for Monitoring Accuracy in Problem Solving
Janssen, N., & Lazonder, A. W. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09936-4 · Cited by 35 · see citing works
David, L., Biwer, F., Baars, M., Wijnia, L., PAAS, F. G. W. C., & de Bruin, A. B. H. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09903-z · Cited by 26 · see citing works
Impact of performance and information feedback on medical interns' confidence–accuracy calibration
Staal, J., Katarya, K., Speelman, M., Brand, R., Alsma, J., Sloane, J., van den Broek, W. W., & Zwaan, L. (2023) · Advances in Health Sciences Education · doi:10.1007/s10459-023-10252-9 · Cited by 15 · see citing works
Self-citationToward Safer Diagnoses: A SEIPS-Based Narrative Review of Diagnostic Errors
Yen, C., Epling, J. W., Rockwell, M., & Vaughn-Cooke, M. (2026) · Diagnostics · doi:10.3390/diagnostics16020347 · Cited by 6 · see citing works
Roemer, A., Schlicker, N., Kernder, A. L., Albe, B., Hack, J., Hirsch, M., Mayr, A. J., Kuhn, S., & Knitza, J. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09079-w · Cited by 12 · see citing works
Butler, J. M., Taft, T., Taber, P. A., Rutter, E. D., Fix, M., Baker, A., Weir, C. R., Nevers, M. R., Classen, D. C., Cosby, K. S., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae112 · Cited by 6 · see citing works
Schaffler‐Schaden, D., Stöllinger, L., Avian, A., Terebessy, A., Scott, A. M., Streit, S., Piccoliori, G., Zelko, E., Huter, S., Mergenthal, K., et al. (2023) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08166-8 · Cited by 6 · see citing works
Confidence and certainty in medical diagnoses within acute healthcare: a scoping review
Aiyer, S., Higham, H., & Yeung, N. (2025) · BMJ Quality & Safety · doi:10.1136/bmjqs-2024-017997 · Cited by 5 · see citing works
Impact of diagnostic checklists on the interpretation of normal and abnormal electrocardiograms
Staal, J., Zegers, R., Caljouw-Vos, J., Mamede, S., & Zwaan, L. (2022) · Diagnosis · doi:10.1515/dx-2022-0092 · Cited by 5 · see citing works
Self-citationBergl, P. A., Shukla, N., Shah, J. A., Khan, M., Patel, J. J., & Nanchal, R. S. (2023) · Diagnosis · doi:10.1515/dx-2023-0026 · Cited by 5 · see citing works
The global progress for improving diagnosis: what we’ve learned, what comes next
Shimizu, T., Hautz, W. E., van Sassen, C., & Zwaan, L. (2025) · Diagnosis · doi:10.1515/dx-2025-0109 · Cited by 4 · see citing works
Self-citationInsights into diagnostic errors in endocrinology: a prospective, case-based, international study
Frey, J., Braun, L., Handgriff, L., Kendziora, B., Fischer, M. R., Reincke, M., Zwaan, L., & Schmidmaier, R. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04927-5 · Cited by 3 · see citing works
Self-citationSupporting Self-Assessment: A Systematic Review and Meta-Analysis of Feedback EffectsPreprint
Liebenow, L. W., Schmidt, F. T. C., Meyer, J., Panadero, E., & Fleckenstein, J. (2024) · doi:10.31219/osf.io/k94eq · Cited by 3 · see citing works
Opioid Treatment in Primary Care: Knowledge and Practical Use of Opioid Therapy
Biesiada, A. M., Ciałkowska-Rysz, A. D., & Mastalerz–Migas, A. (2024) · Healthcare · doi:10.3390/healthcare12020217 · Cited by 2 · see citing works
Development of a Prototype System for Milling Tool Condition Monitoring
Mushtaq, H., & Zahid, M. (2011) · Chalmers Publication Library (Chalmers University of Technology)
Grubenhoff, J. A., Dillon, M., Geanacopoulos, A. T., Melvin, J. E., Widmer, K. M., & Olson, A. P. J. (2025) · Journal of Continuing Education in the Health Professions · doi:10.1097/ceh.0000000000000632
Recker, F., Michel, S., Lehmann, M., Mathis, G., & Osterwalder, J. J. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09336-y
van der Wal, M. B. , Akkerman, O. W., Kerstjens, H. A. M., Faber, T., Boerma, E. C., & Terlouw, G. (2026) · doi:10.2196/preprints.106081
Methods to measure clinicians’ diagnostic calibration: an appraisal for real world application
Bradford, A., Shahid, U., Cholankeril, R. T., Astik, G. J., Olson, A. P. J., Singh, H., & Meyer, A. N. D. (2026) · Diagnosis · doi:10.1515/dx-2026-0039
Closure debt: the hidden educational deficit of fragmented clinical training
Jerjes, W., & Majeed, A. (2026) · Postgraduate Medical Journal · doi:10.1093/postmj/qgag139
- Authors
- Kuhn, J., van den Berg, P., Mamede, S., Zwaan, L., Bindels, P., & van Gog, T.
- Affiliations
- Erasmus MC (NL)
- Corresponding author
- Kuhn, Josepha
- Venue
- Advances in Health Sciences Education
- Published
- 2021-11-05
- Volume
- 27
- Issue
- 1
- Pages
- 189-200
- ISSN
- 1382-4996 (ISSN-L)
- 1573-1677
- PMC
- PMC8938348
- OpenAlex
- W3210610909
- Cited by
- 21
- References
- 39
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 5 · 2023: 5 · 2022: 1
- FWCI
- 3.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Advances in Health Sciences Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, cc-by)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- ZonMw (839130007)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- diagnostic calibration, diagnostic accuracy, physician confidence, medical residents, diagnostic errors, unnecessary testing, case complexity, general practice residency, underconfidence
- MeSH terms
- Diagnostic Errors, Feedback, Humans, Internship and Residency, Physicians, Self-Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
