CoChat
Sign Up
Cited by 32Clinical Reasoning and Diagnostic SkillsPatient Safety and Medication ErrorsHealth Sciences Research and Education

Effect on diagnostic accuracy of cognitive reasoning tools for the workplace setting: systematic review and meta-analysis

Open access
Staal, Justine, Hooftman, Jacky, Gunput, Sabrina T G, Mamede, Sílvia, Frens, Maarten A, Van den Broek, Walter W, Alsma, Jelmer, Zwaan, Laura·2022·BMJ Quality & Safety
DOI 10.1136/bmjqs-2022-014865PMC PMC9685706
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Preventable diagnostic errors are a large burden on healthcare. Cognitive reasoning tools, that is, tools that aim to improve clinical reasoning, are commonly suggested interventions. However, quantitative estimates of tool effectiveness have been aggregated over both workplace-oriented and educational-oriented tools, leaving the impact of workplace-oriented cognitive reasoning tools alone unclear. This systematic review and meta-analysis aims to estimate the effect of cognitive reasoning tools on improving diagnostic performance among medical professionals and students, and to identify factors associated with larger improvements. Methods: Controlled experimental studies that assessed whether cognitive reasoning tools improved the diagnostic accuracy of individual medical students or professionals in a workplace setting were included. Embase.com, Medline ALL via Ovid, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar were searched from inception to 15 October 2021, supplemented with handsearching. Meta-analysis was performed using a random-effects model. Results: The literature search resulted in 4546 articles of which 29 studies with data from 2732 participants were included for meta-analysis. The pooled estimate showed considerable heterogeneity (I 2=70%). This was reduced to I 2=38% by removing three studies that offered training with the tool before the intervention effect was measured. After removing these studies, the pooled estimate indicated that cognitive reasoning tools led to a small improvement in diagnostic accuracy (Hedges’ g=0.20, 95% CI 0.10 to 0.29, p<0.001). There were no significant subgroup differences. Conclusion: Cognitive reasoning tools resulted in small but clinically important improvements in diagnostic accuracy in medical students and professionals, although no factors could be distinguished that resulted in larger improvements. Cognitive reasoning tools could be routinely implemented to improve diagnosis in practice, but going forward, more large-scale studies and evaluations of these tools in practice are needed to determine how these tools can be effectively implemented. PROSPERO registration number: CRD42020186994.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjqs-2022-014865

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The assessment tool demonstrated substantial agreement between graders and internal reliability similar or superior to other measures used in the assessment of reasoning.

    “This assessment tool demonstrated substantial agreement between graders and internal reliability similar or superior to other measures used in the assessment of reasoning.39,40,41,42”

    From Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial (2024) · cites this paper as [39]

    The Methods section describes tools for assessing study quality, and the Results section reports high agreement between reviewers on risk-of-bias scores. Neither passage addresses grader agreement or internal reliability for an assessment tool of reasoning, or compares its reliability with other measures. Searched for grader agreement, inter-rater reliability, and internal reliability. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [1]Balogh EP, Miller BT, Ball JR. Improving diagnosis in health care 2015. 26803862
  2. [2]Prakash S, Sladek RM, Schuwirth L. Interventions to improve diagnostic decision making: a systematic review and meta-analysis on reflective strategies. Med Teach 2019; 41: 517– 24. 10.1080/0142159X.2018.1497786 30244625 · DOI
  3. [3]Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf 2012; 21: 535– 57. 10.1136/bmjqs-2011-000149 22543420 · DOI
  4. [4]Zwaan L, de Bruijne M, Wagner C, et al. Patient record review of the incidence, consequences, and causes of diagnostic adverse events. Arch Intern Med 2010; 170: 1015– 21. 10.1001/archinternmed.2010.146 20585065 · DOI
  5. [6]Gawande A. The checklist manifesto: how to get things right. J Nurs Regul 2011; 1: 64. 10.1016/S2155-8256(15)30310-0 · DOI
  6. [8]Gupta A, Graber ML. Web Exclusive. Annals for Hospitalists Inpatient Notes - Just What the Doctor Ordered-Checklists to Improve Diagnosis. Ann Intern Med 2019; 170: HO2. 10.7326/M19-0829 30986851 · DOI
  7. [10]Schiff GD, Hasan O, Kim S, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009; 169: 1881– 7. 10.1001/archinternmed.2009.333 19901140 · DOI · Paper page
  8. [11]Newman-Toker DE, Schaffer AC, Yu-Moe CW, et al. Serious misdiagnosis-related harms in malpractice claims: The "Big Three" - vascular events, infections, and cancers. Diagnosis 2019; 6: 227– 40. 10.1515/dx-2019-0019 31535832 · DOI · Paper page
  9. [12]Hartigan S, Brooks M, Hartley S, et al. Review of the basics of cognitive error in emergency medicine: still no easy answers. West J Emerg Med 2020; 21: 125. 10.5811/westjem.2020.7.47832 33207157 PMC7673867 · DOI
  10. [13]Mamede S, Schmidt HG. Reflection in medical diagnosis: a literature review. Health Professions Education 2017; 3: 15– 25. 10.1016/j.hpe.2017.01.003 · DOI
  11. [14]Astik GJ, Olson APJ. Learning from missed opportunities through reflective practice. Crit Care Clin 2022; 38: 103– 12. 10.1016/j.ccc.2021.09.003 34794624 · DOI
  12. [15]Mamede S, Hautz WE, Berendonk C, et al. Think twice: effects on diagnostic accuracy of returning to the case to reflect upon the initial diagnosis. Acad Med 2020; 95: 1223– 9. 10.1097/ACM.0000000000003153 31972673 · DOI
  13. [16]Kwan JL, Lo L, Ferguson J, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 169: m3216. 10.1136/bmj.m3216 PMC7495041 32943437 · DOI
  14. [17]Winters BD, Aswani MS, Pronovost PJ. Commentary: reducing diagnostic errors: another role for checklists? Acad Med 2011; 86: 279– 81. 10.1097/ACM.0b013e3182082692 21346432 · DOI
  15. [19]McDonald KM, Matesic B, Contopoulos-Ioannidis DG, et al. Patient safety strategies targeted at diagnostic errors: a systematic review. Ann Intern Med 2013; 158: 381– 9. 10.7326/0003-4819-158-5-201303051-00004 23460094 · DOI
  16. [20]Dave N, Bui S, Morgan C, et al. Interventions targeted at reducing diagnostic error: systematic review. BMJ Qual Saf 2022; 31: 297-307. 10.1136/bmjqs-2020-012704 34408064 · DOI
  17. [21]Croskerry P, Singhal G, Mamede S. Cognitive debiasing 2: impediments to and strategies for change. BMJ Qual Saf 2013; 22 Suppl 2: ii65– 72. 10.1136/bmjqs-2012-001713 PMC3786644 23996094 · DOI · Paper page
  18. [22]Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009; 62: e1– 34. 10.1016/j.jclinepi.2009.06.006 19631507 · DOI · Paper page
  19. [23]Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159– 74. 10.2307/2529310 843571 · DOI · Paper page
  20. [25](EPOC) CEPaOoC. Epoc resources for review authors. Oslo: Norwegian knowledge centre for the health services, 2013. Available: https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/Resources-for-authors2017/screening_data_extraction_and_management.pdf [Accessed 01 Jul 2022].
  21. [26]Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343: d5928. 10.1136/bmj.d5928 22008217 PMC3196245 · DOI · Paper page
  22. [29]O’Sullivan ED, Schofield SJ. A cognitive forcing tool to mitigate cognitive bias–a randomised control trial. BMC Med Educ 2019; 19: 1– 8. 10.1186/s12909-018-1444-3 30621679 PMC6325867 · DOI
  23. [30]Röver C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Med Res Methodol 2015; 15: 1– 7. 10.1186/s12874-015-0091-1 26573817 PMC4647507 · DOI · Paper page
  24. [31]Higgins JPT, Thomas J, Chandler J. Cochrane Handbook for systematic reviews of interventions. John Wiley & Sons, 2019.
  25. [32]Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629– 34. 10.1136/bmj.315.7109.629 9310563 PMC2127453 · DOI · Paper page
  26. [36]Letourneau KM, Horne D, Soni RN, et al. Advancing prenatal detection of congenital heart disease: a novel screening protocol improves early diagnosis of complex congenital heart disease. J Ultrasound Med 2018; 37: 1073– 9. 10.1002/jum.14453 29027708 · DOI
  27. [37]Sibbald M, de Bruin ABH, van Merrienboer JJG. Checklists improve experts' diagnostic decisions. Med Educ 2013; 47: 301– 8. 10.1111/medu.12080 23398016 · DOI · Paper page
  28. [38]Sibbald M, De Bruin ABH, van Merrienboer JJG. Finding and fixing mistakes: do checklists work for clinicians with different levels of experience? Adv Health Sci Educ Theory Pract 2014; 19: 43– 51. 10.1007/s10459-013-9459-3 23625338 · DOI
  29. [39]Sibbald M, de Bruin ABH, Yu E, et al. Why verifying diagnostic decisions with a checklist can help: insights from eye tracking. Adv Health Sci Educ Theory Pract 2015; 20: 1053– 60. 10.1007/s10459-015-9585-1 25672896 PMC4564443 · DOI · Paper page
  30. [40]Segal MM, Athreya B, Son MBF, et al. Evidence-Based decision support for pediatric rheumatology reduces diagnostic errors. Pediatr Rheumatol Online J 2016; 14: 67. 10.1186/s12969-016-0127-z 27964737 PMC5155385 · DOI
  31. [41]Billingsley S. Evaluating chest x-rays. Use mnemonics to develop a systematic approach. Adv Nurse Pract 2009; 17: 24– 5. 19999416
  32. [42]Dryver E, Johannsson G, Mokhtari A, et al. [Checklists and "crowdsourcing" for increased patient safety in the emergency department]. Lakartidningen 2014; 111: 493– 4. 24720026
  33. [43]Shimizu T, Matsumoto K, Tokuda Y. Effects of the use of differential diagnosis checklist and general de-biasing checklist on diagnostic performance in comparison to intuitive diagnosis. Med Teach 2013; 35: e1218– 29. 10.3109/0142159X.2012.742493 23228085 · DOI
  34. [44]Kämmer JE, Schauber SK, Hautz SC, et al. Differential diagnosis checklists reduce diagnostic error differentially: a randomised experiment. Med Educ 2021; 55: 1172– 82. 10.1111/medu.14596 34291481 PMC9290564 · DOI
  35. [45]Mamede S, Schmidt HG, Penaforte JC. Effects of reflective practice on the accuracy of medical diagnoses. Med Educ 2008; 42: 468– 75. 10.1111/j.1365-2923.2008.03030.x 18412886 · DOI · Paper page
  36. [46]Ilgen JS, Bowen JL, McIntyre LA, et al. Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. Acad Med 2013; 88: 1545– 51. 10.1097/ACM.0b013e3182a31c1e 23969355 · DOI
  37. [47]Ilgen JS, Bowen JL, Yarris LM, et al. Adjusting our lens: can developmental differences in diagnostic reasoning be harnessed to improve health professional and trainee assessment? Acad Emerg Med 2011; 18 Suppl 2: S79– 86. 10.1111/j.1553-2712.2011.01182.x 21999563 PMC3205465 · DOI
  38. [48]Cairns AW, Bond RR, Finlay DD, et al. A computer-human interaction model to improve the diagnostic accuracy and clinical decision-making during 12-lead electrocardiogram interpretation. J Biomed Inform 2016; 64: 93– 107. 10.1016/j.jbi.2016.09.016 27687552 · DOI
  39. [49]Martinez-Franco AI, Sanchez-Mendiola M, Mazon-Ramirez JJ, et al. Diagnostic accuracy in family medicine residents using a clinical decision support system (DXplain): a randomized-controlled trial. Diagnosis 2018; 5: 71– 6. 10.1515/dx-2017-0045 29730649 · DOI
  40. [50]Talebian MT, Zamani MM, Toliat A, et al. Evaluation of emergency medicine residents competencies in electrocardiogram interpretation. Acta Med Iran 2014; 52: 848– 54. 25415819
  41. [51]Thompson M, Johansen D, Stoner R, et al. Comparative effectiveness of a mnemonic-use approach vs. self-study to interpret a lateral chest X-ray. Adv Physiol Educ 2017; 41: 518– 21. 10.1152/advan.00034.2017 28978520 · DOI
  42. [52]Myung SJ, Kang SH, Phyo SR, et al. Effect of enhanced analytic Reasoning on diagnostic accuracy: a randomized controlled study. Med Teach 2013; 35: 248– 50. 10.3109/0142159X.2013.759643 23327617 · DOI
  43. [53]Griffith PB, Doherty C, Smeltzer SC, et al. Education initiatives in cognitive debiasing to improve diagnostic accuracy in student providers: a scoping review. J Am Assoc Nurse Pract 2020; 33: 862-871. 10.1097/JXX.0000000000000479 32773538 · DOI
  44. [54]Abimanyi-Ochom J, Bohingamu Mudiyanselage S, Catchpool M, et al. Strategies to reduce diagnostic errors: a systematic review. BMC Med Inform Decis Mak 2019; 19: 1– 14. 10.1186/s12911-019-0901-1 31470839 PMC6716834 · DOI
  45. [55]Sibbald M, de Bruin ABH, Cavalcanti RB, et al. Do you have to re-examine to reconsider your diagnosis? checklists and cardiac exam. BMJ Qual Saf 2013; 22: 333– 8. 10.1136/bmjqs-2012-001537 23386730 · DOI
  46. [56]Sibbald M, Sherbino J, Ilgen JS, et al. Debiasing versus knowledge retrieval checklists to reduce diagnostic error in ECG interpretation. Adv Health Sci Educ Theory Pract 2019; 24: 427– 40. 10.1007/s10459-019-09875-8 30694452 · DOI
  47. [57]Chew KS, Durning SJ, van Merriënboer JJ. Teaching metacognition in clinical decision-making using a novel mnemonic checklist: an exploratory study. Singapore Med J 2016; 57: 694– 700. 10.11622/smedj.2016015 26778635 PMC5165179 · DOI
  48. [59]Walayat S, Chaucer B, Kim M. Diagnostic Reboot: a proposal to improve diagnostic Reasoning: ncbi.nlm.nih.gov 2021. PMC7883530 33614306
  49. [60]Li P, Cheng ZY, Liu GL, yan Cheng Z, lin Liu G. Availability bias causes Misdiagnoses by physicians: direct evidence from a randomized controlled trial. Intern Med 2020; 59: 3141– 6. 10.2169/internalmedicine.4664-20 32788532 PMC7807127 · DOI · Paper page
  50. [61]Kok EM, Abed A, Robben SGF. Does the use of a checklist help medical students in the detection of abnormalities on a chest radiograph? J Digit Imaging 2017; 30: 726– 31. 10.1007/s10278-017-9979-0 28560508 PMC5681473 · DOI
  51. [62]Schmidt HG, Mamede S, van den Berge K, et al. Exposure to media information about a disease can cause doctors to misdiagnose similar-looking clinical cases. Acad Med 2014; 89: 285– 91. 10.1097/ACM.0000000000000107 24362387 · DOI
  52. [63]Schmidt HG, Van Gog T, Schuit SC, et al. Do patients' disruptive behaviours influence the accuracy of a doctor's diagnosis? a randomised experiment. BMJ Qual Saf 2017; 26: 19– 23. 10.1136/bmjqs-2015-004109 26951795 · DOI
  53. [64]Mamede S, Schmidt HG, Rikers RMJP, et al. Conscious thought beats deliberation without attention in diagnostic decision-making: at least when you are an expert. Psychol Res 2010; 74: 586– 92. 10.1007/s00426-010-0281-8 20354726 PMC2938445 · DOI
  54. [65]Mamede S, van Gog T, van den Berge K, et al. Effect of availability bias and reflective Reasoning on diagnostic accuracy among internal medicine residents. JAMA 2010; 304: 1198– 203. 10.1001/jama.2010.1276 20841533 · DOI · Paper page
  55. [66]Berbaum K, Franken EA, Caldwell RT, et al. Can a checklist reduce SOS errors in chest radiography? Acad Radiol 2006; 13: 296– 304. 10.1016/j.acra.2005.11.032 16488841 · DOI
  56. [67]Chartan C, Singh H, Krishnamurthy P, et al. Isolating red flags to enhance diagnosis (I-RED): an experimental vignette study. Int J Qual Health Care 2019; 31: G97– 102. 10.1093/intqhc/mzz082 31665303 · DOI
  57. [68]DiNardo D, Tilstra S, McNeil M, et al. Identification of facilitators and barriers to residents' use of a clinical Reasoning tool. Diagnosis 2018; 5: 21– 8. 10.1515/dx-2017-0037 29601296 · DOI
  58. [69]Graber ML, Tompkins D, Holland JJ. Resources medical students use to derive a differential diagnosis. Med Teach 2009; 31: 522– 7. 10.1080/01421590802167436 19811168 · DOI
  59. [70]Kilian M, Sherbino J, Hicks C, et al. Understanding diagnosis through action: evaluation of a point-of-care checklist for junior emergency medical residents. Diagnosis 2019; 6: 151– 6. 10.1515/dx-2018-0073 30990784 · DOI
  60. [71]Lambe KA, Hevey D, Kelly BD. Guided reflection interventions show no effect on diagnostic accuracy in medical students. Front Psychol 2018; 9: 2297. 10.3389/fpsyg.2018.02297 30532723 PMC6265413 · DOI
  61. [72]Nickerson J, Taub ES, Shah K. A checklist manifesto: can a checklist of common diagnoses improve accuracy in ECG interpretation? Am J Emerg Med 2020; 38: 18– 22. 10.1016/j.ajem.2019.03.048 30952602 · DOI
  62. [73]Costa Filho GB, Moura AS, Brandão PR, et al. Effects of deliberate reflection on diagnostic accuracy, confidence and diagnostic calibration in dermatology. Perspect Med Educ 2019; 8: 230– 6. 10.1007/s40037-019-0522-5 31290117 PMC6684533 · DOI

Cited by

Sorted by citations

Authors
Staal, J., Hooftman, J., Gunput, S. T. G., Mamede, S., Frens, M. A., Van den Broek, W. W., Alsma, J., & Zwaan, L.
Affiliations
  • Erasmus MC (NL)
  • Institute of Medical Education Research Rotterdam, Erasmus Medical Center, Rotterdam, The Netherlands
Corresponding author
Staal, Justine
ORCID
Venue
BMJ Quality & Safety
Publisher
BMJ
Published
2022-09-02
Volume
31
Issue
12
Pages
899-910
ISSN
  • 1475-3898 (ISSN-L)
  • 2044-5415
  • 2044-5423
DOI
10.1136/bmjqs-2022-014865
PMC
PMC9685706
OpenAlex
W4294316742
Cited by
32
References
5
Citations by year
2026: 11 · 2025: 3 · 2024: 13 · 2023: 5
FWCI
1.25
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Quality & Safety (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by-nc)
  • Pure Amsterdam UMC (acceptedVersion, cc-by)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Funders
  • Erasmus MC Fellowship (Not applicable)
  • Dutch National Scientific Organization (45116032)
Topics
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
diagnostic accuracy, diagnostic errors, clinical reasoning, work-based training, medical students, medical professionals, diagnostic performance, clinical decision support
MeSH terms
Cognition, Diagnostic Errors, Humans, Students, Medical, Workplace

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review