CoChat
Sign Up
Cited by 129Clinical Reasoning and Diagnostic SkillsElectronic Health Records SystemsPatient Safety and Medication Errors

Application of Electronic Trigger Tools to Identify Targets for Improving Diagnostic Safety

Open access
Murphy, Daniel R, Meyer, Ashley ND, Sittig, Dean F, Meeks, Derek W, Thomas, Eric J, Singh, Hardeep·2019·BMJ Quality & Safety
DOI 10.1136/bmjqs-2018-008086PMC PMC6365920
Open this paper in CoChat — free accountOpen PDF

Abstract

Progress in reducing diagnostic errors remains slow partly due to poorly defined methods to identify errors, high-risk situations, and adverse events. Electronic trigger (e-trigger) tools, which mine vast amounts of patient data to identify signals indicative of a likely error or adverse event, offer a promising method to efficiently identify errors. The increasing amounts of longitudinal electronic data and maturing data warehousing techniques and infrastructure offer an unprecedented opportunity to implement new types of e-trigger tools that use algorithms to identify risks and events related to the diagnostic process. We present a knowledge discovery framework, the Safer Dx Trigger Tools Framework, that enables health systems to develop and implement e-trigger tools to identify and measure diagnostic errors using comprehensive electronic health record (EHR) data. Safer Dx e-trigger tools detect potential diagnostic events, allowing health systems to monitor event rates, study contributory factors and identify targets for improving diagnostic safety. In addition to promoting organisational learning, some e-triggers can monitor data prospectively and help identify patients at high-risk for a future adverse event, enabling clinicians, patients or safety personnel to take preventive actions proactively. Successful application of electronic algorithms requires health systems to invest in clinical informaticists, information technology professionals, patient safety professionals and clinicians, all of who work closely together to overcome development and implementation challenges. We outline key future research, including advances in natural language processing and machine learning, needed to improve effectiveness of e-triggers. Integrating diagnostic safety e-triggers in institutional patient safety strategies can accelerate progress in reducing preventable harm from diagnostic errors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjqs-2018-008086

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Inpatient and acute care settings commonly use inpatient diagnostic error triggers and EHR-based interventions, whereas outpatient and primary care settings rely more on population risk-stratification tools and ambulatory e-triggers targeting missed follow-up or unexpected ED visits.

    “Inpatient and acute care settings commonly use inpatient diagnostic error triggers [375] and EHR-based interventions [376], whereas outpatient and primary care settings rely more on population risk-stratification tools [377] and ambulatory e-triggers targeting missed follow-up or unexpected ED visits [378].”

    From Toward Safer Diagnoses: A SEIPS-Based Narrative Review of Diagnostic Errors (2026) · cites this paper as [378]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [1]Kohn LT, Corrigan JM, Donaldson MS, Committee on quality of health Care in America, Institute of Medicine. To Err is human: building a safer health system. Washington, DC: The National Academies Press, 2000. 25077248
  2. [2]Nepple KG, Joudi FN, Hillis SL, et al Prevalence of delayed clinician response to elevated prostate-specific antigen values. Mayo Clin Proc 2008; 83: 439– 45. 10.4065/83.4.439 18380989 · DOI · Paper page
  3. [3]Singh H, Meyer AN, Thomas EJ The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf 2014; 23: 727– 31. 10.1136/bmjqs-2013-002627 PMC4145460 24742777 · DOI · Paper page
  4. [5]Liberman AL, Newman-Toker DE Symptom-Disease Pair Analysis of Diagnostic Error (SPADE): a conceptual framework and methodological approach for unearthing misdiagnosis-related harms using big data. BMJ Qual Saf 2018; 27: 557– 66. 10.1136/bmjqs-2017-007032 PMC6049698 29358313 · DOI
  5. [6]Dhaliwal G, Shojania KG The data of diagnostic error: big, large and small. BMJ Qual Saf 2018; 27: 499– 501. 10.1136/bmjqs-2018-007917 29507123 · DOI
  6. [7]Lorincz C, Drazen E, Sokol P Research in ambulatory patient safety 2000–2010: a 10-year review. Chicago IL: American Medical Association, 2011.
  7. [9]Howard IL, Bowen JM, Al Shaikh LAH, et al Development of a trigger tool to identify adverse events and harm in Emergency Medical Services. Emerg Med J 2017; 34: 391– 7. 10.1136/emermed-2016-205746 28153866 · DOI
  8. [10]De Almeida SM, Romualdo A, De Abreu Ferraresi A, et al Use of a trigger tool to detect adverse drug reactions in an emergency department. BMC Pharmacol Toxicol 2017; 18: 71 10.1186/s40360-017-0177-y 29141696 PMC5688728 · DOI
  9. [11]Unbeck M, Lindemalm S, Nydert P, et al Validation of triggers and development of a pediatric trigger tool to identify adverse events. BMC Health Serv Res 2014; 14: 655 10.1186/s12913-014-0655-5 25527905 PMC4300839 · DOI
  10. [12]Lipitz-Snyderman A, Classen D, Pfister D, et al Performance of a trigger tool for identifying adverse events in oncology. J Oncol Pract 2017; 13: e223– e230. 10.1200/JOP.2016.016634 28095173 PMC5482407 · DOI
  11. [13]Lindblad M, Schildmeijer K, Nilsson L, et al Development of a trigger tool to identify adverse events and no-harm incidents that affect patients admitted to home healthcare. BMJ Qual Saf 2018; 27: 502-511 10.1136/bmjqs-2017-006755 PMC6047163 28971884 · DOI
  12. [14]Sammer C, Miller S, Jones C, et al Developing and evaluating an automated all-cause harm trigger system. Jt Comm J Qual Patient Saf 2017; 43: 155– 65. 10.1016/j.jcjq.2017.01.004 28325203 · DOI
  13. [15]Menendez ME, Janssen SJ, Ring D Electronic health record-based triggers to detect adverse events after outpatient orthopaedic surgery. BMJ Qual Saf 2016; 25: 25– 30. 10.1136/bmjqs-2015-004332 26359272 · DOI
  14. [16]Triggers and Targeted Injury Detection Systems (TIDS) Expert Panel Meeting: conference summary report. Rockville, MD: Agency for Healthcare Research and Quality, 2009.
  15. [18]Clancy CM Common formats allow uniform collection and reporting of patient safety data by patient safety organizations. Am J Med Qual 2010; 25: 73– 5. 10.1177/1062860609352438 20042765 · DOI
  16. [19]Shenvi EC, El-Kareh R Clinical criteria to screen for inpatient diagnostic errors: a scoping review. Diagnosis 2015; 2: 3– 19. 10.1515/dx-2014-0047 26097801 PMC4474234 · DOI · Paper page
  17. [20]Classen DC, Resar R, Griffin F, et al 'Global trigger tool' shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff 2011; 30: 581– 9. 10.1377/hlthaff.2011.0190 21471476 · DOI · Paper page
  18. [21]Asgari H, Esfahani SS, Yaghoubi M, et al Investigating selected patient safety indicators using medical records data. J Educ Health Promot 2015; 4: 54 10.4103/2277-9531.162351 26430681 PMC4579765 · DOI
  19. [22]Kane-Gill SL, MacLasco AM, Saul MI, et al Use of text searching for trigger words in medical records to identify adverse drug reactions within an intensive care unit discharge summary. Appl Clin Inform 2016; 7: 660– 71. 10.4338/ACI-2016-03-RA-0031 27453336 PMC5052540 · DOI
  20. [23]Classen DC, Pestotnik SL, Evans RS Computerized surveillance of adverse drug events in hospital patients. JAMA 1991; 266: 2847– 51. 10.1001/jama.1991.03470200059035 1942452 · DOI
  21. [24]Doupi P, Svaar H, Bjørn B, et al Use of the global trigger tool in patient safety improvement efforts: nordic experiences. Cogn Technol Work 2015; 17: 45– 54. 10.1007/s10111-014-0302-2 · DOI
  22. [25]Murphy DR, Meyer AN, Vaghani V, et al Application of electronic algorithms to improve diagnostic evaluation for bladder cancer. Appl Clin Inform 2017; 8: 279– 90. 10.4338/ACI-2016-10-RA-0176 28326433 PMC5373770 · DOI
  23. [26]Murphy DR, Wu L, Thomas EJ, et al Electronic trigger-based intervention to reduce delays in diagnostic evaluation for cancer: a cluster randomized controlled trial. J Clin Oncol 2015; 33: 3560– 7. 10.1200/JCO.2015.61.1301 26304875 PMC4622097 · DOI
  24. [27]Murphy DR, Laxmisan A, Reis BA, et al Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf 2014; 23: 8– 16. 10.1136/bmjqs-2013-001874 23873756 · DOI
  25. [28]Murphy DR, Thomas EJ, Meyer AN, et al Development and validation of electronic health record-based triggers to detect delays in follow-up of abnormal lung imaging findings. Radiology 2015; 277: 81– 7. 10.1148/radiol.2015142530 25961634 PMC4613876 · DOI
  26. [29]Danforth KN, Smith AE, Loo RK, et al Electronic clinical surveillance to improve outpatient care: diverse applications within an integrated delivery system. EGEMS 2014; 2: 1056 10.13063/2327-9214.1056 25848588 PMC4371433 · DOI
  27. [30]Wandtke B, Gallagher S Reducing delay in diagnosis: multistage recommendation tracking. AJR Am J Roentgenol 2017; 209: 970– 5. 10.2214/AJR.17.18332 28742377 · DOI
  28. [31]Singh H, Thomas EJ, Khan MM, et al Identifying diagnostic errors in primary care using an electronic screening algorithm. Arch Intern Med 2007; 167: 302– 8. 10.1001/archinte.167.3.302 17296888 · DOI · Paper page
  29. [32]Resar RK, Rozich JD, Simmonds T, et al A trigger tool to identify adverse events in the intensive care unit. Jt Comm J Qual Patient Saf 2006; 32: 585– 90. 10.1016/S1553-7250(06)32076-4 17066996 · DOI
  30. [33]Bhise V, Sittig DF, Vaghani V, et al An electronic trigger based on care escalation to identify preventable adverse events in hospitalised patients. BMJ Qual Saf 2018; 27: 241-246 10.1136/bmjqs-2017-006975 PMC5867429 28935832 · DOI
  31. [34]Singh H, Sittig DF Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework. BMJ Qual Saf 2015; 24: 103– 10. 10.1136/bmjqs-2014-003675 PMC4316850 25589094 · DOI
  32. [35]Murphy DR, Meyer AN, Bhise V, et al Computerized triggers of big data to detect delays in follow-up of chest imaging results. Chest 2016; 150: 613– 20. 10.1016/j.chest.2016.05.001 27178786 · DOI
  33. [36]Sim JJ, Rutkowski MP, Selevan DC, et al Kaiser permanente creatinine safety program: a mechanism to ensure widespread detection and care for chronic kidney disease. Am J Med 2015; 128: 1204– 11. 10.1016/j.amjmed.2015.05.037 26087046 · DOI
  34. [37]Meyer AND, Murphy DR, Al-Mutairi A, et al Electronic detection of delayed test result follow-up in patients with hypothyroidism. J Gen Intern Med 2017; 32: 753– 9. 10.1007/s11606-017-3988-z 28138875 PMC5481223 · DOI
  35. [38]Hedriana HL, Wiesner S, Downs BG, et al Baseline assessment of a hospital-specific early warning trigger system for reducing maternal morbidity. Int J Gynaecol Obstet 2016; 132: 337– 41. 10.1016/j.ijgo.2015.07.036 26797195 · DOI
  36. [39]Shields LE, Wiesner S, Klein C, et al Use of maternal early warning trigger tool reduces maternal morbidity. Am J Obstet Gynecol 2016; 214: 527.e1– 527.e6. 10.1016/j.ajog.2016.01.154 26924745 · DOI
  37. [40]Singh H Improving diagnostic safety in primary care by unlocking digital data. Jt Comm J Qual Patient Saf 2017; 43: 29– 31. 10.1016/j.jcjq.2016.10.007 28334582 · DOI
  38. [41]Bhise V, Meyer AND, Singh H, et al Errors in diagnosis of spinal epidural abscesses in the era of electronic health records. Am J Med 2017; 130: 975– 81. 10.1016/j.amjmed.2017.03.009 28366427 · DOI
  39. [42]Hripcsak G, Bakken S, Stetson PD, et al Mining complex clinical data for patient safety research: a framework for event discovery. J Biomed Inform 2003; 36(1-2): 120– 30. 10.1016/j.jbi.2003.08.001 14552853 · DOI
  40. [44]Olson APJ, Graber ML, Singh H Tracking progress in improving diagnosis: a framework for defining undesirable diagnostic events. J Gen Intern Med 2018; 33: 1187– 91. 10.1007/s11606-018-4304-2 29380218 PMC6025685 · DOI
  41. [46]Singh H, Hirani K, Kadiyala H, et al Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record-based study. J Clin Oncol 2010; 28: 3307– 15. 10.1200/JCO.2009.25.6636 20530272 PMC2903328 · DOI
  42. [47]Zwaan L, Thijs A, Wagner C, et al Relating faults in diagnostic reasoning with diagnostic errors and patient harm. Acad Med 2012; 87: 149– 56. 10.1097/ACM.0b013e31823f71e6 22189886 · DOI
  43. [48]Zwaan L, Schiff GD, Singh H Advancing the research agenda for diagnostic error reduction. BMJ Qual Saf 2013; 22(Suppl 2): ii52– ii57. 10.1136/bmjqs-2012-001624 PMC3786655 23942182 · DOI
  44. [49]Berlin L Malpractice issues in radiology admitting mistakes. AJR Am J Roentgenol 1999; 172: 879– 84. 10.2214/ajr.172.4.10587115 10587115 · DOI
  45. [50]Gale BD, Bissett-Siegel DP, Davidson SJ, et al Failure to notify reportable test results: significance in medical malpractice. J Am Coll Radiol 2011; 8: 776– 9. 10.1016/j.jacr.2011.06.023 22051461 · DOI
  46. [52]Brender J, Ammenwerth E, Nykänen P, et al Factors influencing success and failure of health informatics systems--a pilot Delphi study. Methods Inf Med 2006; 45: 125– 36. 16482383
  47. [53]Danforth KN, Early MI, Ngan S, et al Automated identification of patients with pulmonary nodules in an integrated health system using administrative health plan data, radiology reports, and natural language processing. J Thorac Oncol 2012; 7: 1257– 62. 10.1097/JTO.0b013e31825bd9f5 22627647 PMC3443078 · DOI
  48. [54]Chapman WW, Nadkarni PM, Hirschman L, et al Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions. J Am Med Inform Assoc 2011; 18: 540– 3. 10.1136/amiajnl-2011-000465 21846785 PMC3168329 · DOI · Paper page
  49. [55]Deo RC Machine learning in medicine. Circulation 2015; 132: 1920– 30. 10.1161/CIRCULATIONAHA.115.001593 26572668 PMC5831252 · DOI · Paper page
  50. [56]Bramer M Principles of data mining. London: Springer London, 2013.
  51. [57]Medford-Davis L, Park E, Shlamovitz G, et al Diagnostic errors related to acute abdominal pain in the emergency department. Emerg Med J 2016; 33: 253– 9. 10.1136/emermed-2015-204754 26531859 · DOI
  52. [58]Rosen AK, Mull HJ, Kaafarani H, et al Applying trigger tools to detect adverse events associated with outpatient surgery. J Patient Saf 2011; 7: 45– 59. 10.1097/PTS.0b013e31820d164b 21921867 · DOI
  53. [59]Resar RK, Rozich JD, Classen D Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003; 12: 39– 45. 10.1136/qhc.12.suppl_2.ii39 PMC1765771 14645894 · DOI
  54. [60]Murphy DR, Meyer AND, Vaghani V, et al Electronic triggers to identify delays in follow-up of mammography: harnessing the power of big data in health care. J Am Coll Radiol 2018; 15: 287– 95. 10.1016/j.jacr.2017.10.001 29102539 · DOI · Paper page
  55. [61]Murphy DR, Meyer AND, Vaghani V, et al Development and validation of trigger algorithms to identify delays in diagnostic evaluation of gastroenterological cancer. Clin Gastroenterol Hepatol 2018; 16: 90– 8. 10.1016/j.cgh.2017.08.007 28804030 · DOI
  56. [62]Meyer AN, Murphy DR, Singh H Communicating findings of delayed diagnostic evaluation to primary care providers. J Am Board Fam Med 2016; 29: 469– 73. 10.3122/jabfm.2016.04.150363 27390378 · DOI
  57. [64]Soundrarajan BR, Ginter T, DuVall SL An interface for rapid natural language processing development in UIMAProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Systems Demonstrations. Association for Computational Linguistics 2011: 139– 44.
  58. [65]Divita G, Carter ME, Tran LT, et al v3NLP Framework: tools to build applications for extracting concepts from clinical text. EGEMS 2016; 4: 1228 10.13063/2327-9214.1228 27683667 PMC5019303 · DOI
  59. [66]Wu J, Roy J, Stewart WF Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches. Med Care 2010; 48(6 Suppl): S106– 13. 10.1097/MLR.0b013e3181de9e17 20473190 · DOI
  60. [67]Russo E, Sittig DF, Murphy DR, et al Challenges in patient safety improvement research in the era of electronic health records. Healthc 2016; 4: 285– 90. 10.1016/j.hjdsi.2016.06.005 27473472 · DOI
  61. [68]Schwartz A, Weiner SJ, Weaver F, et al Uncharted territory: measuring costs of diagnostic errors outside the medical record. BMJ Qual Saf 2012; 21: 918– 24. 10.1136/bmjqs-2012-000832 22773889 · DOI
  62. [69]Weiner SJ, Schwartz A Directly observed care: can unannounced standardized patients address a gap in performance measurement? J Gen Intern Med 2014; 29: 1183– 7. 10.1007/s11606-014-2860-7 24756945 PMC4099461 · DOI

Cited by

Sorted by citations

Authors
Murphy, D. R., Meyer, A. N., Sittig, D. F., Meeks, D. W., Thomas, E. J., & Singh, H.
Affiliations
  • Michael E. DeBakey VA Medical Center (US)
  • Memorial Hermann (US)
Corresponding author
Murphy, Daniel R
ORCID
Venue
BMJ Quality & Safety
Publisher
BMJ
Published
2018-10-05
Volume
28
Issue
2
Pages
151-159
ISSN
  • 1475-3898 (ISSN-L)
  • 2044-5415
  • 2044-5423
DOI
10.1136/bmjqs-2018-008086
PMC
PMC6365920
OpenAlex
W2895457576
Cited by
129
References
5
Citations by year
2026: 10 · 2025: 16 · 2024: 22 · 2023: 17 · 2022: 24 · 2021: 13 · 2020: 15 · 2019: 11 · 2017: 1
FWCI
11.17
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • BMJ Quality & Safety (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • HighWire Press Open Archive (submittedVersion)
Funders
Health Services Research and Development (CRE-12-033)
Topics
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
Keywords
diagnostic errors, diagnostic safety, adverse events, electronic health records, knowledge discovery, high-risk patient identification, preventable harm, patient safety monitoring, natural language processing, machine learning, organizational learning
MeSH terms
Algorithms, Diagnostic Errors, Humans, Natural Language Processing, Safety Management, Risk Assessment, Electronic Health Records, Patient Safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review