CoChat
Sign Up
Cited by 92Electronic Health Records SystemsArtificial Intelligence in Healthcare and EducationTechnology and Human Factors in Education and Health

Governance for clinical decision support: case studies and recommended practices from leading institutions

Open access
Wright, A., Sittig, D. F., Ash, J. S., Bates, D. W., Feblowitz, J., Fraser, G., Maviglia, S. M., McMullen, C. +4 more·2011·Journal of the American Medical Informatics Association
DOI 10.1136/jamia.2009.002030PMC PMC3116253
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVE: Clinical decision support (CDS) is a powerful tool for improving healthcare quality and ensuring patient safety; however, effective implementation of CDS requires effective clinical and technical governance structures. The authors sought to determine the range and variety of these governance structures and identify a set of recommended practices through observational study. DESIGN: Three site visits were conducted at institutions across the USA to learn about CDS capabilities and processes from clinical, technical, and organizational perspectives. Based on the results of these visits, written questionnaires were sent to the three institutions visited and two additional sites. Together, these five organizations encompass a variety of academic and community hospitals as well as small and large ambulatory practices. These organizations use both commercially available and internally developed clinical information systems. MEASUREMENTS: Characteristics of clinical information systems and CDS systems used at each site as well as governance structures and content management approaches were identified through extensive field interviews and follow-up surveys. RESULTS: Six recommended practices were identified in the area of governance, and four were identified in the area of content management. Key similarities and differences between the organizations studied were also highlighted. CONCLUSION: Each of the five sites studied contributed to the recommended practices presented in this paper for CDS governance. Since these strategies appear to be useful at a diverse range of institutions, they should be considered by any future implementers of decision support.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper reports an observational, qualitative study of clinical decision support (CDS) governance at five diverse US healthcare organizations and distills recommended practices for governance and content management. It identifies six governance practices and four content-management practices, while cautioning that the findings come from a small, purposively selected, nonrepresentative sample.

Key findings

  • Five organizations were studied: Partners HealthCare System, Mid-Valley Independent Physicians Association, Vanderbilt Medical Group, Veterans Health Administration, and University of Texas Physicians’ Practice Plan.
  • All sites had at least one example of each of the six Osheroff CDS types, but depth varied widely: condition-specific order sets ranged from less than 50 to more than 800, and unique clinical alerts ranged from less than 50 to more than 7000.
  • The authors identified six recommended practices for CDS governance and four recommended practices for content management.
  • Governance differed substantially between sites using vendor-based clinical information systems and those with internally developed systems; vendor-based sites had fewer, more centralized committees and limited customization, while internally developed sites had larger specialized staff, more committees, and more local customization.

Methods

  • Observational qualitative study using the Rapid Assessment Process, with three site visits and five comprehensive surveys at purposively selected exemplary institutions with a history of successful CDS research and utilization.
  • Site visits involved four to seven researchers (physicians, nurses, pharmacists, informaticians), interviews with CDS developers and information system leaders, observations of clinical end users, tape-recorded interviews, and detailed field notes.
  • Analysis used grounded theory, NVivo 8 qualitative data-analysis software, card-sorting to distill themes, and follow-up written questionnaires plus a CDS content inventory based on Osheroff et al.’s six CDS categories.

Limitations

  • The paper states that data came from five exemplary sites that were “purposefully nonrepresentative of typical practice,” limiting generalizability.
  • The paper states that only US institutions were studied, further limiting generalizability, and that recommended practices were based on consensus and expert opinion, so some effective practices may have been missed.
  • The paper states that it could not characterize the cost of CDS governance or study financial implications of different governance approaches; it also notes no interventional or wide-scale observational evidence exists for the effect of CDS governance practices.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1136/jamia.2009.002030

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 15 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    All of the sites provided some capability for user feedback, ranging from feedback buttons embedded in the EHR application to email, helpdesk query tracking, and clinical representation on committees.

    “All of the sites provided some capability for user feedback, ranging from feedback buttons embedded in the EHR application to email, helpdesk query tracking, and clinical representation on committees.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The sites varied dramatically in the depth of decision support they had.

    “Despite having similar breadth of CDS, the sites varied dramatically in the depth of decision support they had.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Vendor-based clinical information systems have regular reports from vendors.

    “Regular reports from vendors”
    Quote in the paperNot a substantial claim
  • Findingown data

    Several of the sites reported having special-purpose software tools for managing content throughout the knowledge management life cycle.

    “Several of the sites reported having special-purpose software tools for managing content throughout this knowledge management life cycle.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Internally developed clinical information systems have more, specialized committees.

    “More, specialized committees”
    Quote in the paperSubstantial claim
  • May combine two claims. 26
  • Classified as probably not a substantial claim. 43

  • May combine two claims.

    Findingown data

    Some organizations carried out an annual review, while other organizations reviewed content based on user feedback or as new evidence became available.

    “Some organizations carried out an annual review, while other organizations reviewed content based on user feedback or as new evidence became available.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Design changes that affect clinical content at MVIPA are reviewed by the medical director of information systems.

    “Design changes that affect clinical content are reviewed by the medical director of information systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Internally developed clinical information systems permit customization, although standardization may be preferred.

    “Permitted, although standardization may be preferred”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The VA emphasizes standard practice and common data elements within and across their system of care, and is in the process of standardizing such practices when there is evidence or experience to support such efforts.

    “The VA also emphasizes standard practice and common data elements within and across their system of care, and is in the process of standardizing such practices when there is evidence or experience to support such efforts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Common metrics included firing rate, acceptance rate, analysis of override reasons, and measurement of the decision support's effects on intermediate processes, and ultimately its effects on patient safety and quality of care.

    “Common metrics included firing rate, acceptance rate, analysis of override reasons, and measurement of the decision support's effects on intermediate processes, and ultimately its effects on patient safety and quality of care.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Evidence-based guidelines from nationally recognized authorities are reviewed regularly for inclusion at the VA.

    “Evidence-based guidelines from nationally recognized authorities are also reviewed regularly for inclusion.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Vendor-based clinical information systems have all or almost all content come from the content vendor, and system operations may be run locally.

    “All or almost all content comes from content vendor; system operations”
    • Also stated in: § CONCLUSIONS · hedged
      “may be run locally”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The five sites studied were quite diverse.

    “The five sites we studied were quite diverse; however, they had also learned many overlapping and complementary lessons about governance for CDS.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Content at MVIPA is updated periodically through the NextGen release cycle and is first tested by MVIPA staff.

    “Content is updated periodically through the NextGen release cycle and is first tested by MVIPA staff.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All sites reported some form of ongoing content review, but the frequency and regularity of these reviews varied.

    “All sites reported some form of ongoing content review, but the frequency and regularity of these reviews varied.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Vanderbilt conducts regular reviews of content and updates content as clinical evidence surfaces that would necessitate a change.

    “Vanderbilt conducts regular reviews of content and then updates the content as clinical evidence surfaces that would necessitate a change.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The director also monitors regulatory requirements and prioritizes updates according to their effective dates and their impact.

    “The director also monitors regulatory requirements and prioritizes updates according to their effective dates and their impact.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The Practice Plan conducts surveys of clinical users and creates monthly reports that provide all clinics with a breakdown of utilization of all aspects of the EHR.

    “The Practice Plan conducts surveys of clinical users and creates monthly reports that provide all clinics with a breakdown of the utilization of all aspects of the EHR (electronic health record) (eg, percentage of prescriptions written electronically).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In some cases, content was purchased from an outside vendor; in others it was developed by in-house staff based on institutional needs.

    “In some cases, content was purchased from an outside vendor, and in others it was developed by inhouse staff based on institutional needs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    CDS capabilities at the VA are constantly updated and revised to reflect internal research activities, review of performance measures, input from regulatory groups, networking with academic affiliates, changing emphasis of national program and patient priorities, and involvement in national professional societies and working groups.

    “CDS capabilities at the VA are constantly being updated and revised to reflect internal research activities, review of performance measures, input from regulatory groups, networking with academic affiliates, changing emphasis of national program and patient priorities, and involvement in national professional societies and working groups.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Vanderbilt prioritizes changes based on quality, safety, clinical volumes, and cost/benefit.

    “Vanderbilt prioritizes changes based on quality, safety, clinical volumes, and cost/benefit.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Sites mentioned and underscored the importance of field-testing content and listening and responding to user feedback.

    “Sites also mentioned and underscored the importance of fieldtesting content and listening and responding to user feedback.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    At MVIPA, the primary method of evaluating the effectiveness of CDS is ongoing field testing.

    “Finally, at MVIPA, the primary method of evaluating the effectiveness of CDS is ongoing field testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The VA national pharmacy program is pursuing integration of a commercially available database for drug-drug/allergy interactions, which would be maintained and periodically updated.

    “Additionally, the VA national pharmacy program is pursuing integration of a commercially available database for drugedrug/allergy interactions, which would be maintained and periodically updated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    University of Texas, Houston Physicians' Practice Plan relies heavily on Allscripts' content, and updates are tied to Allscripts' release schedule.

    “In contrast to the other academic institutions, the University of Texas, Houston Physicians' Practice Plan relies heavily on Allscripts' content, and updates are tied to the Allscripts' release schedule.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Evaluation of the effectiveness of CDS at the VA is indirect and based on measurement of outcomes it was designed to impact.

    “Evaluation of the effectiveness of the CDS is indirect and based on the measurement of outcomes it was designed to impact.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Outside the healthcare industry, a variety of models have been proposed to describe knowledge management and IT governance practices.

    “Outside of the healthcare industry, a variety of models have been proposed to describe knowledge management and IT governance practices.12e14”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites 12e14

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Governance can vary widely depending on the type and size of the organization.

    “Governance can vary widely depending on the type and size of the organization.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Researchers note the substantial investment required to implement IT and knowledge management systems and the uncertainty associated with these investments.

    “Researchers also note the substantial investment required to implement IT and knowledge management systems and the uncertainty associated with these investments, further underscoring the need for effective governance.12 15”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites 12 15

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    As a result of limited ability to develop and monitor CDS, most small sites tend to purchase their decision support from commercial content suppliers, rather than developing the content themselves.

    “The primary differences are that, in general, most small sites have a limited ability to develop and monitor CDS and, as a result, tend to purchase their decision support from commercial content suppliers, rather than developing the content themselves, and tend to rely on general-purpose software tools and built-in EHR functions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The volume of clinical content and rule logic contained in these systems makes tools to monitor CDS inventory necessary for ensuring the system is up to date and functioning properly, and for guiding future development and implementation.

    “The volume of clinical content and rule logic contained in these systems makes these tools necessary for ensuring the system is up to date and functioning properly, and for guiding future development and implementation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Two new sites were used to validate and extend the initial findings.

    “We then developed questions based on these themes and sent written questionnaires to a purposive sample of five sites: three from our initial site visits and two new sites to validate and extend our initial findings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Overall, small sites must confront the same issues as large sites but with the added challenge of limited financial resources and personnel.

    “Overall, small sites must confront the same issues but with the added challenge of limited financial resources and personnel.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Many of the recommended practices may be nearly universal.

    “Although the needs of every implementer of decision support are different, we believe that many of these recommended practices may be nearly universal and that all implementers of decision support should consider employing them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    In a small practice environment, some tasks, such as monitoring inventory and user feedback, may be simpler to accomplish.

    “In a small practice environment, some tasks, such as monitoring inventory and user feedback, may be simpler to accomplish.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The governance issues discussed in the paper are not limited to commercial EHR vendors and healthcare institutions.

    “The governance issues discussed in this paper are not limited to commercial EHR vendors and healthcare institutions, and in the future, we hope to investigate the role of these additional key plays in CDS development and implementation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The availability and use of special-purpose content management tools, when well designed, can improve efficiency, allow for asynchronous communication, and ensure that content is periodically reviewed and kept up to date.

    “The availability and use of these tools, when well designed, can improve efficiency, allow for asynchronous communication, and ensure that content is periodically reviewed and kept up to date.30”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [30]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The development and implementation of effective CDS depend on sound content-management strategies.
    • Rests on an assumption: The large knowledge bases required for robust CDS necessitate frequent and thorough management to keep clinical content up to date.

  • May combine two claims.

    Study design and conductreasoning

    The five exemplary sites presented are purposefully nonrepresentative of typical practice, which limits the generalizability of the data.

    “First, we presented data on a total of five exemplary sites which are purposefully nonrepresentative of typical practicedwe believe that this approach maximized what we could learn, but it also limits the generalizability of our data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Large organizations can potentially develop and implement CDS internally, while smaller organizations may need to rely on a partnership with a commercial vendor.

    “For example, large organizations, such as academic medical centers and governmentrun hospitals, can potentially develop and implement CDS internally, while smaller organizations, such as group practices and community hospitals, may need to rely on a partnership with a commercial vendor.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Measurement allows content developers to assess the impact of their content and to identify opportunities for improvement.

    “Measurement allows content developers to assess the impact of their content and to identify opportunities for improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Monitoring of CDS interventions is necessary for properly assessing the impact of CDS on clinician behavior and patient care.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    Knowledge bases that are developed must reflect both universal best practices and local practices and needs.

    “The knowledge bases that are developed must reflect both universal best practices and local practices and needs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Small sites that are earlier in the process of developing CDS also face similar governance challenges.

    “However, small sites that are earlier in the process of developing CDS also face similar governance challenges.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The sites studied included only US institutions, which further limits the generalizability of the findings.

    “Finally, the sites studied included only US institutions, which further limits the generalizability of our findingsdfuture study of an internationally representative collection of sites is also likely to be fruitful.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Employing better governance practices throughout the design and implementation phases, such as increased end-user involvement, can potentially stave off these types of problems.

    “Employing better governance practices throughout the design and implementation phases, such as increased end-user involvement, can potentially stave off these types of problems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    Delivering excellent CDS requires addressing many governance issues.

    “Delivering excellent CDS requires addressing many governance issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Special-purpose content management tools can enhance the auditability of content, as decisions can be traced.

    “Such tools can also enhance the auditability of content, as decisions can be traced.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The development and implementation of effective CDS depend on sound content-management strategies.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study used a purposive sampling strategy selecting five exemplary institutions with a history of successful CDS research and utilization.

    “Our sampling strategy for these visits was purposive; we selected five exemplary institutions with a history of successful CDS research and utilization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    These models emphasize the need for continuous knowledge management, the value of studying real-world governance best practices, and the importance of well-planned implementation.

    “These models emphasize the need for continuous knowledge management, the value of studying real-world governance best practices and the importance of well-planned implementation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites 12e14

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Governance processes take many forms.

    “These governance processes take many forms.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The recommended practices were identified based on consensus and expert opinion, and as a result, some effective practices may have been missed.

    “We identified recommended practices based on consensus and expert opinion, and as a result, some effective practices may have been missed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Knowledge management is an iterative process that involves both the creation of new content and the continuous review and revision of existing content.

    “Knowledge management is an iterative process that involves both the creation of new content and the continuous review and revision of existing content.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    In general, most small sites have a limited ability to develop and monitor CDS.

    “The primary differences are that, in general, most small sites have a limited ability to develop and monitor CDS and, as a result, tend to purchase their decision support from commercial content suppliers, rather than developing the content themselves, and tend to rely on general-purpose software tools and built-in EHR functions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The authors were not able to characterize the cost of CDS governance, or to study the financial implications of different governance approaches.

    “Finally, we were not able to characterize the cost of CDS governance, or to study the financial implications of different governance approaches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The present relative paucity of sites with advanced or even formal governance structures may make it premature to perform a study to more rigorously quantify the methods for and value of governance.

    “Because of the present relative paucity of sites with advanced (or even formal) governance structures, it may yet be premature to perform such a study, but once there exists a sufficient set of institutions with reasonably robust governance practices, we believe that such a study would add great value to the literature.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Many rules interact, and as more rules are added, interactions become more frequent.

    “A related issue is that many rules interact, and as more rules are added, interactions become more frequent, one example being rules relating to reducing cholesterol levels in patients with diabetes and coronary disease.28”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [28]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Formal governance models for decision support are relatively new, no interventional or wide-scale observational evidence exists for the effect of various CDS governance practices, and there is no particular regulation relevant to such practices.

    “Because formal governance models for decision support are relatively new, no interventional (or even wide-scale observational) evidence exists for the effect of various CDS governance practices, and there is no particular regulation relevant to such practices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    A broad understanding of the institutional pressures organizations face and real-world assessment of these institutional pressures is valuable and necessary to establish effective governance practices and realize the potential of IT and knowledge management systems.

    “In order to establish effective governance practices and realize the potential of IT and knowledge management systems, a broad understanding of the institutional pressures organizations face and real-world assessment of these institutional pressures is both valuable and necessary.16”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Sites may need to tailor or selectively adopt these practices to develop solutions that work for them.

    “As evidenced in the case studies, however, decision support is an intensely local phenomenon, 26 and as such, sites may need to tailor or selectively adopt these practices to develop solutions that work for them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The research team identified themes in the data using a grounded theory approach.

    “The research team, lead by trained ethnographers, identified themes in the data using a grounded theory approach.”
    Quote in the paperNot a substantial claim
54 of 54

  • The paper relies on this without support.

    The five sites studied are sufficiently diverse and exemplary to yield recommended practices that are useful to other institutions.

    “Our sampling strategy for these visits was purposive; we selected five exemplary institutions with a history of successful CDS research and utilization.”

  • The paper relies on this without support.

    Effective CDS often requires extremely large knowledge bases of clinical facts.

    “Effective CDS often requires extremely large knowledge bases of clinical facts (eg, drug-interaction tables).”

  • The paper relies on this without support.

    The development and implementation of effective CDS depend on sound content-management strategies.

    “The development and implementation of effective CDS also depend on sound content-management strategies.”

  • The paper relies on this without support.

    The large knowledge bases required for robust CDS necessitate frequent and thorough management to keep clinical content up to date.

    “The large knowledge bases required for robust CDS necessitate frequent and thorough management in order to keep clinical content up to date.”

  • The paper relies on this without support.

    A seemingly minor change can have a major impact on how many warnings are being delivered and on overall system performance.

    “Often, a seemingly minor change can have a major impact on how many warnings are being delivered, and even on overall system performance.”

  • The paper relies on this without support.

    Listening and responding to feedback is an important tool for product improvement.

    “Listening and responding to feedback is an important tool for product improvement.”

  • The paper relies on this without support.

    User feedback fosters a sense of ownership of the product by end users when they feel that their input is valued.

    “It also has the added advantage of fostering a sense of ownership of the product by end users when they feel that their input is valued.”

  • The paper relies on this without support.

    Monitoring of CDS interventions is necessary for properly assessing the impact of CDS on clinician behavior and patient care.

    “In addition to ongoing monitoring of CDS inventory (as outlined in Recommendation 3), monitoring of CDS interventions is necessary for properly assessing the impact of CDS on clinician behavior and patient care.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Effective governance of CDS has been demonstrated in other institutions, resulting in the transition of alert and reminder content to other more appropriate areas in the EHR, downgrading the severity of alerts, and assessing the effect of CDS on clinician behavior and patient care.

    “Effective governance of CDS has been demonstrated in other institutions, resulting in the transition of alert and reminder content to other more appropriate areas in the EHR, downgrading the severity of alerts (often leading to them no longer firing), 38 39 and assessing the effect of CDS on clinician behavior and patient care. 40 41”

    From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [40]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Our findings assessing interruptive alert helpfulness and recommendations to change interruptive alert presentation, timing, and targeting were also like those found in previous studies.

    “Our findings assessing interruptive alert helpfulness and recommendations to change interruptive alert presentation, timing, and targeting were also like those found in previous studies. 35 36 40”

    From Qualitative Study of Inpatient Hospital Medicine Interruptive Alerts and Clinical Decision Support (2026) · cites this paper as [40]

    The sections on ongoing monitoring and user feedback (pages not specified in our copy) describe assessing responses to alerts and using feedback to improve decision support, including interface changes. They do not specifically recommend changing the presentation, timing, and targeting of interruptive alerts or assess their helpfulness as such. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • A key challenge with clinical decision support is knowledge management as guidelines and best practices change over time, as well as governance.

    “A key challenge with clinical decision support is knowledge management as guidelines and best practices change over time,40–42 as well as governance.43–45”

    From Clinician collaboration to improve clinical decision support: the Clickbusters initiative (2022) · cites this paper as [43]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Same article is cited with different publication years

    Minor

    References 6 and 30 give the same authors, title, journal, volume, and pages, but cite different publication years: 2009 and 2010.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. 2009, International journal of medical informatics, 79, 44
  2. Morrissey, 2004, Modern healthcare, 34, 16
  3. Middleton, 2009, Studies in health technology and informatics, 150, 26
  4. Ash, 2010, Studies in health technology and informatics, 160, 806

Cited by

Sorted by citations

Related

Authors
Wright, A., Sittig, D. F., Ash, J. S., Bates, D. W., Feblowitz, J., Fraser, G., Maviglia, S. M., McMullen, C., Nichol, W. P., Pang, J. E., Starmer, J., & Middleton, B.
Affiliations
  • Brigham and Women's Hospital (US)
  • Memorial Hermann (US)
  • Oregon Health & Science University (US)
  • Kaiser Permanente Center for Health Research (US)
  • University of Washington (US)
  • Vanderbilt University Medical Center (US)
Corresponding author
Wright, A.
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2011-03-01
Volume
18
Issue
2
Pages
187-194
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1136/jamia.2009.002030
PMC
PMC3116253
OpenAlex
W2096222958
Semantic Scholar
e4a7710aaaf5c6257bc3edc6e57e12cfc045188f
Cited by
92
References
36
Citations by year
2026: 4 · 2025: 5 · 2024: 3 · 2023: 3 · 2022: 8 · 2021: 4 · 2020: 4 · 2019: 9 · 2018: 6 · 2017: 5 · 2016: 8 · 2015: 7 · 2014: 5 · 2013: 13 · 2012: 4
FWCI
12.80
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Kaiser Permanente
  • Agency for Healthcare Research and Quality (HHSA290200810010)
  • U.S. National Library of Medicine (R56LM006942, 563 R56-LM006942)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
Keywords
clinical decision support, clinical governance, technical governance, content management, clinical information systems, patient safety, quality of care, hospitals
MeSH terms
Health Plan Implementation, Humans, Quality Assurance, Health Care, United States, Organizational Case Studies, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review