CoChat
Sign Up
Cited by 237Health Policy Implementation ScienceHealthcare Quality and ManagementEvaluation and Performance Assessment

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Understanding what works—and why—in quality improvement: the need for theory-driven evaluation

Walshe, Kieran·2007·International Journal for Quality in Health Care
DOI 10.1093/intqhc/mzm004
Open this paper in CoChat — free account

Abstract

Clinicians who are asked to participate in quality improvement programmes in healthcare organizations are often heard to ask for the evidence that they ‘work’. By that, they often mean they want randomized controlled trials, which show that accreditation, or credentialing, or criterion-based audit, or adverse event monitoring, or continuous quality improvement programmes, or whatever approach is being used cause meaningful and worthwhile improvements in the quality of care [1]. When they learn that there are relatively few experimental studies of quality improvement interventions [2], and those which do exist often show weak or moderate effects at best, this state of affairs is sometimes used to argue that it is not worthwhile investing time and effort in quality improvement. After all, the argument goes, we should not embark on using a new clinical intervention such as a drug or a surgical procedure without solid experimental evidence of its effectiveness, so why should we have a lower threshold for the adoption of organizational interventions like quality improvement programmes? Surely, they too should be proven to ‘work’ before they are adopted or implemented widely? This point of view needs to be challenged. It is founded on a fundamental misunderstanding of the place of experimental methods in investigating and understanding complex social interventions, which is commonplace particularly among clinicians and biomedical researchers and which can seriously hamper those both researching and implementing quality improvement in healthcare. In all research, our starting point should be to match our research methods to the questions or issues being investigated (and not the other way around). Before selecting an experimental research design, we should ask whether it fits the intervention we want to study. In particular, we need to consider the context in which the intervention is used, the content of the intervention itself, the process by which it is applied and the nature of its results or outcomes. In each of these domains, we may find either low variance (homogeneity) or high variance (heterogeneity). Table 1 offers some examples by way of illustration. Variance in context, content, application and outcome for interventions Variance in context, content, application and outcome for interventions When an intervention has low variance across these domains, then the experimental method elegantly eliminates all potential biases and confounders, proves causality beyond dispute and quantifies effect. The theoretical basis for the intervention is a secondary consideration—our focus is on its empirical performance. However, when we find high variance in one or more domains, the value of the experimental method is less clear, because the variance reduces or even eliminates our ability to generalize empirically about the impact of an intervention from any specific experiment, done with a certain context, content and application combination, and to draw conclusions about how it might work in a different context, with a different content and different application. Then, the theoretical basis for the intervention (why and how it works) becomes more important than its empirical performance (whether it works) in any particular study. As the natural heterogeneity of an intervention increases, experimental methods become progressively less helpful in understanding its effectiveness. We see this in the findings from experimental research into some clinical interventions, such as controlled trials of surgical procedures where application variance can be high, or of behavioural therapy interventions where content, application and outcome variance are significant. In such studies, the findings are more difficult to interpret and apply to clinical practice than those for other clinical interventions where variance is low—for example, many pharmaceutical therapies. But quality improvement initiatives are complex social interventions, for which high levels of variance in context, content and application are often inherent and desired characteristics of the initiative [3]. For example, the responses of different healthcare organizations to a continuous quality improvement programme, or a system for adverse event reporting and investigation, will be quite different—and the programme or system will rightly be tailored or modified to make it work better in the individual organizational setting. Attempts to ‘standardize’ or control such interventions to make them fit an experimental paradigm completely miss the point—that their multiple outcomes are a complex co-product of context, content and application variables. A different approach to evaluating effectiveness is needed. In researching healthcare quality improvement, we should learn from those in other settings—such as education [4] and criminal justice [5]—who have decades of experience in researching complex social interventions. Programmes designed to teach children to read, to

Overview

Integrity signals

Registry

Crossref · 10.1093/intqhc/mzm004

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • All improvement work, whether or not it is intended to generate publishable new knowledge, needs to be designed in a way that makes it possible to know whether the intervention has actually resulted in improvements.

    “This role is invaluable because all improvement work, whether or not it is intended to generate publishable new knowledge, needs to be designed in a way that makes it possible to know whether the intervention has actually resulted in improvements.31”

    From Demystifying theory and its use in improvement (2015) · cites this paper as [31]

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Pawson, 1997, Realistic Evaluation

Cited by

Sorted by citations

Related

Metadata

Authors
Walshe, K.
Affiliations
University of Manchester (GB)
Corresponding author
Walshe, Kieran
ORCID
Walshe, Kieran
Venue
International Journal for Quality in Health Care
Publisher
Oxford University Press (OUP)
Published
2007-03-02
Volume
19
Issue
2
Pages
57-59
ISSN
  • 1353-4505 (ISSN-L)
  • 1464-3677
DOI
10.1093/intqhc/mzm004
OpenAlex
W2058504953
Cited by
237
References
7
Citations by year
2026: 5 · 2024: 5 · 2023: 7 · 2022: 15 · 2021: 10 · 2020: 10 · 2019: 15 · 2018: 16 · 2017: 14 · 2016: 19 · 2015: 23 · 2014: 23 · 2013: 19 · 2012: 12
FWCI
17.70
Type
journal article
Language
en
Open access
closed
Locations
  • International Journal for Quality in Health Care (publishedVersion)
  • PubMed (publishedVersion)
  • Research Explorer (The University of Manchester) (submittedVersion)
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Healthcare Quality and Management
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
Keywords
quality improvement, theory-based evaluation, organizational interventions, randomized controlled trials, intervention heterogeneity, causal inference, generalizability, implementation research, quality of care, adverse event monitoring, continuous quality improvement, evidence-based practice, research design selection, healthcare organizations
MeSH terms
United Kingdom, Models, Theoretical, Quality Assurance, Health Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review