No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index; 4 of the 58 sources its claims cite were checked for support.
Abstract
BACKGROUND: Audit and feedback interventions in healthcare have been found to be effective, but there has been little progress with respect to understanding their mechanisms of action or identifying their key 'active ingredients.' DISCUSSION: Given the increasing use of audit and feedback to improve quality of care, it is imperative to focus further research on understanding how and when it works best. In this paper, we argue that continuing the 'business as usual' approach to evaluating two-arm trials of audit and feedback interventions against usual care for common problems and settings is unlikely to contribute new generalizable findings. Future audit and feedback trials should incorporate evidence- and theory-based best practices, and address known gaps in the literature. SUMMARY: We offer an agenda for high-priority research topics for implementation researchers that focuses on reviewing best practices for designing audit and feedback interventions to optimize effectiveness.
Overview
CoChat Summary
TL;DR
This paper argues that continuing “business as usual” two-arm trials of audit and feedback (A&F) versus usual care is unlikely to produce new generalizable findings, and instead proposes an agenda for optimizing A&F design. It summarizes consensus best practices and high-priority research topics from a 2012 international expert meeting, grounded in Cochrane review evidence of a median 4.3% absolute improvement.
Key findings
- The Cochrane review of 140 randomized A&F trials found a median 4.3% absolute improvement (interquartile range 0.5% to 16%) in provider compliance; one-quarter of interventions had relatively large positive effects, while another quarter had negative or null effects.
- Tentative best practices include valid and recent audit data about the individual/team’s own behavior, repeated audit cycles, multi-modal feedback, delivery from a trusted source, comparison data with relevant others, targets/goals, and a clear action plan when discrepancies are evident.
- A&F appears more effective when feedback is presented both verbally and in writing, comes from a supervisor or senior colleague, is repeated, and uses achievable benchmarks of care (top 10% of peers) rather than highlighting median peer performance; it is less effective when from a regulatory body.
- The authors recommend shifting future research from A&F versus usual care toward head-to-head comparisons of different A&F designs, fractional factorial trials, embedded process evaluations, and studies of contextual/recipient, intervention-design, and targeted-behavior factors.
Methods
- Design: a discussion/agenda paper summarizing outcomes from a two-day international expert meeting held in December 2012; participants were purposively sought based on contributions to A&F or related literature or as potential knowledge users, with a maximum of 30 attendees.
- Data: the most recent Cochrane review of A&F (140 randomized trials) and the collective experiences and research programs of meeting participants.
- Analysis: narrative synthesis and consensus discussion, producing Table 1 of tentative best practices and Table 2 of example high-priority research topics.
Limitations
- Stated in the paper: the recommendations reflect the limitations of the studies reviewed; the considerations only reflect opinions expressed at the meeting; with a maximum of 30 attendees, important perspectives may have been omitted; and there is no present universally accepted, comprehensive list of A&F intervention components.
- Evident from the text: the best practices are labeled “tentative,” and the paper does not conduct a formal systematic review, quantitative synthesis, or empirical validation of the proposed agenda.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 0 Not valid · 37 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Furthermore, most A&F interventions tested in RCTs are designed without explicitly building on previous research or extant theory [5,6].”
§ No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention > Background
What the sources say
- Not checked yetcites [5] Sales A Smith J Curran G Kochevar L Models, strategies, and tools. Theory in implementing evidence-based findings into health care practice J Gen Intern Med 2006 21 Suppl 2 43 49 PMC2557135 16637960
- Partly supportedcites [6] Colquhoun HL Brehaut JC Sales A Ivers N Grimshaw J Michie S Carroll K Chalifoux M Eva KW A systematic review of the use of theory in randomized controlled trials of audit and feedback Implement Sci 2013 8 1 66 10.1186/1748-5908-8-66 23759034 PMC3702512 sourceExperimental — accuracy not yet measured
The Results and Discussion report that only 13 of 140 A&F trials (9%) explicitly used theory to develop the intervention, supporting the claim about extant theory. The review assesses reported theory use, not whether interventions built on previous research; it also cautions that authors may have used theory without reporting it. No page numbers are available in our copy. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Organizational and system-level factors that may affect recipients’ ability to act upon a discrepancy highlighted by feedback could also be considered, including competing priorities for the recipients, teams and organizations [34].”
§ No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention > Discussion > 1b. Applying relevant theory to improve design and increase contribution to the literature
What the sources say
- Partly supportedcites [34] Presseau J Francis JJ Campbell NC Sniehotta FF Goal conflict, goal facilitation, and health professionals’ provision of physical activity advice in primary care: An exploratory prospective study Implement Sci 2011 6 73 10.1186/1748-5908-6-73 21762486 PMC3224555 sourceExperimental — accuracy not yet measured
Our copy supports only the 'competing priorities for the recipients' element: the Background states that clinical practice involves numerous goal-directed behaviours (GDBs) 'each competing for limited resources in patient consultations, in particular time-related resources', and the Discussion (Predicting health professional behaviour...) reports that the more participants perceived competing GDBs as taking time away from giving PA advice, the fewer patients received it. However, this is framed as the individual health professional's perception of intergoal conflict, not as 'organizational and system-level' factors, and nothing in our copy addresses competing priorities at the level of teams or organisations, nor feedback or acting on a highlighted discrepancy. The broader framing of the citing sentence therefore exceeds what Presseau et al. (2011) state. (LLM, conf=0.65)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Specifically, the theory of planned behaviour emphasizes the potential for social pressure to influence intentions to change professional practice [23]”
§ No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention > Discussion > 1a.i. A&F components
What the sources say
- Partly supportedcites [23] Ajzen I The theory of planned behavior Organ Behav Hum Decis Process 1991 50 2 179 211 10.1016/0749-5978(91)90020-T sourceExperimental — accuracy not yet measured
The section 'Predicting Intentions: Attitudes, Subjective Norms, and Perceived Behavioral Control' states that subjective norm is 'the perceived social pressure to perform or not to perform the behavior' and that, together with attitude and perceived behavioral control, it determines intention — so the social-pressure-to-intention part of the claim is directly supported (also the Abstract, which lists subjective norms as an intention predictor). What our copy does not support is the qualifier 'to change professional practice': the applications and studies it reports are leisure behavior, condom use, problem drinking, weight loss, and unethical acts among college students (cheating, shoplifting, lying), and its Conclusions frame the theory as a general framework for understanding or changing behavior in specified contexts, with no reference to professional or clinical practice. The citing sentence therefore extends this general theory paper to a professional-practice domain it does not address. (LLM, conf=0.78)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
In the paper
“Finally, the A&F review suggested that repeated feedback cycles led to greater improvements in performance than once-only feedback.”
§ No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention > Discussion > 1a.i. A&F components
What the sources say
- Not checked yetcites [3] Ivers N Jamtvedt G Flottorp S Young JM Odgaard-Jensen J French SD O’Brien MA Johansen M Grimshaw J Oxman AD Audit and feedback: effects on professional practice and healthcare outcomes Cochrane Database Syst Rev 2012 6 CD000259 PMC11338587 22696318
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The challenge of identifying factors that differentiate more and less successful A&F interventions is exacerbated by poor reporting of both intervention components and contextual factors in the literature [4].”
§ No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention > Background
What the sources say
- Not checked yetcites [4] Glasziou P Meats E Heneghan C Shepperd S What is missing from descriptions of treatment in trials and reviews? BMJ 2008 336 7659 1472 1474 10.1136/bmj.39590.732037.47 18583680 PMC2440840 source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
27 more claims
Create a free account to see all 54 claims and to verify the 54 references still open.
- It may state two separate claims as one. 10
- Probably not a substantial claim. 14
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, active research agendas have focused on determining the essential components of many of these “discrete” implementation strategies, such as audit and feedback [44], learning collaboratives [45], and supervision [46].”
From A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project (2015) · cites this paper as [44]
In the citing paper
“There are over 140 randomized trials of audit and feedback, but Ivers et al. (83) conclude that there is much to learn about when it will work best and why, and how to design reliable and effective audit and feedback strategies across different settings and providers.”
From Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda (2019) · cites this paper as [83]
In the citing paper
“However, the rationales and behaviour change techniques underpinning such interventions are often poorly developed and described, limiting cumulative learning that can enhance effects.72,73”
From Barriers to effective management of type 2 diabetes in primary care: qualitative systematic review (2016) · cites this paper as [72]
Citations
Cites
The theory of planned behavior
Ajzen, I. (1991) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(91)90020-t · Cited by 90,587
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
Building a practically useful theory of goal setting and task motivation: A 35-year odyssey.
Locke, E. A., & Latham, G. P. (2002) · American Psychologist · doi:10.1037/0003-066x.57.9.705 · Cited by 7,766 · cited 5×
“Alternatively, co-interventions that increase recipients’ skills in quality improvement may be useful because skill development and resultant selfefficacy may increase the likelihood that feedback will lead to improved performance [11,36].”
BackgroundMethodInfluentialKluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882 · cited 4×
“A&F has been used as a method for changing behaviour in educational [8], organizational [9-11], cognitive [12], and health psychology [13,14]; research in these areas has produced hypotheses about mechanisms of action.”
BackgroundMethodInfluentialAudit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485
Self-citationCane, J., O’Connor, D., & Michie, S. (2012) · Implementation Science · doi:10.1186/1748-5908-7-37 · Cited by 4,888 · cited 1×
“While the optimal design of A&F will vary to some extent for each purpose or setting, the strategies chosen for each component (Table 1) should be justified both by empirical evidence and theory [29].”
BackgroundSelf-citationShute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Gluesing, J. C. (2012) · Leadership in Science and Technology: A Reference Handbook · doi:10.4135/9781412994231.n14 · Cited by 3,210
Carver, C. S., & Scheier, M. F. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.1.111 · Cited by 2,650 · cited 4×
“A&F has been used as a method for changing behaviour in educational [8], organizational [9-11], cognitive [12], and health psychology [13,14]; research in these areas has produced hypotheses about mechanisms of action.”
BackgroundMethodInfluentialDavis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006) · JAMA · doi:10.1001/jama.296.9.1094 · Cited by 2,375 · cited 1×
“A&F is an improvement over self-assessment [1] or self-monitoring [2] as it can provide objective data regarding discrepancies between current practice and target performance, as well as comparisons of performance to other health professionals.”
BackgroundConsequences of individual feedback on behavior in organizations.
Ilgen, D. R., Fisher, C. D., & Taylor, M. S. (1979) · Journal of Applied Psychology · doi:10.1037/0021-9010.64.4.349 · Cited by 1,822 · cited 4×
“A&F has been used as a method for changing behaviour in educational [8], organizational [9-11], cognitive [12], and health psychology [13,14]; research in these areas has produced hypotheses about mechanisms of action.”
BackgroundMethodInfluentialSelf-evaluative and self-efficacy mechanisms governing the motivational effects of goal systems.
Bandura, A., & Cervone, D. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.5.1017 · Cited by 1,511 · cited 1×
“For example, at the individual level, increasing self-efficacy [30-32] and goal commitment [11,33] is likely to increase performance.”
BackgroundFeedback as an individual resource: Personal strategies of creating information
Ashford, S. J., & Cummings, L. L. (1983) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(83)90156-3 · Cited by 1,309 · cited 1×
“For example, at the individual level, increasing self-efficacy [30-32] and goal commitment [11,33] is likely to increase performance.”
BackgroundGodin, G., Bélanger-Gravel, A., Eccles, M., & Grimshaw, J. (2008) · Implementation Science · doi:10.1186/1748-5908-3-36 · Cited by 1,096 · cited 1×
“There is evidence suggesting that physicians respond differently than other health professionals to implementation interventions [42], and it is possible that this is also true for A&F.”
Self-citationLocal opinion leaders: effects on professional practice and health care outcomes
Flodgren, G., Parmelli, E., Doumit, G., Gattellari, M., O'Brien, M. A., Grimshaw, J., & Eccles, M. P. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000125.pub4 · Cited by 1,070
Self-citationBuilding a Practically Useful Theory of Goal Setting and Task Motivation
Locke, E. A., & Latham, G. P. (2002) · American Psychologist · Cited by 1,070
Baker, R., Camosso-Stefinovic, J., Gillies, C., Shaw, E. J., Cheater, F., Flottorp, S., & Robertson, N. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005470.pub2 · Cited by 1,003
Grol, R., Dalhuijsen, J., Thomas, S., Veld, C. I. ‘., Rutten, G., & Mokkink, H. (1998) · BMJ · doi:10.1136/bmj.317.7162.858 · Cited by 877 · cited 1×
“The complexity of clinical practice recommendations and the degree to which benefits can readily be observed may play an important role [21,38-40].”
BackgroundWhat is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614 · cited 1×
“The challenge of identifying factors that differentiate more and less successful A&F interventions is exacerbated by poor reporting of both intervention components and contextual factors in the literature [4].”
BackgroundExplaining Michigan: Developing an Ex Post Theory of a Quality Improvement Program
DIXON-WOODS, M., BOSK, C. L., AVELING, E. L., GOESCHEL, C. A., & PRONOVOST, P. J. (2011) · Milbank Quarterly · doi:10.1111/j.1468-0009.2011.00625.x · Cited by 603 · cited 1×
“In addition, a process evaluation within a trial can be used to investigate and develop program-specific theory [45].”
BackgroundKiefe, C. I., Allison, J. J., Williams, O. D., Person, S. D., Weaver, M. T., & Weissman, N. W. (2001) · JAMA · doi:10.1001/jama.285.22.2871 · Cited by 503 · cited 2×
“This illustrates the potential for A&F to have an effect even upon targeted behaviours reported by clinicians not to be compatible with their values [20,24].”
BackgroundMethodAn implementation research agenda
Eccles, M. P., Armstrong, D., Baker, R., Cleary, K., Davies, H., Davies, S., Glasziou, P., Ilott, I., Kinmonth, A.-L., Leng, G., et al. (2009) · Implementation Science · doi:10.1186/1748-5908-4-18 · Cited by 481
Self-citationGrilli, R., & Lomas, J. (1994) · Medical Care · doi:10.1097/00005650-199403000-00002 · Cited by 392 · cited 1×
“The complexity of clinical practice recommendations and the degree to which benefits can readily be observed may play an important role [21,38-40].”
BackgroundAudit and feedback and clinical practice guideline adherence: Making feedback actionable
Hysong, S. J., Best, R. G., & Pugh, J. A. (2006) · Implementation Science · doi:10.1186/1748-5908-1-9 · Cited by 371 · cited 4×
“Examining mechanisms of action and qualitative studies prior to trials may help identify barriers and facilitators to change that can inform intervention design [15].”
BackgroundInfluentialMeta-Analysis: Audit and Feedback Features Impact Effectiveness on Care Quality
Hysong, S. J. (2009) · Medical Care · doi:10.1097/mlr.0b013e3181893f6b · Cited by 306 · cited 1×
“For example, it incorporated re-analyses [14,17] of the previous Cochrane review informed by control theory [13] and feedback intervention theory [10], resulting in the finding that feedback is more effective when accompanied by both explicit goals and an action plan.”
MethodFoy, R., Eccles, M., Jamtvedt, G., Young, J., Grimshaw, J., & Baker, R. (2005) · BMC Health Services Research · doi:10.1186/1472-6963-5-50 · Cited by 250 · cited 1×
“As a result, there has been little progress with respect to identifying the key ingredients for a successful A&F intervention or understanding the mechanisms of action of effective A&F interventions in healthcare [7].”
BackgroundSelf-citationA systematic review of the use of theory in randomized controlled trials of audit and feedback
Colquhoun, H. L., Brehaut, J. C., Sales, A., Ivers, N., Grimshaw, J., Michie, S., Carroll, K., Chalifoux, M., & Eva, K. W. (2013) · Implementation Science · doi:10.1186/1748-5908-8-66 · Cited by 231 · cited 1×
“Furthermore, most A&F interventions tested in RCTs are designed without explicitly building on previous research or extant theory [5,6].”
BackgroundSelf-citationGardner, B., Whittington, C., McAteer, J., Eccles, M. P., & Michie, S. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.01.039 · Cited by 225 · cited 2×
“A&F has been used as a method for changing behaviour in educational [8], organizational [9-11], cognitive [12], and health psychology [13,14]; research in these areas has produced hypotheses about mechanisms of action.”
MethodSelf-citationMeasure, Learn, And Improve: Physicians’ Involvement In Quality Improvement
Audet, A.-M. J., Doty, M. M., Shamasdin, J., & Schoenbaum, S. C. (2005) · Health Affairs · doi:10.1377/hlthaff.24.3.843 · Cited by 219 · cited 1×
“A&F is an improvement over self-assessment [1] or self-monitoring [2] as it can provide objective data regarding discrepancies between current practice and target performance, as well as comparisons of performance to other health professionals.”
BackgroundSales, A. E., Smith, J. L., Curran, G. M., & Kochevar, L. K. (2006) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2006.00362.x · Cited by 217 · cited 1×
“Furthermore, most A&F interventions tested in RCTs are designed without explicitly building on previous research or extant theory [5,6].”
BackgroundSelf-citationMoving from cognition to behavior: What the research says.
Johnson, R. E., Chang, C.-H., & Lord, R. G. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.3.381 · Cited by 193 · cited 1×
“For example, at the individual level, increasing self-efficacy [30-32] and goal commitment [11,33] is likely to increase performance.”
BackgroundFoy, R., MacLennan, G., Grimshaw, J., Penney, G., Campbell, M., & Grol, R. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00403-1 · Cited by 185 · cited 1×
“An observational study showed that clinical practice recommendations rated as incompatible with clinician values and norms were associated with lower compliance at baseline but greater behavior change following feedback than recommendations perceived as highly compatible [26].”
BackgroundSelf-citationvan der Veer, S. N., de Keizer, N. F., Ravelli, A. C. J., Tenkink, S., & Jager, K. J. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.01.011 · Cited by 184 · cited 2×
“The effectiveness of A&F may be influenced by the characteristics of the targeted behavior, the recipients, their context, and the components of the A&F intervention itself [9,15,16].”
BackgroundAchievable benchmarks of care: the ABCTMs of benchmarking
Weissman, N. W., Allison, J. J., Kiefe, C. I., Farmer, R. M., Weaver, M. T., Williams, O. D., Child, I. G., Pemberton, J. H., Brown, K. C., & Baker, C. S. (1999) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.1999.00203.x · Cited by 167 · cited 1×
“In particular, recipients who received feedback with ‘achievable benchmarks of care’ [25] (i.”
BackgroundShekelle, P., Kravitz, R. L., Beart, J., Marger, M., Wang, M., & Lee, M. L. (2000) · PubMed · Cited by 155
Feedback, the various tasks of the doctor, and the feedforward alternative
Kluger, A. N., & Van Dijk, D. (2010) · Medical Education · doi:10.1111/j.1365-2923.2010.03849.x · Cited by 134 · cited 2×
“Conversely, feedback is less effective when it comes from a regulatory body, agreeing with qualitative work suggesting that feedback perceived to be potentially punitive appeared less effective than a supportive approach [15,20].”
BackgroundBuilding theories of knowledge translation interventions: Use the entire menu of constructs
Brehaut, J. C., & Eva, K. W. (2012) · Implementation Science · doi:10.1186/1748-5908-7-114 · Cited by 120 · cited 3×
“From a cognitive perspective, receipt of feedback that indicates a discrepancy between expected and actual performance may generate a response described as cognitive dissonance [12].”
BackgroundMethodAntimicrobial Stewardship Programs
Leuthner, K. D., & Doern, G. V. (2013) · Journal of Clinical Microbiology · doi:10.1128/jcm.01751-13 · Cited by 81
Collins, L. M., Chakraborty, B., Murphy, S. A., & Strecher, V. (2009) · Clinical Trials · doi:10.1177/1740774508100973 · Cited by 80 · cited 1×
“However, where there is an opportunity to consider many A&F-related factors at once, a multi-phase optimization strategy including a fractional factorial trial offers an efficient approach [44].”
BackgroundLi, T., Vedula, S. S., Scherer, R., & Dickersin, K. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-5-201203060-00009 · Cited by 78 · cited 1×
“Just as the need to identify research gaps and prioritize research agendas is accepted for clinical research [18], it is also a prerequisite for making substantial progress in implementation research [19].”
BackgroundPresseau, J., Sniehotta, F. F., Francis, J. J., & Campbell, N. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-77 · Cited by 77
Self-citationGlynn, R. J., Brookhart, M. A., Stedman, M., Avorn, J., & Solomon, D. H. (2007) · Medical Care · doi:10.1097/mlr.0b013e318070c0a0 · Cited by 72 · cited 1×
“There are opportunities to embed research within ongoing organizational initiatives at low cost in settings where routine data are available and where mechanisms for audit are in place, and this should be considered in the planning and conduct of A&F studies [41].”
BackgroundIvers, N. M., Barnsley, J. M., Upshur, R. E. G., Tu, K., Shah, B. R., Grimshaw, J., & Zwarenstein, M. (2014) · Europe PMC (PubMed Central) · Cited by 49
Self-citationPresseau, J., Francis, J. J., Campbell, N. C., & Sniehotta, F. F. (2011) · Implementation Science · doi:10.1186/1748-5908-6-73 · Cited by 47
Self-citationGoodwin, N. (2011) · Practice Nursing · doi:10.12968/pnur.2011.22.5.222 · Cited by 46
Ivers, N. M., Tu, K., Young, J., Francis, J. J., Barnsley, J., Shah, B. R., Upshur, R. E., Moineddin, R., Grimshaw, J. M., & Zwarenstein, M. (2013) · Implementation Science · doi:10.1186/1748-5908-8-142 · Cited by 34
Self-citationBonetti, D., Johnston, M., Pitts, N. B., Deery, C., Ricketts, I., Tilley, C., & Clarkson, J. E. (2009) · International Journal of Behavioral Medicine · doi:10.1007/s12529-008-9017-1 · Cited by 32 · cited 1×
“Alternatively, co-interventions that increase recipients’ skills in quality improvement may be useful because skill development and resultant selfefficacy may increase the likelihood that feedback will lead to improved performance [11,36].”
BackgroundDevelopment of quality of care indicators from systematic reviews: the case of hospital delivery
Bonfill, X., Figuls, M. R. I., Aller, M.-B., Osorio, D., Foradada, C. M., Vives, À., & Comas, D. R. (2013) · Implementation Science · doi:10.1186/1748-5908-8-42 · Cited by 30
Ivers, N. M., Tu, K., Francis, J., Barnsley, J., Shah, B., Upshur, R., Kiss, A., Grimshaw, J. M., & Zwarenstein, M. (2010) · Implementation Science · doi:10.1186/1748-5908-5-98 · Cited by 27
Self-citationFinancial incentives and physician commitment to guideline-recommended hypertension management.
Hysong, S. J., Simpson, K., Pietz, K., SoRelle, R., Smitham, K. B., & La, P. (2012) · The American Journal of Managed Care · Cited by 21
Theory in Implementing Evidence-Based Findings into Health Care Practice
Sales, A. E., Smith, J. L., Curran, G. M., & Kochevar, L. K. (2006) · Cited by 19
Self-citation(2009) · Clinical Trials · doi:10.1177/1740774508336326 · Cited by 2
Research works printed but not linked: 1
- Doran GT There’s a S.M.A.R.T. way to write management’s goals and objectives. AMA Forum Management review 1981 70 11 35 36
Cited by
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015) · Implementation Science · doi:10.1186/s13012-015-0209-1 · Cited by 4,963
“In fact, active research agendas have focused on determining the essential components of many of these “discrete” implementation strategies, such as audit and feedback [44], learning collaboratives [45], and supervision [46].”
BackgroundEnhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda
Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., & Weiner, B. J. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00003 · Cited by 775
“(83) conclude that there is much to learn about when it will work best and why, and how to design reliable and effective audit and feedback strategies across different settings and providers.”
BackgroundPractice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness
Brehaut, J. C., Colquhoun, H. L., Eva, K. W., Carroll, K., Sales, A., Michie, S., Ivers, N., & Grimshaw, J. M. (2016) · Annals of Internal Medicine · doi:10.7326/m15-2248 · Cited by 491
Self-citationLevers for addressing medical underuse and overuse: achieving high-value health care
Elshaug, A. G., Rosenthal, M. B., Lavis, J. N., Brownlee, S., Schmidt, H., Nagpal, S., Littlejohns, P., Srivastava, D., Tunis, S., & Saini, V. (2017) · The Lancet · doi:10.1016/s0140-6736(16)32586-7 · Cited by 440
Brown, B., Gude, W. T., Blakeman, T., van der Veer, S. N., Ivers, N., Francis, J. J., Lorencatto, F., Presseau, J., Peek, N., & Daker-White, G. (2019) · Implementation Science · doi:10.1186/s13012-019-0883-5 · Cited by 387
“Table 2 Popular theories to model clinical performance feedback in the literature [5]”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Ivers, N. M., Sales, A., Colquhoun, H., Michie, S., Foy, R., Francis, J. J., & Grimshaw, J. M.
- Affiliations
- University of Toronto (CA)
- University of Michigan (US)
- Ottawa Hospital (CA)
- University College London (GB)
- University of Leeds (GB)
- City, University of London (GB)
- University of Ottawa (CA)
- Corresponding author
- Ivers, Noah M
- ORCID
- Venue
- Implementation Science
- Published
- 2014-01-17
- Volume
- 9
- Issue
- 1
- Pages
- 14-14
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC3896824
- OpenAlex
- W2097831015
- Semantic Scholar
- bbd8c6056f8e825734b5543e95f88722fa0988e6
- Cited by
- 400
- References
- 52
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 39 · 2023: 36 · 2022: 54 · 2021: 54 · 2020: 35 · 2019: 32 · 2018: 33 · 2017: 33 · 2016: 21 · 2015: 19 · 2014: 10
- FWCI
- 49.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- City Research Online (City University London) (acceptedVersion)
- Deep Blue (University of Michigan) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- uO Research (University of Ottawa) (submittedVersion, public-domain)
- TSpace (University of Toronto) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- University of Toronto
- Canadian Institutes of Health Research
- Department of Family and Community Medicine, University of Toronto
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- audit and feedback, intervention design, mechanism of action, healthcare quality improvement, two-arm trials, usual care comparison, research agenda, implementation research
- MeSH terms
- Attitude of Health Personnel, Behavior, Feedback, Health Services Research, Humans, Medical Audit, Quality of Health Care, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.