Developing and evaluating complex interventions: the new Medical Research Council guidance
Open accessAbstract
Evaluating complex interventions is complicated. The Medical Research Council9s evaluation framework (2000) brought welcome clarity to the task. Now the council has updated its guidance
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 31 cited sources checked · 0 supported · 0 flagged
- Premisecitation
Evidence strength does not always determine uptake; strong evidence can be disregarded and weak evidence adopted based on political acceptability or alignment with other beliefs about what works.
“Strong evidence may be ignored or weak evidence taken up, depending on its political acceptability or fit with other ideas about what works.11”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [11] 11. Muir H. Let science rule: the rational way to run societies. New Scientist 2008;198:40-3.
Not checked in the free run.
- Premisecitation
Recent publications have pointed out shortcomings in the 2000 MRC framework.
“Several recent papers have identified limitations in the framework, recommending, for example, greater attention to early phase piloting and development work, 3 a less linear model of evaluation process, 4 integration of process and outcome evaluation, 5”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [3] 3. Hardeman W, Sutton S, Griffin S, Johnston M, White A, Wareham NJ, et al. A causal modelling approach to the development of theory-based behaviour change programmes for trial evaluation. Health Educ Res 2005;20:676-87.
Not checked in the free run.
- cites [4] 4. Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. BMJ 2007;334:455-9.
Not checked in the free run.
- cites [5] 5. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Ripple Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ 2006;332:413-6.
Not checked in the free run.
- Premisecitation
Using experimental methods to assess health effects can face political or ethical opposition, particularly when the intervention yields significant non-health benefits.
“but there may be political or ethical objections to using them to assess health effects, especially if the intervention provides important non-health benefits.13”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [13] 13. Thomson H, Hoskins R, Petticrew M, Ogilvie D, Craig N, Quinn T, et al. Evaluating the health effects of social interventions. BMJ 2004;328:282-5.
Not checked in the free run.
- Premisecitation
In certain evaluations, such as those aiming to identify active ingredients in a complex intervention, strict standardisation and controls to limit implementation variation may be necessary.
“In some evaluations, such as those seeking to identify active ingredients within a complex intervention, strict standardisation may be required and controls put in place to limit variation in implementation.31”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [31] 31. Farmer A, Wade A, Goyder E, Yudkin P, French D, Craven A, et al. Impact of self-monitoring of blood glucose in the management of patients with non-insulin treated diabetes: open parallel group randomised trial. BMJ 2007;335:132-9.
Not checked in the free run.
- Premisecitation
Reporting of complex intervention studies has historically been inadequate.
“This has been a weakness of the reporting of complex intervention studies in the past, 34”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [34] 34. Glasziou P, Meats E, Heneghan C, Shepperd S. What is missing from descriptions of treatments in trials and reviews? BMJ 2008;336:1472-4.
Not checked in the free run.
- Premisecitation
Fidelity is complex when dealing with complex interventions.
“Fidelity is not straightforward in relation to complex interventions.30”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [30] 30. Hawe P, Shiell A, Riley T. Complex interventions: how "out of control" can a randomised trial be? BMJ 2004;328:1561-3.
Not checked in the free run.
- Premisecitation
Observational evaluations that succeed, such as those of suicide means restriction, air pollution reduction, or public smoking bans, usually involve interventions with rapid and large effects.
“Successful examples, such as the evaluation of legislation to restrict access to means of suicide, 24 reduce air pollution, 25 or ban smoking in public places, 26 tend to occur where interventions have rapid, large effects.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [24] 24. Gunnell D, Fernando R, Hewagama M, Priyangika W, Konradsen F, Eddleston M. The impact of pesticide regulations on suicide in Sri Lanka. Int J Epidemiol 2007;36:1235-42.
Not checked in the free run.
- cites [25] 25. Clancy L, Goodman P, Sinclair H, Dockery DW. Effect of air pollution control on death rates in Dublin, Ireland: an intervention study. Lancet 2002;360:1210-4.
Not checked in the free run.
- cites [26] 26. Haw SJ, Gruer L. Changes in exposure of adult non-smokers to secondhand smoke after implementation of smoke-free legislation in Scotland: national cross sectional survey. BMJ 2007;335:549-52.
Not checked in the free run.
- Rests on an assumption: Randomisation may be unnecessary when intervention effects are very large or immediate, making confounding or underlying trends unlikely explanations.
- May combine two claims. 11
- Classified as probably not a substantial claim. 4
May combine two claims.
FindingcitationA modelling study conducted before a falls prevention trial in older people indicated that the planned screening and referral system was very unlikely to be cost-effective, which led to a decision not to proceed with the trial.
“A modelling exercise to prepare for a trial of falls prevention in elderly people showed that the proposed system of screening and referral was highly unlikely to be cost effective and informed the decision not to proceed with the trial.17”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [17] 17. Eldridge S, Spencer A, Cryer C, Pearsons S, Underwood M, Feder G. Why modelling a complex intervention is an important precursor to trial design: lessons from studying an intervention to reduce falls-related injuries in elderly people. J Health Services Res Policy 2005;10:133-42.
Not checked in the free run.
May combine two claims.
FindingcitationPilot work for a free home insulation trial indicated that attrition could be high, prompting a design change where control participants received the intervention after the study.
“Pilot studies for a trial of free home insulation suggested that attrition might be high, so the design was amended such that participants in the control group received the intervention after the study.20”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [20] 20. Howden-Chapman P, Crane J, Matheson A, Viggers H, Cunningham M, Blakely T, et al. Retrofitting houses with insulation to reduce health inequalities: aims and methods of a clustered community-based trial. Soc Sci Med 2005;61:2600-10.
Not checked in the free run.
May combine two claims.
FindingcitationAn evaluation of a preschool programme for disadvantaged children found improved educational attainment and a variety of economic and social benefits at ages 27 and 40.
“Evaluation of a preschool programme for disadvantaged children showed that, as well as improved educational attainment, there was a range of economic and social benefits at ages 27 and 40.27”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [27] 27. Wortman PM. An exemplary evaluation of a program that worked: the High/Scope Perry preschool project. Am J Eval 1995;16:257-65.
Not checked in the free run.
May combine two claims.
FindingcitationA process evaluation alongside a trial of educational visits to encourage GPs to follow prescribing guidelines found that the visits were well received and guideline recall was good, but prescribing behaviour changed little because of other constraints such as patient preferences and local hospital policy.
“A process evaluation 28 carried out in connection with a trial of educational visits to encourage general practitioners to follow prescribing guidelines 29 found that the visits were well received and recall of the guidelines was good, yet there was little change in prescribing behaviour, which was constrained by other factors such as patients' preferences and local hospital policy.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [28] 28. Nazareth I, Freemantle N, Duggan C, Mason J, Haines A. Evaluation of a complex intervention for changing professional behaviour: the evidence based out reach (EBOR) trial. J Health Services Res Policy 2002;7:230-8.
Not checked in the free run.
- cites [29] 29. Freemantle N, Nazareth I, Eccles M, Wood J, Haines A, EBOR Triallists. A randomised controlled trial of the effect of educational outreach by community pharmacists on prescribing in UK general practice. Br J Gen Pract 2002;52:290-5.
Not checked in the free run.
- Rests on an assumption: Process evaluations can offer valuable understanding of why an intervention fails, has unexpected consequences, or succeeds and how it can be improved.
May combine two claims.
FindingcitationA feasibility study for an adolescent sexual health intervention in rural Zimbabwe determined that the intended classroom-based programme was unsuitable because of cultural norms, teaching styles, and teacher-pupil relationships, so it was replaced with a community-based programme.
“A feasibility study for an evaluation of an adolescent sexual health intervention in rural Zimbabwe found that the planned classroom based programme was inappropriate, given cultural norms, teaching styles, and relationships between teachers and pupils in the country, and it was replaced by a community based programme.19”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [19] 19. Power R, Langhaug L, Nyamurera T, Wilson D, Bassett M, Cowan F. Developing complex interventions for rigorous evaluation-a case study from rural Zimbabwe. Health Educ Res 2004;19:570-5.
Not checked in the free run.
May combine two claims.
FindingcitationIn a school-based health and wellbeing trial, schools used a standardised process to develop strategies suited to them instead of a fixed curriculum, leading to wide variation in practice across schools.
“In a trial of a school based intervention to promote health and wellbeing, schools were encouraged to use a standardised process to develop strategies which suited them rather than adopt a fixed curriculum, resulting in widely varied practice between schools.32”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [32] 32. Patton G, Bond L, Butler H, Glover S. Changing schools, changing health? Design and implementation of the Gatehouse Project. J Adolesc Health 2003;33:231-9.
Not checked in the free run.
- Rests on an assumption: Requiring strict standardisation may be inappropriate.
May combine two claims.
PremisecitationComplex intervention ideas come from diverse sources, and this can strongly influence how much freedom researchers have to alter the intervention, affect its implementation, or use an ideal evaluation design.
“Ideas for complex interventions emerge from various sources, which may greatly affect how much leeway the researcher has to modify the intervention, to influence the way it is implemented, or to adopt an ideal evaluation design.8”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [8] 8. Belsky J, Melhuish E, Barnes J, Leyland AH, Romaniuk H, National Evaluation of Sure Start Research Team. Effects of Sure Start local programmes on children and families: early findings from a quasi-experimental, cross sectional study. BMJ 2006;332:1476-81.
Not checked in the free run.
May combine two claims.
PremisecitationThe development and evaluation of a complex intervention involves multiple phases, but these phases may not proceed in a linear or cyclical order.
“The process of developing and evaluating a complex intervention has several phases, although they may not follow a linear sequence”
- Also stated in: § Development, evaluation, and implementation · hedged
“Although it is useful to think in terms of phases, in practice these may not follow a linear or even a cyclical sequence (figure⇓).4”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [4] 4. Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. BMJ 2007;334:455-9.
Not checked in the free run.
- Also stated in: § Development, evaluation, and implementation · hedged
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationExperimental methods are gaining broader acceptance for evaluating policy.
“Experimental methods are becoming more widely accepted as methods to evaluate policy, 12”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [12] 12. Creegan C, Hedges A. Towards a policy evaluation service: developing infrastructure to support the use of experimental and quasi-experimental methods . London: Ministry of Justice, 2007.
Not checked in the free run.
May combine two claims.
PremisecitationProblems with acceptability, compliance, intervention delivery, recruitment, retention, and smaller-than-expected effect sizes often weaken evaluations, and these issues could have been anticipated through thorough piloting.
“Evaluations are often undermined by problems of acceptability, compliance, delivery of the intervention, recruitment and retention, and smaller than expected effect sizes that could have been predicted by thorough piloting.18”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] 18. Eldridge S, Ashby D, Feder G, Rudnicka AR, Ukoumunne OC. Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care. Clin Trials 2004;1:80-90.
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationEven methods that are not optimal for internal validity, if they are the best available, can produce useful results.
“the best available methods, even if they are not optimal in terms of internal validity, may yield useful results.14”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [14] 14. Ogilvie D, Mitchell R, Mutrie N, Petticrew M, Platt S. Evaluating health effects of transport interventions: methodologic case study. Am J Prev Med 2006;31:118-26.
Not checked in the free run.
May combine two claims.
PremisecitationA process evaluation embedded in a trial can assess implementation fidelity and quality, clarify causal mechanisms, and identify contextual factors linked to outcome variation.
“A process evaluation nested inside a trial can be used to assess fidelity and quality of implementation, clarify causal mechanisms, and identify contextual factors associated with variation in outcomes.5”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5] 5. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Ripple Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ 2006;332:413-6.
Not checked in the free run.
- Rests on an assumption: Process evaluations can offer valuable understanding of why an intervention fails, has unexpected consequences, or succeeds and how it can be improved.
May combine two claims.
PremisecitationRandomisation can sometimes be unnecessary and other designs may be better, but observational methods only yield reliable effect estimates under limited conditions.
“In some circumstances, randomisation may be unnecessary and other designs preferable, 21 22 but the conditions under which observational methods can yield reliable estimates of effect are limited (box 4).23”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [21] 21. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312:1215-8.
Not checked in the free run.
- cites [22] 22. Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are randomised trials unnecessary? Picking signal from noise. BMJ 2007;334:349-51.
Not checked in the free run.
- cites [23] 23. MacMahon S, Collins R. Reliable assessment of the effects of treatment on mortality and major morbidity. II. observational studies. Lancet 2001;357:455-62.
Not checked in the free run.
- Rests on an assumption: Randomisation may be unnecessary when intervention effects are very large or immediate, making confounding or underlying trends unlikely explanations.
Probably not a substantial claim: may not pass S2 (checkable).
PremisecitationComplex interventions may be most effective when adapted to local circumstances rather than fully standardised.
“Complex interventions may work best if tailored to local circumstances rather than being completely standardised”
- Also stated in: § Abstract · hedged
“recognition that complex interventions may work best if they are tailored to local contexts rather than completely standardised, 6”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [6] 6. Campbell M, Donner A, Klar N. Developments in cluster randomised trials and Statistics in Medicine. Stat Med 2007;26:2-19.
Not checked in the free run.
- Rests on an assumption: Requiring strict standardisation may be inappropriate.
- Also stated in: § Abstract · hedged
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThe existence of comprehensive reporting guidelines covering non-drug trials and observational studies, accessible via a single website, should improve reporting.
“but the availability of a comprehensive range of reporting guidelines, now covering non-drug trials 35 and observational studies 36 and accessible through a single website (www.equator-network.org)should lead to improvement.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [33] 33. Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol 2008;27:379-87.
Not checked in the free run.
- cites [35] 35. Boutron I, Moher D, Altman D, Scultz K, Ravaud P. Extending the CONSORT statement to randomized trials of non-pharmacologic treatment: explanation and elaboration. Ann Int Med 2008;148:295-309.
Not checked in the free run.
- cites [36] 36. Von Elm E, Altman D, Egger M, Pocock SJ, Gotzsche P, Vandenbroucke JP, et al. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 2007;335:806-8. RESEARCH METHODS & REPORTING on 10 July 2023 by guest. Protected by copyright. http://www.bmj.com/ BMJ: first published as 10.1136/bmj.a1655 on 29 September 2008. Downloaded from
Not checked in the free run.
- Rests on an assumption: Because implementation varies, full reporting of process and outcome evaluations and a clear intervention description are important for replication and evidence synthesis.
The paper relies on this without support.
Randomisation is the most robust way to prevent selection bias.
“Randomisation should always be considered because it is the most robust method of preventing selection bias.”
The paper relies on this without support.
Experimental designs are generally preferred over observational designs, though they are not always feasible.
“Experimental designs are preferred to observational designs in most circumstances, but are not always practicable”
The paper relies on this without support.
Understanding processes is valuable but cannot replace outcome evaluation.
“Understanding processes is important but does not replace evaluation of outcomes”
The paper relies on this without support.
A lack of effect might indicate implementation failure rather than true ineffectiveness.
“Lack of effect may reflect implementation failure (or teething problems) rather than genuine ineffectiveness”
The paper relies on this without support.
Variation in individual outcomes may stem from higher-level processes.
“Variability in individual level outcomes may reflect higher level processes”
The paper relies on this without support.
Using a single primary outcome may not be the best use of data.
“A single primary outcome may not make best use of the data”
The paper relies on this without support.
Requiring strict standardisation may be inappropriate.
“Ensuring strict standardisation may be inappropriate”
The paper relies on this without support.
Randomisation may be unnecessary when intervention effects are very large or immediate, making confounding or underlying trends unlikely explanations.
“Randomisation may be unnecessary if the effects of the intervention are so large or immediate that confounding or underlying trends are unlikely to explain differences in outcomes before and after exposure.”
The paper relies on this without support.
Post-hoc adjustment is inferior, and its effectiveness is limited by measurement error in confounders and by unknown or unmeasured confounders.
“Post-hoc adjustment is a second best solution; its effectiveness is limited by errors in the measurement of the confounding variables and the difficulty of dealing with unknown or unmeasured confounders”
The paper relies on this without support.
Randomisation may be impractical when the intervention is already widely used or when key implementation decisions have already been made.
“Randomisation may be impractical if the intervention is already in widespread use, or if key decisions about how it will be implemented have already been taken”
The paper relies on this without support.
There is no clear dividing line between simple and complex interventions.
“There is no sharp boundary between simple and complex interventions.”
The paper relies on this without support.
Evaluation occurs in diverse settings that restrict researchers' choices of interventions and evaluation methods.
“In practice, evaluation takes place in a wide range of settings that constrain researchers' choice of interventions to evaluate and their choice of evaluation methods.”
The paper relies on this without support.
Modelling a complex intervention before a full-scale evaluation can yield important information for designing both the intervention and the evaluation.
“Modelling a complex intervention before a full scale evaluation can provide important information about the design of both the intervention and the evaluation.”
The paper relies on this without support.
When an intervention is rolled out more widely, effects may be smaller or more variable and response rates may be lower.
“Effects may be smaller or more variable and response rates lower when the intervention is rolled out across a wider range of settings.”
The paper relies on this without support.
Process evaluations can offer valuable understanding of why an intervention fails, has unexpected consequences, or succeeds and how it can be improved.
“Process evaluations, which explore the way in which the intervention under study is implemented, can provide valuable insight into why an intervention fails or has unexpected consequences, or why a successful intervention works and how it can be optimised.”
The paper relies on this without support.
Process evaluation does not replace outcome evaluation.
“However, it is not a substitute for evaluation of outcomes.”
The paper relies on this without support.
Because implementation varies, full reporting of process and outcome evaluations and a clear intervention description are important for replication and evidence synthesis.
“Variability in implementation, preplanned or otherwise, makes it important that both process and outcome evaluations are reported fully and that a clear description of the intervention is provided to enable replication and synthesis of evidence.33”
The paper relies on this without support.
A solid theoretical understanding of how the intervention produces change is necessary to identify and strengthen weak links in the causal chain.
“A good theoretical understanding is needed of how the intervention causes change, so that weak links in the causal chain can be identified and strengthened”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Complex interventions containing several interacting components may work best if tailored to individual circumstances.
“Complex interventions containing several interacting components may work best if tailored to individual circumstances,39”
From Coached, Coordinated, Enhanced Neonatal Transition (CCENT): Protocol for a multicentre pragmatic randomised controlled trial of transition-to-home support for parents of high-risk infants (2021) · cites this paper as [39]
The paper describes complex interventions as having several interacting components (What are complex interventions?) and says they may work best when tailored to local circumstances (Conclusions; Box 2). It discusses adaptation to settings, not tailoring to an individual's circumstances, so that part of the claim is not supported. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAlmost all changes to practice in primary care involve complex interventions, i.e. interventions with multiple interconnecting components.
“Almost all changes to practice in primary care involve “complex interventions”, i.e. interventions with multiple interconnecting components [11]”
From Achieving change in primary care—causes of the evidence to practice gap: systematic reviews of reviews (2015) · cites this paper as [11]
In “What are complex interventions?” (page not given), the paper describes complex interventions as having several interacting components and says few interventions are truly simple. It does not establish that almost all changes to practice specifically in primary care are complex interventions. (LLM, conf=0.83)
Experimental — accuracy not yet measuredRecent research into digital health intervention development highlights the importance of an iterative, multidisciplinary approach in which the end user is placed at the heart of the system and the technology is grounded in user wants and needs.
“Recent research into digital health intervention development highlights the importance of an iterative, multidisciplinary approach in which the end user is placed at the heart of the system and the technology is grounded in user wants and needs [45-47].”
From British South Asian Patients’ Perspectives on the Relevance and Acceptability of Mobile Health Text Messaging to Support Medication Adherence for Type 2 Diabetes: Qualitative Study (2020) · cites this paper as [46]
The Development, evaluation, and implementation section (page not available) recommends phased testing, while Developing a complex intervention (page not available) describes progressively refining a design. These passages support iterative development, but the paper does not establish the claim about digital health technology, multidisciplinary development, or placing end users’ wants and needs at its centre. (LLM, conf=0.87)
Experimental — accuracy not yet measuredGiven the complexity and multifactorial nature of adherence, interventions aimed at improving adherence to inhaled therapies are likely to be complex, or contain several interacting components (Craig 2008).
“Given the complexity and multifactorial nature of adherence, interventions aimed at improving adherence to inhaled therapies are likely to be complex, or contain several interacting components (Craig 2008).”
From Psychological interventions for improving adherence to inhaled therapies in people with cystic fibrosis (2023) · cites this paper as Craig 2008
The paper’s ‘What are complex interventions?’ discussion defines complex interventions as containing several interacting components, and Box 1 lists features that contribute to complexity. It does not address adherence to inhaled therapies or establish that interventions targeting it are likely to be complex; that application is the citing authors’ inference. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWhen developing a complex intervention, a theoretical understanding of the likely processes eliciting behavior change should be explored.
“When developing a complex intervention, it is recommended that a theoretical understanding of the likely processes eliciting behavior change be explored”
From Improving Mental Health Service Utilization Among Men: A Systematic Review and Synthesis of Behavior Change Techniques Within Interventions Targeting Help-Seeking (2019) · cites this paper as Craig et al., 2008
The development of the intervention was guided by the recommendations of the Medical Research Council (MRC) framework for complex interventions.
“The development of the intervention was guided by the recommendations of the Medical Research Council (MRC) framework for complex interventions [29–31].”
Feasibility outcomes were determined using the MRC framework and reported in line with CONSORT guidelines.
“Feasibility outcomes were determined using the MRC framework26 and reported in line with CONSORT guidelines28”
From Mixed-methods feasibility outcomes for a novel ACT-based video game ‘ACTing Minds’ to support mental health (2024) · cites this paper as [26]
The lack of a comprehensive process evaluation is a deficiency in the study.
“Another deficiency in the study is the lack of a comprehensive process evaluation such as the one Craig et al69 described in the Medical Research Council guidelines for complex interventions.69”
It is recommended to conduct a pilot trial before the real RCT to assess feasibility and improve the quality of the real RCT.
“It is recommended to conduct a pilot trial before the real RCT to assess feasibility and improve the quality of the real RCT [20–22].”
Given the widespread interest in taking positive psychology online for clinical and subclinical populations, it is salient to ensure that interventions are not undermined by issues of acceptability.
“Given the widespread interest in taking positive psychology online for clinical and subclinical populations, [3, 5, 6] it is salient to ensure that interventions are not undermined by issues of acceptability [8].”
From What might affect acceptability of online positive psychology interventions for depression: A qualitative study on patient expectations' (2018) · cites this paper as [8]
The early evaluation stage is very useful in providing insights regarding the appropriateness of the procedures, the recruitment process, participant retention and how acceptable the participants find the program.
“The early evaluation stage is very useful in providing insights regarding the appropriateness of the procedures, the recruitment process, participant retention and how acceptable the participants find the program [28].”
The MRC framework emphasizes the importance of assessing feasibility and acceptability prior to embarking on a definitive trial.
“The MRC framework emphasizes the importance of assessing feasibility and acceptability prior to embarking on a definitive trial [28].”
Qualitative data were analyzed using a thematic analysis process with coding reliability, and authors further analyzed findings and discussed until consensus was achieved.
“Qualitative data (ie, derived from 267 follow-up phone call notes, equaling approximately 55 single-line pages) were uploaded by coauthor K.B. to the software program NVivo version 12 (QSR International, Victoria, Australia) and analyzed using a thematic analysis process (ie, coding reliability).12,14 Authors L.S.N., E.B., and E.B.S. then further analyzed findings and discussed until consensus was”
From Living well with chronic pain: a 12-month randomized controlled trial revealing impact from the digital pain self-management program EPIO (2024) · cites this paper as [14]
The cited paper’s “Understanding processes” section discusses process evaluations but does not describe thematic analysis, coding reliability, or consensus coding. Searches for those methods found no passage addressing them; the authors’ later discussion to consensus is described as their own procedure, not a finding of this source. (LLM, conf=0.76)
Experimental — accuracy not yet measuredAdverse mental burden may lead to emotional eating and decreased physical activity and thus to increases in bodyweight.
“Adverse mental burden may lead to emotional eating and decreased physical activity [9, 10, 11, 12, 13, 14] and thus to increases in bodyweight [15].”
From Supporting Weight Management during COVID-19: A Randomized Controlled Trial of a Web-Based, ACT-Based, Guided Self-Help Intervention (2022) · cites this paper as [9]
The 'Developing a complex intervention' section mentions physical activity only as an example of an intervention being modelled to prevent type 2 diabetes. Searches for emotional eating, mental burden, reduced physical activity and bodyweight found no passage linking them as the citing sentence does; page numbers were not available in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredImplementation of innovations into practice is a complex process.
“Implementation of innovations into practice is a complex process [1].”
From A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework (2015) · cites this paper as [1]
The importance of thoroughly establishing the feasibility of intervention and evaluation plans prior to embarking on an expensive, fully powered evaluation was indicated in the Medical Research Council's framework for the development and evaluation of complex interventions to improve health.
“The importance of thoroughly establishing the feasibility of intervention and evaluation plans prior to embarking on an expensive, fully powered evaluation was indicated in the Medical Research Council’s (MRC) framework for the development and evaluation of complex interventions to improve health [9, 10].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Pre-requisite activities were often discussed within the context of the MRC's intervention development-evaluation cycle.
“These were often discussed within the context of the MRC’s intervention development-evaluation cycle [6, 9, 10, 13, 25–28].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Updated MRC guidance in 2008 moved away from the sole focus on RCTs of its predecessor, reflecting recognition that not all interventions can be tested using an RCT and that the next most robust methods may sometimes be the best available option.
“Updated MRC guidance in 2008 moved away from the sole focus on RCTs (randomised controlled trials) of its predecessor reflecting recognition that not all interventions can be tested using an RCT and that the next most robust methods may sometimes be the best available option [10, 14].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Pilot and feasibility studies were sometimes used interchangeably.
“‘Pilot’ and ‘feasibility’ studies were sometimes used interchangeably [10, 20, 25–28, 31] while others made distinctions between the two according to design features or particular aims [7, 8, 19, 29, 32–34].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Identifying potential implementation issues was highlighted.
“Understanding how intervention components interact with various contextual settings [6, 27, 29] and identifying unintended harms [6, 29] as well as potential implementation issues [6, 9, 10, 30] were also highlighted.”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
These were sometimes referred to as aims within an embedded process evaluation that took place alongside an exploratory study, although the term process evaluation was never defined.
“These were sometimes referred to as aims within an embedded process evaluation that took place alongside an exploratory study, although the term process evaluation was never defined [7, 10, 15, 29, 40].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Later guidance from the MRC is more ambiguous than earlier guidance, stating that estimates should be interpreted with caution, while simultaneously stating 'safe' assumptions of effect sizes as a pre-requisite before continuing to a full evaluation.
“Later guidance from the MRC is more ambiguous than earlier guidance, stating that estimates should be interpreted with caution, while simultaneously stating ‘safe’ assumptions of effect sizes as a pre-requisite before continuing to a full evaluation [10].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Assessing feasibility of future evaluation procedures was commonly recommended to avert problems that could undermine the conduct or acceptability of future evaluation.
“Assessing feasibility of future evaluation procedures was commonly recommended [6, 7, 10, 15, 30, 32–34, 37, 41] to avert problems that could undermine the conduct or acceptability of future evaluation [6, 15, 30].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Updated MRC guidance moved to a more iterative model, accepting that movement back between feasibility/piloting and intervention development may sometimes be needed.
“While updated MRC guidance [10] moved to a more iterative model, accepting that movement back between feasibility/piloting and intervention development may sometimes be needed”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
Other guidance highlighted specific items concerning cost-effectiveness, refining causal hypotheses to be tested in a future evaluation, and meeting recruitment targets.
“while others highlight specific items concerning cost-effectiveness [10], refining causal hypotheses to be tested in a future evaluation [29] and meeting recruitment targets [20, 34].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [10]
A feasibility study for a randomised controlled trial does not necessarily involve a pilot randomised controlled trial.
“A feasibility study for a randomised controlled trial does not necessarily involve a pilot randomised controlled trial [1] but may do so”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
The feasibility phase follows the development phase of an intervention, in which qualitative methods may also be used.
“The feasibility phase follows the development phase of an intervention, in which qualitative methods may also be used [1].”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
Qualitative methods can be used alone or in conjunction with quantitative methods, such as modelling and surveys, in the feasibility phase.
“Qualitative methods can be used alone or in conjunction with quantitative methods, such as modelling and surveys, in the feasibility phase [1].”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
The United Kingdom Medical Research Council guidance on the development and evaluation of complex interventions recommends an early phase of assessing feasibility prior to a full evaluation.
“The United Kingdom Medical Research Council (UK MRC) guidance on the development and evaluation of complex interventions recommends an early phase of assessing feasibility prior to a full evaluation [1].”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
In the feasibility and pilot phase, researchers can identify and address problems which might undermine the acceptability and delivery of the intervention or the conduct of the evaluation.
“In this feasibility and pilot phase, researchers can identify and address problems which might undermine the acceptability and delivery of the intervention or the conduct of the evaluation.”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
A feasibility study may or may not include a pilot randomised controlled trial.
“A feasibility study may or may not include a pilot randomised controlled trial.”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
When the outcome evaluation is a randomised controlled trial, the feasibility phase increases the chances of researchers evaluating the optimum intervention using the most appropriate and efficient recruitment practices and trial design.
“When the outcome evaluation is a randomised controlled trial, this feasibility phase increases the chances of researchers evaluating the optimum intervention using the most appropriate and efficient recruitment practices and trial design.”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [1]
The person-based approach is not intended to replace but to complement and enrich theory-based and evidence-based approaches to intervention development.
“The person-based approach is not intended to replace but to complement and enrich the well-known “theory-based” and “evidence-based” approaches to incorporating behavioral science into intervention development [23,24].”
From The Person-Based Approach to Intervention Development: Application to Digital Health-Related Behavior Change Interventions (2015) · cites this paper as [23]
In many cases, research and development will not move through stages linearly but will go through development-evaluation-development cycles.
“In many cases, the research and development will not move through these stages in a linear fashion but will instead go through development-evaluation-development cycles in order to make improvements to the intervention based on data gained during evaluation phases [23].”
From The Person-Based Approach to Intervention Development: Application to Digital Health-Related Behavior Change Interventions (2015) · cites this paper as [23]
The UK MRC guidance called for improved methods of specifying and reporting intervention content in order to address the problems of lack of consistency and consensus.
“Consequently, the UK MRC guidance 5 for developing and evaluating complex interventions called for improved methods of specifying and reporting intervention content in order to address the problems of lack of consistency and consensus.”
Implementation researchers could advance understanding of the mechanisms through which implementation strategies influence EBI adoption, implementation, scale-up, and sustainment by facilitating the alignment of strategies to theories that explain how they work.
“They also could advance understanding of the mechanisms through which implementation strategies influence EBI adoption, implementation, scale-up, and sustainment by facilitating the alignment of strategies to theories that explain how they work [46].”
From Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice (2017) · cites this paper as [46]
The classification system's focus on targets of action (levels and determinants) has potential to aid in selecting and strategically combining multiple strategies.
“Consistent with the intent of Proctor et al.’s guidelines, this classification system’s focus on targets of action (levels and determinants) has potential to aid in selecting and strategically combining multiple strategies [45, 46].”
From Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice (2017) · cites this paper as [46]
The UK Medical Research Council guidance on developing and evaluating complex interventions recommends the use of theory in the intervention development phase.
“the UK Medical Research Council guidance on developing and evaluating complex interventions recommends the use of theory in the intervention development phase [24].”
From Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research (2014) · cites this paper as [24]
Using theory for identifying determinants of behaviour and selecting interventions can increase the likelihood of the complex interventions being appropriate.
“Using theory for identifying determinants of behaviour and selecting interventions can increase the likelihood of the complex interventions being appropriate [15].”
MOST uses the resource management and continuous optimization principles to improve smoking cessation research efficiency and intervention effectiveness and cost-effectiveness.
“MOST (4, 5), which has some conceptual roots in the phased approach to intervention development and evaluation proposed by the United Kingdom’s Medical Research Council (14, 15) applies the resource management and continuous optimization principles to improve the efficiency of smoking cessation research and the effectiveness and cost-effectiveness of smoking cessation interventions.”
From The Multiphase Optimization Strategy for Engineering Effective Tobacco Use Interventions (2011) · cites this paper as [15]
Within education, coaching is learner centered and has dual goals: to achieve immediate improvement in the specific activity at hand, and to build skills for longitudinal self-monitoring and self-directed learning.
“Within education, coaching is also learner centered and has dual goals: to achieve immediate improvement in the specific activity at hand, and to build skills for longitudinal self-monitoring and selfdirected learning.19”
From The R2C2 Model in Residency Education: How Does It Foster Coaching and Promote Feedback Use? (2018) · cites this paper as [19]
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796 · see citing works
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766 · see citing works
Framework for design and evaluation of complex interventions to improve health
Campbell, M. (2000) · BMJ · doi:10.1136/bmj.321.7262.694 · Cited by 3,474 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698 · see citing works
A taxonomy of behavior change techniques used in interventions
Abraham, C., & Michie, S. (2008) · Health Psychology · doi:10.1037/0278-6133.27.3.379 · Cited by 2,525 · cited 1× · see citing works
“evaluations are reported fully and that a clear description of the intervention is provided to enable replication and synthesis of evidence (Abraham and Michie, 2008).”
Self-citationBoutron, I., Moher, D., Altman, D. G., Schulz, K. F., Ravaud, P., & for the CONSORT Group* (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-4-200802190-00008 · Cited by 2,292 · cited 1× · see citing works
“…weakness of the reporting of complex intervention studies in the past, 34 but the availability of a comprehensive range of reporting guidelines, now covering non-drug trials 35 and observational studies 36 and accessible through a single website (www.equator-network.org) should lead to improvement.”
Why we need observational studies to evaluate the effectiveness of health care
Black, N. (1996) · BMJ · doi:10.1136/bmj.312.7040.1215 · Cited by 1,598 · cited 1× · see citing works
“It is important designs preferable (Black, 1996; Glasziou et al., 2007), but”
Competing interests: none declared.
Hirsch, L. (2006) · Cited by 1,396 · see citing works
Designing and evaluating complex interventions to improve health care
Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F., Guthrie, B., Lester, H., Wilson, P., & Kinmonth, A. L. (2007) · BMJ · doi:10.1136/bmj.39108.379965.be · Cited by 1,331 · see citing works
Process evaluation in randomised controlled trials of complex interventions
Oakley, A., Strange, V., Bonell, C., Allen, E., & Stephenson, J. (2006) · BMJ · doi:10.1136/bmj.332.7538.413 · Cited by 1,247 · see citing works
Complex interventions: how “out of control” can a randomised controlled trial be?
Hawe, P., Shiell, A., & Riley, T. (2004) · BMJ · doi:10.1136/bmj.328.7455.1561 · Cited by 1,244 · see citing works
O'Neil, J. H. (1995) · North East Linguistics Society · Cited by 785 · see citing works
Interventions to change health behaviours: evidence-based or evidence-inspired?
Michie, S., & Abraham, C. (2004) · Psychology and Health · doi:10.1080/0887044031000141199 · Cited by 783 · cited 1× · see citing works
“It is therefore important to understand the whole range of effects and how they vary, for example, among recipients or between sites. A second key question in evaluating complex interventions is how the intervention works: what are the active ingredients and how are they exerting their effect? Answers to this kind of…”
Self-citationEffect of air-pollution control on death rates in Dublin, Ireland: an intervention study
Clancy, L., Goodman, P., Sinclair, H., & Dockery, D. W. (2002) · The Lancet · doi:10.1016/s0140-6736(02)11281-5 · Cited by 698 · see citing works
Complex interventions or complex systems? Implications for health economic evaluation
Shiell, A., Hawe, P., & Gold, L. H. (2008) · BMJ · doi:10.1136/bmj.39569.510521.ad · Cited by 630 · see citing works
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614 · cited 1× · see citing works
“This has been a weakness of the reporting of complex intervention studies in the past,”
When are randomised trials unnecessary? Picking signal from noise
Glasziou, P., Chalmers, I., Rawlins, M. D., & McCulloch, P. G. (2007) · BMJ · doi:10.1136/bmj.39070.527986.68 · Cited by 587 · cited 1× · see citing works
“It is important designs preferable (Black, 1996; Glasziou et al., 2007), but”
Farmer, A., Wade, A., Goyder, E., Yudkin, P., French, D., Craven, A., Holman, R., Kinmonth, A.-L., & Neil, A. (2007) · BMJ · doi:10.1136/bmj.39247.447431.be · Cited by 407 · cited 1× · see citing works
“In some evaluations, such as those seeking to identify active ingredients within a complex intervention, strict standardisation may be required and controls put in place to limit variation in implementation.”
Can it work? Does it work? Is it worth it?
Haynes, B. R. (1999) · BMJ · doi:10.1136/bmj.319.7211.652 · Cited by 402 · cited 1× · see citing works
“A key question in evaluating complex interventions is whether they are effective in everyday practice (Box 2) (Haynes, 1999).”
The impact of pesticide regulations on suicide in Sri Lanka
Gunnell, D., Fernando, R., Hewagama, M., Priyangika, W. D. D., Konradsen, F., & Eddleston, M. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym164 · Cited by 378 · cited 1× · see citing works
“of suicide (Gunnell et al., 2007), reduce air pollution (Clancy et al.”
Hardeman, W., Sutton, S., Griffin, S., Johnston, M., White, A., Wareham, N. J., & Kinmonth, A. L. (2005) · Health Education Research · doi:10.1093/her/cyh022 · Cited by 311 · cited 2× · see citing works
“causal modelling approach that included a range of primary and desk based studies to design the intervention, identify suitable measures, and predict long term outcomes (Hardeman et al., 2005).”
Eldridge, S. M., Ashby, D., Feder, G. S., Rudnicka, A. R., & Ukoumunne, O. C. (2004) · Clinical Trials · doi:10.1191/1740774504cn006rr · Cited by 288 · cited 1× · see citing works
“Evaluations are often undermined by problems of acceptability, compliance, delivery of the intervention, recruitment and retention, and smaller than expected effect sizes that could have been predicted by thorough piloting.”
MacMahon, S. W., & Collins, R. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04017-4 · Cited by 249 · cited 1× · see citing works
“(MacMahon and Collins, 2001).”
Developments in cluster randomized trials and Statistics in Medicine
Campbell, M. J., Donner, A. P., & Klar, N. S. (2006) · Statistics in Medicine · doi:10.1002/sim.2731 · Cited by 245 · see citing works
Belsky, J., Melhuish, E. C., Barnes, J., Leyland, A. H., & Romaniuk, H. (2006) · BMJ · doi:10.1136/bmj.38853.451748.2f · Cited by 215 · cited 1× · see citing works
“Complex interventions are usually described as interventions that contain several interacting components, but they have other characteristics that evaluators should take into account (box 1). There is no sharp boundary between simple and complex interventions. Few interventions are truly simple, but the number of comp…”
Haw, S. J., & Gruer, L. D. (2007) · BMJ · doi:10.1136/bmj.39315.670208.47 · Cited by 211 · cited 1× · see citing works
“, 2002), or ban smoking in public places (Haw and Gruer, 2007), tend to occur where interventions have rapid, large effects.”
Changing schools, changing health? design and implementation of the Gatehouse Project
Patton, G. C., Bond, L. M., Butler, H., & Glover, S. (2003) · Journal of Adolescent Health · doi:10.1016/s1054-139x(03)00204-0 · Cited by 140 · cited 1× · see citing works
“32 The key is to be clear about how much change or adaptation is For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe:http://www.bmj.com/subscribe”
Howden‐Chapman, P., Crane, J., Matheson, A., Viggers, H. E., Cunningham, M. J., Blakely, T. A., O’Dea, D., Cunningham, C. W., Woodward, A. J., Saville‐Smith, K., et al. (2005) · Social Science & Medicine · doi:10.1016/j.socscimed.2005.04.049 · Cited by 109 · cited 1× · see citing works
“A pilot study need not be a scale model of the planned evaluation but should examine the key uncertainties that have been identified during development. Pilot studies for a trial of free home insulation suggested that attrition might be high, so the design was amended such that participants in the control group receiv…”
Evaluating the health effects of social interventions
Thomson, H., Hoskins, R., Petticrew, M. P., Craig, N. J., Quinn, T., Lindsay, G., & Ogilvie, D. (2004) · BMJ · doi:10.1136/bmj.328.7434.282 · Cited by 89 · cited 1× · see citing works
“but there may be political or ethical objections to using them to assess health effects, especially if the intervention provides important non-health benefits.”
Self-citationEvaluating Health Effects of Transport Interventions
Ogilvie, D., Mitchell, R., Mutrie, N., Petticrew, M. P., & Platt, S. D. (2006) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2006.03.030 · Cited by 82 · cited 1× · see citing works
“Given the cost of such interventions, evaluation should still be considered—the best available methods, even if they are not optimal in terms of internal validity, may yield useful results.”
Self-citationFreemantle, N., Nazareth, I., Eccles, M., Wood, J., & Haines, A. (2002) · PubMed Central · Cited by 82 · see citing works
Self-citationEldridge, S., Spencer, A., Cryer, C., Parsons, S., Underwood, M., & Feder, G. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054338942 · Cited by 58 · see citing works
When are randomised trials unnecessary? Picking signal from noise
(2007) · BMJ · doi:10.1136/bmj.39161.547882.be · Cited by 54 · see citing works
Developing complex interventions for rigorous evaluation--a case study from rural Zimbabwe
Power, R. (2004) · Health Education Research · doi:10.1093/her/cyg073 · Cited by 52 · cited 1× · see citing works
“19 As well as illustrating the value of feasibility testing, the example shows the importance of understanding the context in which interventions take place.”
Nazareth, I., Freemantle, N., Duggan, C. A., Mason, J. M., & Haines, A. (2002) · Journal of Health Services Research & Policy · doi:10.1258/135581902320432769 · Cited by 46 · cited 1× · see citing works
“A process evaluation 28 carried out in connection with a trial of educational visits to encourage general practitioners to follow prescribing guidelines 29 found that the visits were well received and recall of the guidelines was good, yet there was little change in prescribing behaviour, which was…”
Self-citationReliable assessment of the effects of treatments on mortality and major morbidity
Collins, R., MacMahon, S. W., & Rothwell, P. M. (2007) · Cited by 38 · see citing works
Reading, R. (2006) · Child Care Health and Development · doi:10.1111/j.1365-2214.2006.00706_4.x · Cited by 21 · see citing works
Kennedy, L. (2008) · Yearbook of Endocrinology · doi:10.1016/s0084-3741(08)79141-8 · Cited by 20 · see citing works
Economic modelling before clinical trials
Torgerson, D. J., & Byford, S. (2002) · BMJ · doi:10.1136/bmj.325.7355.98 · Cited by 17 · cited 1× · see citing works
“Another useful approach is a prior economic evaluation (Torgerson and Byford, 2002).”
An exemplary evaluation of a program that worked: The high/scope Perry preschool project
Wortman, P. M. (1995) · Evaluation Practice · doi:10.1016/0886-1633(95)90039-x · Cited by 9 · see citing works
An Exemplary Evaluation of a Program that Worked: The High/Scope Perry Preschool Project
Wortman, P. M. (1995) · Evaluation Practice · doi:10.1177/109821409501600305 · Cited by 4 · cited 2× · see citing works
“Evaluation of a preschool programme for disadvantaged children showed that, as well as improved educational attainment, there was a range of economic and social benefits at ages 27 and 40.”
Let science rule: the rational way to run societies
Muir, H. (2008) · The New Scientist · doi:10.1016/s0262-4079(08)61305-6 · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 44 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [15]15. For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe 15 Academy of Medical Sciences. Identifying the environmental causes of disease: how should we decide what to believe and when to take action ? London: Academy of Medical Sciences, 2007.
Cited by
Sorted by citations
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · see citing works
“To overcome known challenges with reporting guideline uptake [36,37], we are developing a prospective implementation strategy for PRISMA-P 2015 using knowledge translation principles involving theoretically derived interventions [37] which have demonstrated effectiveness in the development of implementation interventi…”
MethodSelf-citationHoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548 · see citing works
Self-citationMichie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M. P., Cane, J., & Wood, C. E. (2013) · Annals of Behavioral Medicine · doi:10.1007/s12160-013-9486-6 · Cited by 7,629 · see citing works
Self-citationProcess evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204 · see citing works
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., et al. (2021) · BMJ · doi:10.1136/bmj.n2061 · Cited by 5,808 · see citing works
Self-citationSekhon, M., Cartwright, M., & Francis, J. J. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2031-8 · Cited by 3,938 · see citing works
Atkins, L., Francis, J., Islam, R., O’Connor, D., Patey, A., Ivers, N., Foy, R., Duncan, E. M., Colquhoun, H., Grimshaw, J. M., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0605-9 · Cited by 3,819 · see citing works
Self-citationA tutorial on pilot studies: the what, why and how
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, M., Giangregorio, L., & Goldsmith, C. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-1 · Cited by 3,384 · see citing works
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,237 · see citing works
2016 ESC/EAS Guidelines for the Management of Dyslipidaemias
Catapano, A. L., Graham, I., De Backer, G., Wiklund, O., Chapman, M. J., Drexel, H., Hoes, A. W., Jennings, C. S., Landmesser, U., Pedersen, T. R., et al. (2016) · European Heart Journal · doi:10.1093/eurheartj/ehw272 · Cited by 3,162 · see citing works
Interventions for enhancing medication adherence
Nieuwlaat, R., Wilczynski, N., Navarro, T., Hobson, N., Jeffery, R., Keepanasseril, A., Agoritsas, T., Mistry, N., Iorio, A., Jack, S., et al. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000011.pub4 · Cited by 2,945 · see citing works
Langhorne, P., Bernhardt, J., & Kwakkel, G. (2011) · The Lancet · doi:10.1016/s0140-6736(11)60325-5 · Cited by 2,726 · see citing works
Implementation strategies: recommendations for specifying and reporting
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-139 · Cited by 2,585 · see citing works
“Implementation strategies are inherently complex social interventions, as they address multifaceted and complicated processes within interpersonal, organizational, and community contexts [12,56-58].”
BackgroundEffective techniques in healthy eating and physical activity interventions: A meta-regression.
Michie, S., Abraham, C., Whittington, C., McAteer, J., & Gupta, S. (2009) · Health Psychology · doi:10.1037/a0016136 · Cited by 2,229 · see citing works
Self-citationInterprofessional education: effects on professional practice and healthcare outcomes
Reeves, S., Perrier, L., Goldman, J., Freeth, D., & Zwarenstein, M. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002213.pub3 · Cited by 1,946 · see citing works
Early palliative care for patients with advanced cancer: a cluster-randomised controlled trial
Zimmermann, C., Swami, N., Krzyzanowska, M., Hannon, B., Leighl, N., Oza, A., Moore, M., Rydall, A., Rodin, G., Tannock, I., et al. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62416-2 · Cited by 1,781 · see citing works
No surgical innovation without evaluation: the IDEAL recommendations
McCulloch, P. G., Altman, D. G., Campbell, M., Flum, D. R., Glasziou, P., Marshall, J. C., & Nicholl, J. P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61116-8 · Cited by 1,831 · see citing works
Michie, S., Ashford, S., Sniehotta, F. F., Dombrowski, S. U., Bishop, A., & French, D. P. (2011) · Psychology & Health · doi:10.1080/08870446.2010.540664 · Cited by 1,756 · see citing works
“This is not straightforward, as interventions to change health-related behaviours are usually complex, comprising many, often interacting components (Craig et al., 2008).”
BackgroundSelf-citationGuidance on how to develop complex interventions to improve health and healthcare
O"Cathain, A., Croot, L., Duncan, E., Rousseau, N., Sworn, K., Turner, K. M., Yardley, L., & Hoddinott, P. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-029954 · Cited by 1,591 · see citing works
“The UK Medical Research Council (MRC) published influential guidance on developing and evaluating complex interventions, presenting a framework of four phases: development, feasibility/piloting, evaluation and implementation.(1) The development phase is what happens between the idea for an intervention and formal pilo…”
BackgroundMethodThe dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-117 · Cited by 1,724 · see citing works
“A linear approach may be particularly challenging to apply to complex, multi-component interventions, psychosocial treatments, treatment of the growing number of people with multimorbid conditions [14,15], and systemic approaches to care [15].”
BackgroundMurray, E., Treweek, S., Pope, C., MacFarlane, A., Ballini, L., Dowrick, C., Finch, T., Kennedy, A., Mair, F., O'Donnell, C., et al. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-63 · Cited by 1,478 · see citing works
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · see citing works
“In many cases, the research and development will not move through these stages in a linear fashion but will instead go through development-evaluation-development cycles in order to make improvements to the intervention based on data gained during evaluation phases [23].”
BackgroundWhat is a pilot or feasibility study? A review of current practice and editorial policy
Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-67 · Cited by 1,483 · see citing works
2016 ESC/EAS Guidelines for the Management of Dyslipidaemias
Catapano, A. L., Graham, I., De Backer, G., Wiklund, O., Chapman, M. J., Drexel, H., Hoes, A. W., Jennings, C. S., Landmesser, U., Pedersen, T. R., et al. (2016) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2016.08.018 · Cited by 1,582 · see citing works
Eldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0150205 · Cited by 1,581 · see citing works
Standards for Reporting Implementation Studies (StaRI) Statement
Pinnock, H., Barwick, M., Carpenter, C. R., Eldridge, S., Grandes, G., Griffiths, C. J., Rycroft-Malone, J., Meissner, P., Murray, E., Patel, A., et al. (2017) · BMJ · doi:10.1136/bmj.i6795 · Cited by 1,446 · see citing works
Kwasnicka, D., Dombrowski, S. U., White, M., & Sniehotta, F. (2016) · Health Psychology Review · doi:10.1080/17437199.2016.1151372 · Cited by 1,476 · see citing works
“Theory may be useful for research to improve our understanding of maintained behaviour as well as for the design and implementation of interventions to achieve behaviour change maintenance (Craig et al., 2008).”
Psychological therapies for the management of chronic pain (excluding headache) in adults
Williams, A. C. D. C., Eccleston, C., & Morley, S. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007407.pub3 · Cited by 1,634 · see citing works
Davis, R., Campbell, R., Hildon, Z., Hobbs, L., & Michie, S. (2015) · Health Psychology Review · doi:10.1080/17437199.2014.941722 · Cited by 1,411 · see citing works
“…of theory is advocated as an integral step in intervention design and evaluation and in evidence synthesis, for example, by the UK Medical Research Council’s guidance for developing and evaluating complex interventions (Campbell et al., 2000, 2007; Craig et al., 2008; Glanz & Bishop, 2010).”
MethodSelf-citationControlled Interventions to Reduce Burnout in Physicians: A Systematic Review and Meta-analysis
Panagioti, M., Panagopoulou, E., Bower, P., Lewith, G., Kontopantelis, E., Chew-Graham, C., Dawson, S., van Marwijk, H., Geraghty, K., & Esmail, A. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.7674 · Cited by 1,395 · see citing works
From ideas to efficacy: The ORBIT model for developing behavioral treatments for chronic diseases
Czajkowski, S. M., Powell, L. H., Adler, N., Naar-King, S., Reynolds, K. D., Hunter, C. M., Laraia, B., Olster, D. H., Perna, F. M., Peterson, J. C., et al. (2015) · Health Psychology · doi:10.1037/hea0000161 · Cited by 1,161 · see citing works
Telemonitoring in patients with heart failure
Chaudhry, S. I., Mattera, J. A., Curtis, J. P., Spertus, J. A., Herrin, J., Lin, Z., Phillips, C. O., Hodshon, B. V., Cooper, L. S., & Krumholz, H. M. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa1010029 · Cited by 1,232 · see citing works
Interventions for providers to promote a patient-centred approach in clinical consultations
Dwamena, F., Holmes-Rovner, M., Gaulden, C. M., Jorgenson, S., Sadigh, G., Sikorskii, A., Lewin, S., Smith, R. C., Coffey, J., Olomu, A., et al. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003267.pub2 · Cited by 1,227 · see citing works
Theorising interventions as events in systems
Hawe, P., Shiell, A., & Riley, T. (2009) · American Journal of Community Psychology · doi:10.1007/s10464-009-9229-9 · Cited by 1,143 · see citing works
Lewin, S. A., Munabi-Babigumira, S. K., Glenton, C., Daniels, K., Bosch‐Capblanch, X., van Wyk, B. E., Odgaard‐Jensen, J., Johansen, M., Aja, G. N., Zwarenstein, M., et al. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004015.pub3 · Cited by 986 · see citing works
Studying complexity in health services research: desperately seeking an overdue paradigm shift
Greenhalgh, T., & Papoutsi, C. (2018) · BMC Medicine · doi:10.1186/s12916-018-1089-4 · Cited by 962 · see citing works
Specifying and reporting complex behaviour change interventions: the need for a scientific method
Michie, S., Fixsen, D., Grimshaw, J. M., & Eccles, M. P. (2009) · Implementation Science · doi:10.1186/1748-5908-4-40 · Cited by 976 · see citing works
“, [29]), but within-person or within-organisation research designs offer an efficient way to experimentally determine the function of individual components of evidence-based practices and programs [18,30-34].”
BackgroundMethodSelf-citationEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847 · see citing works
Craig, P., Cooper, C. C., Gunnell, D., Haw, S. J., Lawson, K. D., Macintyre, S., Ogilvie, D., Petticrew, M. P., Reeves, B. C., Sutton, M., et al. (2012) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2011-200375 · Cited by 918 · see citing works
Self-citationLoneliness and social isolation interventions for older adults: a scoping review of reviews
Fakoya, O. A., McCorry, N. K., & Donnelly, M. (2020) · BMC Public Health · doi:10.1186/s12889-020-8251-6 · Cited by 842 · see citing works
“of participants, health status, environment such as housing, and cultural characteristics) [72].”
BackgroundThe Lancet Commission on the future of care and clinical research in autism
Lord, C., Charman, T., Havdahl, A., Carbone, P., Anagnostou, E., Boyd, B., Carr, T., de Vries, P. J., Dissanayake, C., Divan, G., et al. (2022) · The Lancet · doi:10.1016/s0140-6736(21)01541-5 · Cited by 924 · see citing works
Are interventions theory-based? Development of a theory coding scheme.
Michie, S., & Prestwich, A. (2010) · Health Psychology · doi:10.1037/a0016939 · Cited by 855 · see citing works
Self-citationDe Silva, M. J., Breuer, E., Lee, L., Asher, L., Chowdhary, N., Lund, C., & Patel, V. (2014) · Trials · doi:10.1186/1745-6215-15-267 · Cited by 738 · see citing works
“Background The updated Medical Research Council (MRC) framework for complex interventions [1] is a set of guidelines for designing and evaluating complex interventions which has been widely influential in the field [2].”
BackgroundSocial prescribing: less rhetoric and more reality. A systematic review of the evidence
Bickerdike, L., Booth, A., Wilson, P. M., Farley, K., & Wright, K. E. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013384 · Cited by 783 · see citing works
Werner-Seidler, A., Perry, Y., Calear, A. L., Newby, J. M., & Christensen, H. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.10.005 · Cited by 773 · see citing works
“, randomisation at the school level) protect against this source of bias (Craig et al., 2008).”
BackgroundInterventions targeting social isolation in older people: a systematic review
Dickens, A. P., Richards, S. H., Greaves, C. J., & Campbell, J. L. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-647 · Cited by 767 · see citing works
on behalf of the GrandStand, S. S. A. T. G. O. T. M. R. G., Leask, C. F., Sandlund, M., Skelton, D. A., Altenburg, T. M., Cardon, G., Chinapaw, M. J. M., De Bourdeaudhuij, I., Verloigne, M., & Chastin, S. F. M. (2019) · Research Involvement and Engagement · doi:10.1186/s40900-018-0136-9 · Cited by 673 · see citing works
Carey, R., Bohlen, L. C., Johnston, M., Rothman, A. J., de Bruin, M., Kelly, M. P., & Michie, S. (2018) · Annals of Behavioral Medicine · doi:10.1093/abm/kay078 · Cited by 692 · see citing works
“A common thread among guidelines for intervention development and evaluation [29, 34] is the need for a strong and rigorously applied theoretical basis to optimize effectiveness and enhance our understanding of intervention effects [35].”
BackgroundSelf-citationMichie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680 · see citing works
Self-citationWhat is Comprehensive Geriatric Assessment (CGA)? An umbrella review
Parker, S. G., McCue, P., Phelps, K., McCleod, A., Arora, S., Nockels, K., Kennedy, S., Roberts, H. C., & Conroy, S. (2017) · Age and Ageing · doi:10.1093/ageing/afx166 · Cited by 646 · see citing works
“Such trials will need careful process evaluations embedded within them in line with current research frameworks for the evaluation of complex interventions [26, 27].”
BackgroundESC/EAS Guidelines for the management of dyslipidaemias
Catapano, A. L., Reiner, Ž., De Backer, G., Graham, I., Taskinen, M.-R., Wiklund, O., Agewall, S., Alegria, E., Chapman, M. J., Durrington, P., et al. (2011) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2011.06.028 · Cited by 706 · see citing works
Interventions to promote walking: systematic review
Ogilvie, D., Foster, C. E., Rothnie, H., Cavill, N., Hamilton, V., Fitzsimons, C. F., & Mutrie, N. (2007) · BMJ · doi:10.1136/bmj.39198.722720.be · Cited by 695 · see citing works
“A strategy of changes to the environment combined with advice and support at both individual and institutional levels may, therefore, be required to bring about substantial and sustained changes in travel behaviour in the population. Approaches shown to have promise in this systematic review should continue to be deve…”
Lewin, S., Glenton, C., & Oxman, A. D. (2009) · BMJ · doi:10.1136/bmj.b3496 · Cited by 678 · see citing works
“that act both “independently and inter-dependently.””
Steinsbekk, A., Rygg, L., Lisulo, M., Rise, M. B., & Fretheim, A. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-213 · Cited by 660 · see citing works
“Educational programs are frequently defined as complex interventions where it is often difficult to define the 'active ingredient' framework [7].”
BackgroundChallenges in evaluating surgical innovation
Ergina, P., Cook, J. A., Blazeby, J. M., Boutron, I., Clavien, P., Reeves, B. C., & Seiler, C. M. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61086-2 · Cited by 653 · see citing works
Treasure, J., & Schmidt, U. (2013) · Journal of Eating Disorders · doi:10.1186/2050-2974-1-13 · Cited by 646 · see citing works
“The Medical Research Council (MRC) has developed guidelines on the development of complex interventions as it is acknowledged that they entail unique challenges [7].”
Taxonomy of approaches to developing interventions to improve health: a systematic methods overview
O’Cathain, A., Croot, L., Sworn, K., Duncan, E., Rousseau, N. S., Turner, K. M., Yardley, L., & Hoddinott, P. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0425-6 · Cited by 581 · see citing works
“Theory and evidence based MRC Framework for developing and evaluating interventions [7, 15] Spending time developing interventions systematically based on evidence and theory produces interventions which have a reasonable chance of having a worthwhile effect Complex interventions in health care, public health and soci…”
BackgroundInfluentialLessons from Complex Interventions to Improve Health
Hawe, P. (2015) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031912-114421 · Cited by 607 · see citing works
Lund, C., Tomlinson, M., De Silva, M., Fekadu, A., Shidhaye, R., Jordans, M., Petersen, I., Bhana, A., Kigozi, F., Prince, M., et al. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001359 · Cited by 584 · see citing works
“Use of robust frameworks for the design and evaluation of complex interventions: The Medical Research Council framework for complex interventions [31] is the methodological basis for the development and evaluation of multi-component packages of mental health care in PRIME.”
MethodGuidelines for reporting non-randomised pilot and feasibility studies
Lancaster, G., & Thabane, L. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0499-1 · Cited by 529 · see citing works
Parasites and poverty: The case of schistosomiasis
King, C. H. (2009) · Acta Tropica · doi:10.1016/j.actatropica.2009.11.012 · Cited by 578 · see citing works
“That said, the complexity of the problem may require a more complex medical, governmental, and eco-social combined intervention (Craig et al., 2008) in order to permanently alter the status quo.”
Dineen-Griffin, S., Garcia-Cardenas, V., Williams, K., & Benrimoj, S. I. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220116 · Cited by 600 · see citing works
Grant, A., Treweek, S., Dreischulte, T., Foy, R., & Guthrie, B. (2013) · Trials · doi:10.1186/1745-6215-14-15 · Cited by 541 · see citing works
“terature on process evaluation and the RE-AIM framework [3,11,19,23,26,27,31], and on published process evaluations”
BackgroundMethodBonell, C., Fletcher, A., Morton, M., Lorenc, T., & Moore, L. (2012) · Social Science & Medicine · doi:10.1016/j.socscimed.2012.08.032 · Cited by 582 · see citing works
Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., & Shea, C. M. (2017) · Implementation Science · doi:10.1186/s13012-017-0657-x · Cited by 544 · see citing works
“They also could advance understanding of the mechanisms through which implementation strategies influence EBI adoption, implementation, scale-up, and sustainment by facilitating the alignment of strategies to theories that explain how they work [46].”
BackgroundSekhon, M., Cartwright, M., & Francis, J. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-07577-3 · Cited by 564 · see citing works
Gillison, F., Rouse, P. C., Standage, M., Sebire, S. J., & Ryan, R. M. (2018) · Health Psychology Review · doi:10.1080/17437199.2018.1534071 · Cited by 544 · see citing works
“Theory also helps to ensure that a systematic and comprehensive set of determinants are addressed linking to evidence (Michie et al., 2016), and is thus endorsed as part of best practice in intervention design (Craig et al., 2008; Moore et al., 2015).”
BackgroundLau, R., Stevenson, F., Ong, B. N., Dziedzic, K., Treweek, S., Eldridge, S., Everitt, H., Kennedy, A., Qureshi, N., Rogers, A., et al. (2015) · Implementation Science · doi:10.1186/s13012-016-0396-4 · Cited by 533 · see citing works
“components [11], and complex interventions can be particularly hard to implement, as they are likely to require change at multiple levels.”
BackgroundAvery, K. N. L., Williamson, P. R., Gamble, C., O'Connell Francischetto, E., Metcalfe, C., Davidson, P., Williams, H., & Blazeby, J. M. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013537 · Cited by 575 · see citing works
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 535 · see citing works
Self-citationGriffin, D. R., Dickenson, E. J., Wall, P. D. H., Achana, F. A., Donovan, J., Griffin, J. M., Hobson, R., Hutchinson, C. E., Jepson, M., Parsons, N., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31202-9 · Cited by 565 · see citing works
Theories and techniques of behaviour change: Developing a cumulative science of behaviour change
Michie, S., & Johnston, M. (2012) · Health Psychology Review · doi:10.1080/17437199.2012.654964 · Cited by 522 · see citing works
“Key frameworks for designing and evaluating behaviour change interventions (Collins et al., 2011; Craig et al., 2008) emphasise the importance of using theory to inform intervention design as well as specifying interventions in terms of component BCTs.”
BackgroundSelf-citationGardner, B., Smith, L., Lorencatto, F., Hamer, M., & Biddle, S. (2015) · Health Psychology Review · doi:10.1080/17437199.2015.1082146 · Cited by 518 · see citing works
“Syntheses of previous intervention trials can provide a valuable input into the design of new interventions, by revealing which approaches, techniques, and assumptions show promise in reducing sedentary behaviour, and the strength of the evidence (Craig et al., 2008; Michie, Atkins, & West, 2014).”
Pollock, A., & Berge, E. (2017) · International Journal of Stroke · doi:10.1177/1747493017743796 · Cited by 538 · see citing works
Mead, E., Brown, T. J., Rees, K., Azevedo, L. B., Whittaker, V., Jones, D., Olajide, J., Mainardi, G. M., Corpeleijn, E., O’Malley, C. L., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012651 · Cited by 524 · see citing works
Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness
Brehaut, J., Colquhoun, H. L., Eva, K. W., Carroll, K. A., Sales, A. E., Michie, S., Ivers, N. M., & Grimshaw, J. (2016) · Annals of Internal Medicine · doi:10.7326/m15-2248 · Cited by 489 · see citing works
Self-citationDuncan, E., O'Cathain, A., Rousseau, N., Croot, L., Sworn, K., Turner, K. M., Yardley, L., & Hoddinott, P. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033516 · Cited by 481 · see citing works
Lorenc, T., Clayton, S., Neary, D., Whitehead, M. M., Petticrew, M. P., Thomson, H., Cummins, S., Sowden, A. J., & Renton, A. M. (2012) · Health & Place · doi:10.1016/j.healthplace.2012.04.001 · Cited by 528 · see citing works
“The value of theory-led approaches, particularly those that generate causal pathway frameworks, is now widely acknowledged (Anderson et al., 2011; Craig et al., 2008).”
BackgroundSelf-citationAthletes' career transition out of sport: a systematic review
Park, S., Lavallee, D. E., & Tod, D. (2012) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2012.687053 · Cited by 530 · see citing works
“Craig et al. (2008) highlighted the benefits of conducting systematic reviews in developing interventions and designing future studies.”
BackgroundMichie, S., Carey, R. N., Johnston, M., Rothman, A. J., de Bruin, M., Kelly, M. P., & Connell, L. E. (2018) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9816-6 · Cited by 475 · see citing works
“Preliminary work to address this has been conducted in both primary research and in evidence syntheses [9, 21, 26, 60, 74, 75].”
InfluentialSelf-citationCervero, R. M., & Gaines, J. K. (2015) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.21290 · Cited by 515 · see citing works
Guidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475 · see citing works
A systematic review and meta-analysis of psychological interventions to improve mental wellbeing
van Agteren, J., Iasiello, M., Lo, L., Bartholomaeus, J., Kopsaftis, Z., Carey, M., & Kyrios, M. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01093-w · Cited by 510 · see citing works
An Overview of Research and Evaluation Designs for Dissemination and Implementation
Brown, C. H., Curran, G., Palinkas, L. A., Aarons, G. A., Wells, K. B., Jones, L., Collins, L. M., Duan, N., Mittman, B. S., Wallace, A., et al. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044215 · Cited by 509 · see citing works
“Practical modifications of the individual-level randomized clinical trial have been made to test for program or intervention effectiveness under realistic conditions in randomized field trials (40), as well as for interventions delivered at the group level (77), multilevel interventions (14, 15), complex (34, 70) or m…”
BackgroundMulti-disciplinary rehabilitation for acquired brain injury in adults of working age
Turner‐Stokes, L. F., Pick, A., Nair, A., Disler, P. B., & Wade, D. T. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004170.pub3 · Cited by 503 · see citing works
Francis, J. J., O’Connor, D., & Curran, J. (2012) · Implementation Science · doi:10.1186/1748-5908-7-35 · Cited by 484 · see citing works
O’Cathain, A., Hoddinott, P., Lewin, S., Thomas, K. J., Young, B., Adamson, J., Jansen, Y. J., Mills, N., Moore, G., & Donovan, J. L. (2015) · Pilot and Feasibility Studies · doi:10.1186/s40814-015-0026-y · Cited by 458 · see citing works
“The United Kingdom Medical Research Council (UK MRC) guidance on the development and evaluation of complex interventions recommends an early phase of assessing feasibility prior to a full evaluation [1].”
InfluentialFancourt, D., Aughterson, H., Finn, S., Walker, E., & Steptoe, A. (2021) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30384-9 · Cited by 484 · see citing works
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486 · see citing works
Michie, S., Hyder, N., Walia, A., & West, R. (2011) · Addictive Behaviors · doi:10.1016/j.addbeh.2010.11.016 · Cited by 459 · see citing works
Self-citationAhmadi, A., Noetel, M., Parker, P. D., Ryan, R. M., Ntoumanis, N., Reeve, J., Beauchamp, M. R., Dicke, T., Yeung, A. S., Ahmadi, M., et al. (2023) · Journal of Educational Psychology · doi:10.1037/edu0000783 · Cited by 555 · see citing works
Embodied conversational agents in clinical psychology: a scoping review
Provoost, S., Lau, H. M., Ruwaard, J., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6553 · Cited by 433 · see citing works
“The UK Medical Research Council’s (MRC) framework for complex interventions [11] defines four phases through which such interventions move before being fully embedded in practice: development, piloting, evaluation, and implementation.”
Moore, G. F., Evans, R. E., Hawkins, J., Littlecott, H., Melendez-Torres, G. J., Bonell, C., & Murphy, S. (2019) · Evaluation · doi:10.1177/1356389018803219 · Cited by 471 · see citing works
“First, it acknowledged that RCTs are often infeasible (Craig et al., 2008a), with guidance on the use of natural experiments following shortly after (Craig et al., 2012).”
BackgroundInfluentialKhunti, K. K., Gray, L. J., Skinner, T. C., Carey, M., Realf, K., Dallosso, H. M., Fisher, H., Campbell, M. J., Heller, S. R., & Davies, M. J. (2012) · BMJ · doi:10.1136/bmj.e2333 · Cited by 465 · see citing works
Experiences of mental illness stigma, prejudice and discrimination: a review of measures
Brohan, E., Slade, M., Clement, S., & Thornicroft, G. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-80 · Cited by 454 · see citing works
“This contributes evidence to support the evaluation of outcomes as part of anti-stigma campaigns or social inclusion interventions, fitting with the Medical Research Council’s guidance on developing and evaluating complex interventions [41].”
Early Active Mobilization during Mechanical Ventilation in the ICU
The TEAM Study Investigators and the ANZICS Clinical Trials Group (2022) · New England Journal of Medicine · doi:10.1056/nejmoa2209083 · Cited by 453 · see citing works
Sanders, C., Rogers, A., Bowen, R., Bower, P., Hirani, S., Cartwright, M., Fitzpatrick, R., Knapp, M., Barlow, J., Hendy, J., et al. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-220 · Cited by 450 · see citing works
A guiding framework and approach for implementation research in substance use disorders treatment.
Damschroder, L. J., & Hagedorn, H. J. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022284 · Cited by 427 · see citing works
Internet-based medical education: a realist review of what works, for whom and in what circumstances
Wong, G., Greenhalgh, T., & Pawson, R. (2010) · BMC Medical Education · doi:10.1186/1472-6920-10-12 · Cited by 443 · see citing works
“) that interact in a non-linear fashion to produce outcomes which are highly context dependent [9-11].”
BackgroundAction, actor, context, target, time (AACTT): a framework for specifying behaviour
Presseau, J., McCleary, N., Lorencatto, F., Patey, A. M., Grimshaw, J. M., & Francis, J. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0951-x · Cited by 411 · see citing works
- Authors
- Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M.
- Affiliations
- Medical Research Council (GB)
- Nuffield Orthopaedic Centre (GB)
- MRC/CSO Social and Public Health Sciences Unit (GB)
- University College London (GB)
- London School of Hygiene & Tropical Medicine (GB)
- Corresponding author
- Craig, Peter
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2008-09-29
- Volume
- 337
- Issue
- 5
- Pages
- a1655
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC2769032
- OpenAlex
- W2162544110
- Cited by
- 10,995
- References
- 44
- Citations by year
- 2026: 284 · 2025: 437 · 2024: 626 · 2023: 693 · 2022: 950 · 2021: 1178 · 2020: 1109 · 2019: 902 · 2018: 868 · 2017: 792 · 2016: 666 · 2015: 622 · 2014: 530 · 2013: 433 · 2012: 341
- FWCI
- 352.22
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- BMJ (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Figshare (publishedVersion)
- Open Research Exeter (University of Exeter) (publishedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- complex interventions, intervention development, MRC framework, evaluation framework, research guidance
- MeSH terms
- Diffusion of Innovation, Evaluation Studies as Topic, Therapeutics, Practice Guidelines as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-01.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-01.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
