The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index; 1 of the 40 sources its claims cite were checked for support.
Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research
Open accessAbstract
Population health interventions are essential to reduce health inequalities and tackle other public health priorities, but they are not always amenable to experimental manipulation. Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances. One key challenge in evaluating NEs is selective exposure to the intervention. Studies should be based on a clear theoretical understanding of the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations. Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation. NE studies often rely on existing (including routinely collected) data. Investment in such data sources and the infrastructure for linking exposure and outcome data is essential if the potential for such studies to inform decision making is to be realized.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Natural experiments (NEs) use naturally occurring variation in exposure to evaluate population health interventions that cannot be randomized, but selective exposure is a central threat to causal inference. The paper reviews methods for NE evaluation and argues that inference is strengthened by clear theory, additional design features, and investment in linked routinely collected data.
Key findings
- NE studies are defined broadly as any event not under researcher control that divides a population into exposed and unexposed groups; because exposure is often selective, studies should model the assignment process and consider alternative explanations even when effects are large and rapid.
- The paper reviews regression adjustment, propensity score methods, difference-in-differences (DiD), interrupted time series (ITS), synthetic controls, instrumental variables (IVs), and regression discontinuity (RD), noting that combining methods, replication, and falsification tests can strengthen causal inference.
- Illustrative examples include a 1995 complete pesticide import ban in Sri Lanka associated with a 50% fall in suicide by self-poisoning; Oregon’s 2003 traffic police layoff of more than one-third of the force associated with a 12–14% increase in fatalities; and Scotland’s ban on multipurchase alcohol discounts associated with a 2% decrease in overall sales after adjustment for England and Wales.
- The paper concludes that continued investment in routinely collected data, population health surveys, and data-linkage infrastructure is essential for NEs to inform policy making.
Methods
- Design: narrative overview/review of methods, approaches, and contributions of natural experiments to public health intervention research, using the potential outcomes framework to compare RCTs, NEs, and nonexperimental observational studies.
- Data or sample: no primary study sample; the paper draws on methodological literature and published NE examples, including Sri Lanka’s pesticide ban, Oregon traffic policing, Scotland’s alcohol legislation, California’s tobacco control program, US Head Start, and Mexico’s Oportunidades program.
- Main analysis: conceptual and methodological synthesis of effect-estimation approaches, their assumptions, strengths, limitations, and strategies for strengthening inference; no new quantitative analysis is reported.
Limitations
- The paper states that NEs are not the answer to every evaluation question, that it is not always possible to conduct a good NE when an RCT would be impractical, and that method choice is constrained by intervention features, allocation process, comparators, expected impacts, and data availability.
- The paper states method-specific limitations, including selective exposure, DiD’s common trends assumption, ITS power and autocorrelation concerns, synthetic controls’ lack of conventional statistical inference and donor-pool matching constraints, IVs’ difficulty in finding and justifying valid instruments, and RD’s local effects and bias–precision trade-off.
- Evident from the text: this is a narrative overview without a described systematic search strategy or formal quality appraisal; the cited examples are illustrative rather than a comprehensive evidence synthesis.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
47 Own interpretation · 29 Not checked yet · 1 Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Truly random assignment, although not unknown (14), is extremely rare in policy and practice settings.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Introduction
What the sources say
- Supportedcites [14] source
The citation is attached to “although not unknown,” so this paper serves as an example of random assignment in a policy setting, not as evidence that it is extremely rare. Section 2.2 (p. 5) describes and verifies the random allocation of village councils to reservation for a woman leader. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“In an observational study, knowledge of the assignment mechanism can be used to make the groups conditionally exchangeable, for example, by controlling for variables that influence both assignment and outcomes to the extent that these variables are known and accurately measured (34).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [34] Hernán M, 2017
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interventions that are introduced abruptly, that affect large populations, and that are implemented where it is difficult for individuals to manipulate their treatment status are more straightforward to evaluate.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The potential outcomes model provides a useful framework for clarifying similarities and differences between true experiments on the one hand and observational studies (including NEs) on the other hand (51).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [51] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Extrapolation may also be a problem for some NE studies, such as those using RD designs, which estimate treatment effects at a particular value of a variable used to determine assignment.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Likewise, effects that are large and follow rapidly after implementation are more readily detectable than more subtle or delayed effects.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“After the complete ban, rates of suicide by self-poisoning fell by 50%.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
What the sources say
- Not checked yetcites [32] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Standard multivariable models, which control for observed differences between intervention and control groups, can be used to evaluate NEs when no important differences in unmeasured characteristics between intervention and control groups are expected”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Adjustment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The key strength of the study by Goodman et al. is the way the timing of data gathering in relation to exposure created well-balanced intervention and control groups.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Adjustment
What the sources say
- Not checked yetcites [29] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A common shortcoming in ITS analyses is a lack of statistical power (61).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
What the sources say
- Not checked yetcites [61] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 26
In the paper
“In an observational study, where exchangeability may be achieved by a method such as matching or by conditioning on covariates, intervention groups may be created whose members in practice have no chance of receiving the treatment (55). The meaning of treatment effects estimated in this way is unclear.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [55] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key considerations when choosing an NE evaluation method are the source of variation in exposure and the size and nature of the expected effects.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“this framework indicates why an understanding of the assignment process is so important to the design of an NE study.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NE studies often rely on existing (including routinely collected) data.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Some methods, such as fuzzy regression discontinuity (RD) and instrumental variables (IVs), estimate a different effect, the complier average causal effect (CACE), which is the effect of the intervention on those who comply with their allocation into the control or intervention group (11). Under certain assumptions, the CACE is equivalent to the ITT effect divided by the proportion of compliers.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [11] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If assignment is random, the groups are said to be exchangeable and the intervention’s average causal effect can be estimated from the difference in the average outcomes for the two groups.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A comparison of outcomes between groups assigned to the intervention or control condition provides an estimate of the effect of assignment, known as the intention-to-treat (ITT) effect, rather than the effect of the intervention itself. The two are necessarily the same only if there is perfect compliance.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As-if randomness lacks a precise definition”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In a well-conducted RCT, randomization ensures exchangeability.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The methods discussed in the next section can be seen as different ways of achieving conditional exchangeability.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the methods proposed to identify as-if random processes (such as a good understanding of the assignment process and checks on the balance of covariates between exposed and unexposed groups) are those used to assess threats to validity in any study that attempts to make causal inferences from observational data.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The method relies on the assumption that, in the absence of the intervention, preimplementation trends would continue.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unless autocorrelation is addressed, the standard errors will be underestimated, but models that allow for autocorrelation can be fitted using standard statistical packages.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Although in statistical terms the use of propensity scores may produce results that differ little from those obtained through traditional regression adjustment (70), they encourage clearer thinking about study design and particularly the assignment mechanism (66).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In contrast, this study found no difference, suggesting that the earlier findings reflected the selective provision of training.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Adjustment
What the sources say
- Not checked yetcites [29] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DiD’s strength is that it controls for unobserved as well as observed differences in the fixed (i.e., time-invariant) characteristics of the groups and is therefore less prone to omitted variable bias caused by unmeasured confounders or measurement error.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Understanding the mechanism underlying allocation to intervention and control groups is key when deriving the propensity score.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“There are four principal ways to use the propensity score to obtain an estimated treatment effect: matching, stratification, inverse probability weighting, and covariate adjustment (7). Each method will adjust for differences in characteristics of the intervention and control groups and, in so doing, minimize the effects of confounding.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
What the sources say
- Not checked yetcites [7] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Recent developments, such as the use of propensity scores, rather than traditional covariate adjustment, to account for group-specific time-varying characteristics, add additional complexity, but combining DiD with other approaches in this way may further strengthen causal inference.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Another limitation is that conventional methods of statistical inference cannot be applied, although Abadie et al. (1) suggest an alternative that compares the estimated effect for the intervention area with the distribution of placebo effects derived by comparing each area in the donor pool with its own synthetic control.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Synthetic Controls
What the sources say
- Not checked yetcites [1] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With data for multiple preimplementation time points, the common trends assumption can be investigated directly, and it can be relaxed by extending the model to include terms for group-specific trends.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In the absence of random allocation, the propensity score attempts to recreate the allocation mechanism, defined as the conditional probability of an individual being in the intervention group, given a number of covariates (65).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
What the sources say
- Not checked yetcites [65] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The synthetic control method can be seen as an extension of the DiD method, with a number of advantages. In particular, it relaxes the requirement for a geographical control that satisfies the parallel trends assumption and relies less on subjective choices of control areas.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Synthetic Controls
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Controlled ITS designs, which compare trends in exposed and unexposed groups or in outcomes that are not expected to change as a result of the intervention, can be used to strengthen causal inference still further; in addition, standardization can be used to control for changes in population composition.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By accounting for preintervention trends, well-conducted ITS studies permit stronger causal inference than do cross-sectional or simple prepost designs, but they may be subject to confounding by cointerventions or changes in population composition.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even when such protocols are published, they do not provide a perfect safeguard against selective reporting of positive findings.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Causal inference can be strengthened in NE studies by the inclusion of additional design features alongside the principal method of effect estimation.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Good-quality natural experimental studies typically use a range of approaches.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NE approaches to evaluation have become topical because they address researchers’ and policy makers’ interests in understanding the impact of large-scale population health interventions that, for practical, ethical, or political reasons, cannot be manipulated experimentally.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This restriction to a subset of the population may not matter if the intervention is expected to affect outcomes locally, as in the case of a minimum legal drinking age or if the substantive focus of the study is on the effect of a small change in the assignment rule. It is more serious when the outcome of interest is the effect on the whole population.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Discontinuity
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“IVs are widely used in econometric program evaluation and have attracted much recent interest in epidemiology, particularly in the context of Mendelian randomization studies, which use genetic variants as instruments for environmental exposures (25, 36, 48).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Instrumental Variables
What the sources say
- Not checked yetcites [25] source
- Not checked yetcites [36] 36. Holmes MV, Dale CE, Zuccolo L, Silverwood RJ, Guo Y, et al. 2014. Association between alcohol and cardiovascular disease: Mendelian randomisation analysis based on individual participant data. BMJ 349:g4164
- Not checked yetcites [48] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Reporting of NE studies of all kinds may also be improved by following established reporting guidelines such as STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) (74) or TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) (28).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Replication studies, which by definition retest a previously published hypothesis, are a valuable additional safeguard against retrospectively fitting hypotheses to known features of the data.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Most natural experimental studies are conducted retrospectively, using data collected before the study is planned.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Availability of data also constrains the choice of methods.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Such comparisons are particularly useful in early applications of novel methods whose strengths and weaknesses are not fully understood (49).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
What the sources say
- Not checked yetcites [49] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NEs are not the answer to every evaluation question, and it is not always possible to conduct a good NE study whenever an RCT would be impractical.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“RD methods are widely regarded as the closest approximation of an observational study to an RCT (5), but their real value derives from their wide applicability to the evaluation of social programs for which eligibility is determined by a score on some form of continuous scale and also from their reliance on relatively weak, directly testable assumptions.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Discontinuity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To work in this way, an IV must be associated with exposure to the intervention, must have no association with any other factors associated with exposure, and must be associated with outcomes only through its association with exposure to the intervention (Figure 1).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Instrumental Variables
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“One key shortcoming is that restricting the bandwidth to reduce bias results in a loss of precision (46, 73), and estimates that may hold over only a small segment of the whole population exposed to the intervention.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Discontinuity
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Smaller numbers of observations yield less precise estimates.
None of the sources cited for this claim have been checked yet.
In the paper
“Recent investments in infrastructure to link data across policy sectors for research purposes are a welcome move that should increase opportunities to evaluate NEs (12, 26, 37).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
What the sources say
- Not checked yetcites [12] source
- Not checked yetcites [26] Farr Inst. Health Inf. Res., 2016
- Not checked yetcites [37] House Commons Sci. Technol. Comm, 2016
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Qualitative research can strengthen the design of RCTs of complex public health interventions (10, 56), and this argument applies equally to NEs (38).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
What the sources say
- Not checked yetcites [10] source
- Not checked yetcites [56] source
- Not checked yetcites [38] 38. Humphreys DK, Panter J, Sahlqvist S, Goodman A, Ogilvie D. 2016. Changing the environment to improve population health: a framework for considering exposure in natural experimental studies. J. Epidemiol. Community Health. doi: 10.1136/jech-2015-206381 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“It can also improve understanding of the processes that determine exposure, factors associated with intervention delivery and compliance, mechanisms by which outcomes are realized, and the strengths and limitations of routinely collected measures of exposures and outcomes (13).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
What the sources say
- Not checked yetcites [13] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if the observed effects are large and rapidly follow implementation, confidence in attributing them to the intervention can be markedly improved by a detailed consideration of alternative explanations.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
Age, income, and other continuous variables are often used to determine entitlement to social programs such as means-tested welfare benefits.
“Age, income, and other continuous variables are often used to determine entitlement to social programs, such as means-tested welfare benefits.”
The paper relies on this without support.
Smaller numbers of observations yield less precise estimates.
“One drawback is that smaller numbers of observations will yield less precise estimates, so the choice involves a trade-off between bias and precision.”
The paper relies on this without support.
Most natural experimental studies are conducted retrospectively, using data collected before the study is planned.
“Most NE studies are conducted retrospectively, using data collected before the study is planned.”
The paper relies on this without support.
Good-quality natural experimental studies typically use a range of approaches.
“Good-quality NE studies typically use a range of approaches.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“noting only that what they call ‘qualitative’ techniques (the cited references for these are all QCA studies) require “further studies … to establish their validity and usefulness” [2].”
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.008 · Cited by 17,893
(2002) · Cited by 13,211
Austin, P. C. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.568786 · Cited by 12,242
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204
Abadie, A., Diamond, A., & Hainmueller, J. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08746 · Cited by 5,513
Regression discontinuity designs in economics
Lee, D. S., & Lemieux, T. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.281 · Cited by 4,534
The economic costs of conflict: a case study of the Basque Country
Abadie, A., & Gardeazabal, J. (2003) · American Economic Review · doi:10.1257/000282803321455188 · Cited by 4,517
Segmented regression analysis of interrupted time series studies in medication use research
Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross-Degnan, D. (2002) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1046/j.1365-2710.2002.00430.x · Cited by 3,279 · cited 2×
“Lags in the effect of the intervention can be accounted for by omitting outcome values that occur during the lag period or by modeling the lag period as a separate segment (75).”
BackgroundMethod
Research works printed but not linked: 4
- Acad. Med. Sci., 2007, Identifying the Environmental Causes of Disease: How Should We Decide What to Believe and When to Take Action.
- Farr Inst. Health Inf. Res., 2016, Environmental and public health research.
- Hernán M, 2017, Causal Inference
- House Commons Sci. Technol. Comm, 2016, The Big Data Dilemma. Fourth Report of Session 2015–16.
Cited by
Sorted by citations
Items 101–200 of 570
Organ preservation after neoadjuvant long-course chemoradiotherapy versus short-course radiotherapy
Bercz, A., Park, B. K., Pappou, E. P., Nemirovsky, D., Sarkar, R. R., Yamner, M., Omer, D. M. R., Verheij, F. S., Alvarez, J., Atri, P., et al. (2024) · Annals of Oncology · doi:10.1016/j.annonc.2024.07.729 · Cited by 35
Lawes, M., Hetschko, C., Schöb, R., Stephan, G., & Eid, M. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000417 · Cited by 37
Kessler, M., Thumé, E., Marmot, M. G., Macinko, J. A., Facchini, L. A., Nedel, F. B., Wachs, L. S., Völz, P. M., & de Oliveira, C. M. (2021) · American Journal of Public Health · doi:10.2105/ajph.2020.306146 · Cited by 34
Katikireddi, S. V., Niedzwiedz, C. L., Dundas, R., Kondo, N., Leyland, A. H., & Rostila, M. (2020) · International Journal of Epidemiology · doi:10.1093/ije/dyaa053 · Cited by 34
Self-citationGrabovac, I., Hochfellner, L., Rieger, M., Jewell, J. M., Snell, A., Weber, A., Stüger, H., Schindler, K., Mikkelsen, B., & Dorner, T. E. (2018) · European Journal of Public Health · doi:10.1093/eurpub/cky147 · Cited by 33
Gregg, E. W., Patorno, E., Karter, A. J., Mehta, R., Huang, E. S., White, M. J., Patel, C. J., McElvaine, A. T., Cefalu, W. T., Selby, J. V., et al. (2023) · Diabetes Care · doi:10.2337/dc22-1438 · Cited by 34
Social norms and obesity prevalence: From cohort to system dynamics models
Crielaard, L., Dutta, P., Quax, R., Nicolaou, M., Merabet, N., Stronks, K., & Sloot, P. M. A. (2020) · Obesity Reviews · doi:10.1111/obr.13044 · Cited by 34
Changing Behavior Using the Health Belief Model and Protection Motivation Theory
Orbell, S., Zahid, H., & Henderson, C. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.004 · Cited by 35
Wang, W., & He, B. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2022.104387 · Cited by 32
Changing Behavior Using the Transtheoretical Model
DiClemente, C. C., & Graydon, M. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.010 · Cited by 35
The effects of frequency, duration, and intensity on L2 learning through Duolingo
Sudina, E., & Plonsky, L. (2023) · Journal of Second Language Studies · doi:10.1075/jsls.00021.plo · Cited by 33
Godard‐Sebillotte, C., Karunananthan, S., & Vedel, I. (2019) · Family Practice · doi:10.1093/fampra/cmz003 · Cited by 31
Benton, J. S., Cotterill, S., Anderson, J., Macintyre, V. G., Gittins, M., Dennis, M., Lindley, S. J., & French, D. (2021) · Wellbeing Space and Society · doi:10.1016/j.wss.2021.100029 · Cited by 33
Expert review: prevention of obstetrical venous thromboembolism
Friedman, A. M., & D’Alton, M. E. (2021) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2021.05.004 · Cited by 31
Dental Care Use, Edentulism, and Systemic Health among Older Adults
Meyerhoefer, C. D., Pepper, J. V., Manski, R. J., & Moeller, J. F. (2021) · Journal of Dental Research · doi:10.1177/00220345211019018 · Cited by 31
How should we evaluate sweetened beverage tax policies? A review of worldwide experience
Ng, S. W., Colchero, M. A., & White, M. J. (2021) · BMC Public Health · doi:10.1186/s12889-021-11984-2 · Cited by 36
de Vocht, F., McQuire, C., Brennan, A., Egan, M., Angus, C., Kaner, E., Beard, E., Brown, J., Angelis, D. D., Carter, N., et al. (2020) · Addiction · doi:10.1111/add.15002 · Cited by 31
Income trajectories and self-rated health status in the UK
Akanni, L., Lenhart, O., & Morton, A. (2022) · SSM - Population Health · doi:10.1016/j.ssmph.2022.101035 · Cited by 30
Moffatt, S. M., Wildman, J., Pollard, T. M., Penn, L., O’Brien, N., Pearce, M. S., & Wildman, J. M. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-026826 · Cited by 30
A Future for Observational Epidemiology: Clarity, Credibility, Transparency
Harper, S. B. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwy280 · Cited by 31
Zhang, C., Zhang, R., Gan, Y., Li, D., & Rhodes, R. E. (2019) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2019.06.003 · Cited by 30
Laverty, A. A., Hone, T., Vamos, E. P., Anyanwu, P. E., Taylor-Robinson, D. C., de Vocht, F., Millett, C. J., & Hopkinson, N. S. (2020) · Thorax · doi:10.1136/thoraxjnl-2019-213998 · Cited by 30
Boon, M. H., Burns, J., Craig, P., Griebler, U., Heise, T. L., Katikireddi, S. V., Rehfuess, E. A., Shepperd, S., Thomson, H., & Bero, L. A. (2022) · American Journal of Public Health · doi:10.2105/ajph.2021.306658 · Cited by 30
Self-citationVenkatesh, V., Brown, S. A., & Sullivan, Y. W. (2023) · Virginia Tech Publishing eBooks · doi:10.21061/conducting-mixed-methods-research · Cited by 34
Pozo‐Rosich, P., van Veelen, N., Caronna, E., Vaghi, G., Torres‐Ferrús, M., van der Arend, B. W. H., Goadsby, P. J., Ashina, M., Wang, S., Diener, H., et al. (2025) · Cephalalgia · doi:10.1177/03331024251318016 · Cited by 29
Ross‐Degnan, D., Wallace, J., Zhang, F., Soumerai, S. B., Garabedian, L. F., & Wharam, J. F. (2020) · Medical Care · doi:10.1097/mlr.0000000000001295 · Cited by 28
Changing Behavior Using the Reflective-Impulsive Model
Deutsch, R., & Strack, F. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.012 · Cited by 30
Birds of a feather flock together: Comparing controlled pre–post designs
Fry, C. E., & Hatfield, L. A. (2021) · Health Services Research · doi:10.1111/1475-6773.13697 · Cited by 28
Changing Behavior by Changing Environments
Marteau, T. M., Fletcher, P. C., Hollands, G. J., & Munafo, M. R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.014 · Cited by 32
Public health benefits from urban horticulture in the global north: A scoping review and framework
Cruz-Piedrahita, C., Howe, C., & de Nazelle, A. J. (2020) · Global Transitions · doi:10.1016/j.glt.2020.10.001 · Cited by 27
Donovan, L. M., Hoyos, C. M., Kimoff, R. J., Morrell, M. J., Bosch, N. A., Chooljian, D. M., McEvoy, R. D., Sawyer, A. M., Wagner, T. H., Al-Lamee, R. R., et al. (2023) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202303-258st · Cited by 28
Niedzwiedz, C. L., Katikireddi, S. V., Reeves, A., McKee, M., & Stückler, D. (2017) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2017-209105 · Cited by 27
Self-citationDeveloping Behavior Change Interventions
Hankonen, N. E., & Hardeman, W. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.021 · Cited by 28
Sheridan, P. E., McElroy, S., Casey, J. A., & Benmarhnia, T. (2022) · Epidemiology · doi:10.1097/ede.0000000000001539 · Cited by 26
Context, health and migration: a systematic review of natural experiments
Biddle, L., Hintermeier, M., Costa, D., Wasko, Z., & Bozorgmehr, K. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102206 · Cited by 28
Design, Implementation, and Evaluation of Behavior Change Interventions: A Ten-Task Guide
Abraham, C., & Denford, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.019 · Cited by 27
Evaluation of the impact of a public bicycle share program on population bicycling in Vancouver, BC
Hosford, K., Fuller, D., Lear, S. A., Teschke, K., Gauvin, L. I., Brauer, M. D., & Winters, M. (2018) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2018.09.014 · Cited by 25
Anderson, J., Benton, J. S., Ye, J., Barker, E. A., Macintyre, V. G., Wilkinson, J., Rothwell, J. J., Dennis, M., & French, D. (2024) · Environment International · doi:10.1016/j.envint.2024.108669 · Cited by 25
Bradford, O. B. O. B. S., Dickerson, J., Hub, T. B. S. B. I., Bird, P. K., Bryant, M., Dharni, N., Bridges, S., Willan, K., Ahern, S., Dunn, A., et al. (2019) · BMC Public Health · doi:10.1186/s12889-019-6554-2 · Cited by 25
Anyanwu, P. E., Craig, P., Katikireddi, S. V., & Green, M. J. (2020) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaa101 · Cited by 25
Self-citationRothman, A. J., Klein, W. M. P., & Sheeran, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.020 · Cited by 26
Miraldo, M., Propper, C., & Williams, R. I. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.09.028 · Cited by 24
Druetz, T., Cooper, S., Bicaba, F., Bila, A., Shareck, M., Milot, D.-M., Tiendrebeogo, C. O., & Bicaba, A. (2022) · PLOS Global Public Health · doi:10.1371/journal.pgph.0000174 · Cited by 24
Dyadic Behavior Change Interventions
Scholz, U., Berli, C., Lüscher, J., & Knoll, N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.043 · Cited by 26
Eun, S. J. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.106455 · Cited by 23
Vita‐Barrull, N., Guzmán, N., Estrada‐Plana, V., March‐Llanes, J., Mayoral, M., & Moya-Higueras, J. (2021) · Games for Health Journal · doi:10.1089/g4h.2021.0034 · Cited by 26
Rodríguez, M. P., Friis, C., Andriatsitohaina, R. N. N., Boillat, S., Roig-Boixeda, P., Cortinovis, C., Geneletti, D., Ibarrola‐Rivas, M., Kelley, L. C., Llopis, J. C., et al. (2023) · Landscape Ecology · doi:10.1007/s10980-023-01604-2 · Cited by 24
Craig, P., Gibson, M., Campbell, M., Popham, F., & Katikireddi, S. V. (2017) · Preventive Medicine · doi:10.1016/j.ypmed.2017.12.025 · Cited by 23
Self-citationChanging Behavior Using the Common-Sense Model of Self-Regulation
Cameron, L. D., Fleszar‐Pavlović, S. E., & Khachikian, T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.005 · Cited by 29
Burns, J., Hoffmann, S., Kurz, C., Laxy, M., Polus, S., & Rehfuess, E. A. (2020) · Atmospheric Environment · doi:10.1016/j.atmosenv.2020.118089 · Cited by 23
Impact of small-area lockdowns for the control of the COVID-19 pandemicPreprint
Cuadrado, C., Monsalves, M. J., Jean Gajardo, J., Bertoglia, M. P., Nájera, M., Alfaro, T., Canals, M., Kaufman, J. S., & Peña, S. (2020) · medRxiv · doi:10.1101/2020.05.05.20092106 · Cited by 23
Kypri, K., & Livingston, M. (2020) · Addiction · doi:10.1111/add.15025 · Cited by 22
Sosenko, F., Bramley, G., & Bhattacharjee, A. (2022) · BMC Public Health · doi:10.1186/s12889-022-13738-0 · Cited by 22
Imagery, Visualization, and Mental Simulation Interventions
Hagger, M. S., & Conroy, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.033 · Cited by 23
Using evidence to end homelessness
Teixeira, L. (2023) · Policy Press eBooks · doi:10.51952/9781447365112.ch012 · Cited by 21
Green, M. A., McKee, M., Hamilton, O. K., Shaw, R. J., Macleod, J., Boyd, A., & Katikireddi, S. V. (2023) · BMJ · doi:10.1136/bmj-2023-075133 · Cited by 21
Self-citationToward a modular evaluation approach of real-world laboratories: Findings from a literature review
Kampfmann, T., Bernert, P., & Lang, D. J. (2022) · Research Evaluation · doi:10.1093/reseval/rvac029 · Cited by 21
COVID-19 vaccination and menstrual cycle characteristics: A prospective cohort study
Wesselink, A. K., Lovett, S. M., Weinberg, J., Geller, R. J., Wang, T. R., Regan, A. K., Willis, M. D., Perkins, R. B., Yland, J. J., Koenig, M. R., et al. (2023) · Vaccine · doi:10.1016/j.vaccine.2023.06.012 · Cited by 21
Evaluation of Minimum Unit Pricing of Alcohol: A Mixed Method Natural Experiment in Scotland
Beeston, C., Robinson, M., Giles, L., Dickie, E., Ford, J., MacPherson, M., McAdams, R., Mellor, R. M., Shipton, D., & Craig, N. J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17103394 · Cited by 21
Effect of rail transit on travel behavior: A systematic review and meta-analysis
Wang, J., Cheng, W., Lü, Y., & Wang, D. (2023) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2023.103882 · Cited by 20
Morche, J., Freitag, S., Hoffmann, F., Rissling, O., Langer, G., Nußbaumer-Streit, B., Toews, I., Gartlehner, G., & Meerpohl, J. J. (2020) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2019.11.003 · Cited by 20
Engagement of Stakeholders in the Design, Evaluation, and Implementation of Complex Interventions
Hudson, J. L., Moon, Z., Hughes, L. D., & Moss-Morris, R. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.024 · Cited by 20
Didier, N. (2021) · International Journal of Educational Development · doi:10.1016/j.ijedudev.2021.102467 · Cited by 20
Shih, Y. T., Sabik, L. M., Stout, N. K., Halpern, M. T., Lipscomb, J., Ramsey, S. D., & Ritzwoller, D. P. (2022) · JNCI Monographs · doi:10.1093/jncimonographs/lgac008 · Cited by 20
Rahilly, J., Williams, A., Chang, M., Cummins, S., Derbyshire, D. W., Hassan, S., Huang, Y., Keeble, M., Liu, B., Medina‐Lara, A., et al. (2024) · Health & Place · doi:10.1016/j.healthplace.2024.103237 · Cited by 20
Uri Gneezy, B., Kajackaite, A., & Meier, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.036 · Cited by 20
Edwards, H. B., Redaniel, M. T., Rejon, C. S., Margelyte, R., Peters, T. J., Tilling, K. M., Hollingworth, W., McLeod, H., Craggs, P., Hill, E., et al. (2023) · Archives of Disease in Childhood Fetal & Neonatal · doi:10.1136/archdischild-2022-324579 · Cited by 20
Rabin, B. A., Cain, K. L., & Glasgow, R. E. (2024) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-071321-041652 · Cited by 21
So, V., Millard, A. D., Katikireddi, S. V., Forsyth, R., Allstaff, S., Deluca, P., Drummond, C., Ford, A., Eadie, D. R., Fitzgerald, N., et al. (2021) · Public Health Research · doi:10.3310/phr09110 · Cited by 19
Self-citationHoelscher, D. M., Ganzar, L. A., Salvo, D., Kohl, H. W., Pérez, A., Brown, H. S., Bentley, S. S., Dooley, E. E., Emamian, A., & Durand, C. P. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19031810 · Cited by 19
Erren, T. C., & Lewis, P. (2020) · Chronobiology International · doi:10.1080/07420528.2020.1772811 · Cited by 18
Sun, G., Choe, E. Y., & Webster, C. J. (2022) · Town Planning Review · doi:10.3828/tpr.2022.14 · Cited by 18
Kaufman, J. S. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.02.005 · Cited by 18
Hunter, C. M., Chou, W. S., & Hooper, M. W. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibab067 · Cited by 18
Study Designs to Assess Real-World Interventions to Prevent COVID-19
Digitale, J. C., Stojanovski, K., McCulloch, C. E., & Handley, M. A. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.657976 · Cited by 18
McCormack, G. R., Ghoneim, D., Frehlich, L., Blackstaffe, A., Turley, L., & Bracic, B. (2021) · Journal of Transport & Health · doi:10.1016/j.jth.2021.101239 · Cited by 17
Videoconferencing in Legal Hearings and Procedural Justice
Factor, R., Kariti, D., Lernau, H., & Ayubi, D. Y. (2023) · Victims & Offenders · doi:10.1080/15564886.2023.2248465 · Cited by 17
Lund, R., Christensen, U., Mathisen, J., Sørensen, K., Srivarathan, A., Molbo, D., Halby, K., & Kristiansen, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030936 · Cited by 17
Changing Behavior Using Social Identity Processes
Reynolds, K. J., Branscombe, N. R., Subašić, E., & Willis, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.016 · Cited by 21
Ruiter, R. A. C., Crutzen, R., de Leeuw, E., & Kok, G. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318.018 · Cited by 20
Lian, Q., Ni, J., Zhang, J., Little, J., LUO, S. H., & Zhang, L. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-03206-1 · Cited by 17
Connecting Neighborhoods and Sleep Health
Johnson, D. A., Al‐Ajlouni, Y. A., & Duncan, S. D. T. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780190930448.003.0015 · Cited by 17
Rahilly, J., Amies‐Cull, B., Chang, M., Cummins, S., Derbyshire, D. W., Hassan, S., Huang, Y., Keeble, M., Liu, B., Medina‐Lara, A., et al. (2024) · SSM - Population Health · doi:10.1016/j.ssmph.2024.101646 · Cited by 16
Impact of US industry payment disclosure laws on payments to surgeons: a natural experiment
Rhee, T. G., Stanic, T., & Ross, J. S. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0087-1 · Cited by 16
Disability pension among persons with chronic disease: Differential impact of a Danish policy reform
Mathisen, J., Jensen, N. K., Bjørner, J. B., Brønnum‐Hansen, H., Christensen, U., Thielen, K., Gadeberg, A. K., & Andersen, I. (2020) · European Journal of Public Health · doi:10.1093/eurpub/ckaa228 · Cited by 16
Li, J., Du, G., Clouser, J. M., Stromberg, A. J., Mays, G. P., Sorra, J., Brock, J., Davis, T. C., Mitchell, S. E., Nguyen, H. Q., et al. (2021) · BMC Health Services Research · doi:10.1186/s12913-020-06020-9 · Cited by 18
Miller, H. J., Clifton, K. J., Akar, G., Tufte, K., Gopalakrishnan, S., MacArthur, J., Irwin, E. G., Ramnath, R., & Stiles, J. (2021) · Harvard Data Science Review · doi:10.1162/99608f92.2025202b · Cited by 19
Mortality Among Patients Undergoing Blood Transfusion in Relation to Donor Sex and Parity
Zhao, J., Sjölander, A., & Edgren, G. (2022) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2022.2115 · Cited by 16
On behalf of the NEXT-D Study Group, Ali, M. K., Wharam, J. F., Duru, O. K., Schmittdiel, J. A., Ackermann, R. T., Albu, J., Ross‐Degnan, D., Hunter, C. M., Mangione, C. M., et al. (2018) · Current Diabetes Reports · doi:10.1007/s11892-018-1112-3 · Cited by 16
Exercise and Experiments of Nature
Joyner, M. J., Wiggins, C. C., Baker, S. E., Klassen, S. A., & Senefeld, J. W. (2023) · Comprehensive physiology · doi:10.1002/cphy.c220027 · Cited by 18
Impact of structural-level environmental interventions on physical activity: a systematic review
Hernández-Álvarez, E. D., Cobo, E. A., Cahalin, L. P., & Pamela Serón, S. (2023) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-023-01973-w · Cited by 15
Costa, D., Rohleder, S., & Bozorgmehr, K. (2024) · BMC Public Health · doi:10.1186/s12889-024-19799-7 · Cited by 15
Natural experiments for orthopaedic trauma research: An introduction
van de Wall, B. J. M., Stadhouder, A., Houwert, R. M., Öner, F. C., Beeres, F. J. P., & Groenwold, R. H. H. (2022) · Injury · doi:10.1016/j.injury.2022.11.028 · Cited by 15
Puig-Ribera, A. M., Arumí-Prat, I., Cirera, E., Solà, M., Codina-Nadal, A., Palència, L., León-Gómez, B. B., & Pérez, K. (2022) · Archives of Public Health · doi:10.1186/s13690-022-01005-y · Cited by 15
Richards, A. B., Sheldrick, M. P. R., Swindell, N. J., Barker, H. G., Hudson, J., & Stratton, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280653 · Cited by 15
Facchini, L. A., Nunes, B. P., Felisberto, E., da Silva, J. A. M., da Silva, J. B., & Tomasi, E. (2018) · BMC Public Health · doi:10.1186/s12889-018-6102-5 · Cited by 15
Vandenberg, B., Jiang, H., & Livingston, M. (2019) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2019.04.012 · Cited by 16
Stokes, J., Shah, V., Goldzahl, L., Kristensen, S. R., & Sutton, M. (2020) · Journal of Health Services Research & Policy · doi:10.1177/1355819620963500 · Cited by 15
Exogenous shocks: Definitions, types, and causal identification issues
Jacquart, P., Santoni, S., Schudy, S., Sieweke, J., & Withers, M. C. (2024) · The Leadership Quarterly · doi:10.1016/j.leaqua.2024.101823 · Cited by 15
Duru, O. K., Mangione, C. M., Rodríguez, H. P., Ross‐Degnan, D., Wharam, J. F., Black, B. S., Kho, A., Huguet, N., Angier, H. E., Mayer, V., et al. (2018) · Current Diabetes Reports · doi:10.1007/s11892-018-0977-5 · Cited by 14
Metadata
- Authors
- Craig, P., Katikireddi, S. V., Leyland, A., & Popham, F.
- Affiliations
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Annual Review of Public Health
- Publisher
- Annual Reviews
- Published
- 2017-03-20
- Volume
- 38
- Issue
- 1
- Pages
- 39-56
- ISSN
- 0163-7525 (ISSN-L)
- 1545-2093
- PMC
- PMC6485604
- OpenAlex
- W2516352831
- Semantic Scholar
- 6a9707b998af100fd91830ef34d5aab862721860
- Cited by
- 570
- References
- 76
- Citations by year
- 2026: 48 · 2025: 53 · 2024: 53 · 2023: 70 · 2022: 67 · 2021: 83 · 2020: 126 · 2019: 31 · 2018: 29 · 2017: 8
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Annual Review of Public Health (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion, cc-by-sa)
- PubMed Central (acceptedVersion, cc-by-sa)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
- Funders
- Scottish Government (SPHSU13, SCAF/15/02, SPHSU15)
- Medical Research Council (MC_UU_12017–13 & MC_UU_12017–15, MC_UU_12017/13, MC_UU_12017/15, MC_UU_12017, SPHSU15, SPHSU13)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- natural experiment, population health interventions, causal inference, administrative data, data linkage, health disparities, public health evidence
- MeSH terms
- Decision Making, Health Priorities, Humans, Public Health, Research, Socioeconomic Factors, Healthcare Disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
