The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index; none of the 40 sources its claims cite is checked for support yet.
Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research
Open accessAbstract
Population health interventions are essential to reduce health inequalities and tackle other public health priorities, but they are not always amenable to experimental manipulation. Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances. One key challenge in evaluating NEs is selective exposure to the intervention. Studies should be based on a clear theoretical understanding of the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations. Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation. NE studies often rely on existing (including routinely collected) data. Investment in such data sources and the infrastructure for linking exposure and outcome data is essential if the potential for such studies to inform decision making is to be realized.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Natural experiments (NEs) use naturally occurring variation in exposure to evaluate population health interventions that cannot be randomized, but selective exposure is a central threat to causal inference. The paper reviews methods for NE evaluation and argues that inference is strengthened by clear theory, additional design features, and investment in linked routinely collected data.
Key findings
- NE studies are defined broadly as any event not under researcher control that divides a population into exposed and unexposed groups; because exposure is often selective, studies should model the assignment process and consider alternative explanations even when effects are large and rapid.
- The paper reviews regression adjustment, propensity score methods, difference-in-differences (DiD), interrupted time series (ITS), synthetic controls, instrumental variables (IVs), and regression discontinuity (RD), noting that combining methods, replication, and falsification tests can strengthen causal inference.
- Illustrative examples include a 1995 complete pesticide import ban in Sri Lanka associated with a 50% fall in suicide by self-poisoning; Oregon’s 2003 traffic police layoff of more than one-third of the force associated with a 12–14% increase in fatalities; and Scotland’s ban on multipurchase alcohol discounts associated with a 2% decrease in overall sales after adjustment for England and Wales.
- The paper concludes that continued investment in routinely collected data, population health surveys, and data-linkage infrastructure is essential for NEs to inform policy making.
Methods
- Design: narrative overview/review of methods, approaches, and contributions of natural experiments to public health intervention research, using the potential outcomes framework to compare RCTs, NEs, and nonexperimental observational studies.
- Data or sample: no primary study sample; the paper draws on methodological literature and published NE examples, including Sri Lanka’s pesticide ban, Oregon traffic policing, Scotland’s alcohol legislation, California’s tobacco control program, US Head Start, and Mexico’s Oportunidades program.
- Main analysis: conceptual and methodological synthesis of effect-estimation approaches, their assumptions, strengths, limitations, and strategies for strengthening inference; no new quantitative analysis is reported.
Limitations
- The paper states that NEs are not the answer to every evaluation question, that it is not always possible to conduct a good NE when an RCT would be impractical, and that method choice is constrained by intervention features, allocation process, comparators, expected impacts, and data availability.
- The paper states method-specific limitations, including selective exposure, DiD’s common trends assumption, ITS power and autocorrelation concerns, synthetic controls’ lack of conventional statistical inference and donor-pool matching constraints, IVs’ difficulty in finding and justifying valid instruments, and RD’s local effects and bias–precision trade-off.
- Evident from the text: this is a narrative overview without a described systematic search strategy or formal quality appraisal; the cited examples are illustrative rather than a comprehensive evidence synthesis.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
47 Own interpretation · 30 Not checked yet
In the paper
“In an observational study, knowledge of the assignment mechanism can be used to make the groups conditionally exchangeable, for example, by controlling for variables that influence both assignment and outcomes to the extent that these variables are known and accurately measured (34).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [34] Hernán M, 2017
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interventions that are introduced abruptly, that affect large populations, and that are implemented where it is difficult for individuals to manipulate their treatment status are more straightforward to evaluate.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The potential outcomes model provides a useful framework for clarifying similarities and differences between true experiments on the one hand and observational studies (including NEs) on the other hand (51).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [51] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Truly random assignment, although not unknown (14), is extremely rare in policy and practice settings.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Introduction
What the sources say
- Not checked yetcites [14] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Extrapolation may also be a problem for some NE studies, such as those using RD designs, which estimate treatment effects at a particular value of a variable used to determine assignment.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Likewise, effects that are large and follow rapidly after implementation are more readily detectable than more subtle or delayed effects.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“After the complete ban, rates of suicide by self-poisoning fell by 50%.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
What the sources say
- Not checked yetcites [32] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Standard multivariable models, which control for observed differences between intervention and control groups, can be used to evaluate NEs when no important differences in unmeasured characteristics between intervention and control groups are expected”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Adjustment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The key strength of the study by Goodman et al. is the way the timing of data gathering in relation to exposure created well-balanced intervention and control groups.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Adjustment
What the sources say
- Not checked yetcites [29] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A common shortcoming in ITS analyses is a lack of statistical power (61).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
What the sources say
- Not checked yetcites [61] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 26
In the paper
“In an observational study, where exchangeability may be achieved by a method such as matching or by conditioning on covariates, intervention groups may be created whose members in practice have no chance of receiving the treatment (55). The meaning of treatment effects estimated in this way is unclear.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [55] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Key considerations when choosing an NE evaluation method are the source of variation in exposure and the size and nature of the expected effects.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“this framework indicates why an understanding of the assignment process is so important to the design of an NE study.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NE studies often rely on existing (including routinely collected) data.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Some methods, such as fuzzy regression discontinuity (RD) and instrumental variables (IVs), estimate a different effect, the complier average causal effect (CACE), which is the effect of the intervention on those who comply with their allocation into the control or intervention group (11). Under certain assumptions, the CACE is equivalent to the ITT effect divided by the proportion of compliers.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
What the sources say
- Not checked yetcites [11] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If assignment is random, the groups are said to be exchangeable and the intervention’s average causal effect can be estimated from the difference in the average outcomes for the two groups.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A comparison of outcomes between groups assigned to the intervention or control condition provides an estimate of the effect of assignment, known as the intention-to-treat (ITT) effect, rather than the effect of the intervention itself. The two are necessarily the same only if there is perfect compliance.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As-if randomness lacks a precise definition”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In a well-conducted RCT, randomization ensures exchangeability.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The methods discussed in the next section can be seen as different ways of achieving conditional exchangeability.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Causal Inference in Trials and Observational Studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the methods proposed to identify as-if random processes (such as a good understanding of the assignment process and checks on the balance of covariates between exposed and unexposed groups) are those used to assess threats to validity in any study that attempts to make causal inferences from observational data.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The method relies on the assumption that, in the absence of the intervention, preimplementation trends would continue.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unless autocorrelation is addressed, the standard errors will be underestimated, but models that allow for autocorrelation can be fitted using standard statistical packages.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Although in statistical terms the use of propensity scores may produce results that differ little from those obtained through traditional regression adjustment (70), they encourage clearer thinking about study design and particularly the assignment mechanism (66).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In contrast, this study found no difference, suggesting that the earlier findings reflected the selective provision of training.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Adjustment
What the sources say
- Not checked yetcites [29] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DiD’s strength is that it controls for unobserved as well as observed differences in the fixed (i.e., time-invariant) characteristics of the groups and is therefore less prone to omitted variable bias caused by unmeasured confounders or measurement error.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Understanding the mechanism underlying allocation to intervention and control groups is key when deriving the propensity score.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“There are four principal ways to use the propensity score to obtain an estimated treatment effect: matching, stratification, inverse probability weighting, and covariate adjustment (7). Each method will adjust for differences in characteristics of the intervention and control groups and, in so doing, minimize the effects of confounding.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
What the sources say
- Not checked yetcites [7] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Recent developments, such as the use of propensity scores, rather than traditional covariate adjustment, to account for group-specific time-varying characteristics, add additional complexity, but combining DiD with other approaches in this way may further strengthen causal inference.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Another limitation is that conventional methods of statistical inference cannot be applied, although Abadie et al. (1) suggest an alternative that compares the estimated effect for the intervention area with the distribution of placebo effects derived by comparing each area in the donor pool with its own synthetic control.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Synthetic Controls
What the sources say
- Not checked yetcites [1] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With data for multiple preimplementation time points, the common trends assumption can be investigated directly, and it can be relaxed by extending the model to include terms for group-specific trends.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Difference-in-Differences
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In the absence of random allocation, the propensity score attempts to recreate the allocation mechanism, defined as the conditional probability of an individual being in the intervention group, given a number of covariates (65).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Propensity Score–Based Methods
What the sources say
- Not checked yetcites [65] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The synthetic control method can be seen as an extension of the DiD method, with a number of advantages. In particular, it relaxes the requirement for a geographical control that satisfies the parallel trends assumption and relies less on subjective choices of control areas.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Synthetic Controls
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Controlled ITS designs, which compare trends in exposed and unexposed groups or in outcomes that are not expected to change as a result of the intervention, can be used to strengthen causal inference still further; in addition, standardization can be used to control for changes in population composition.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By accounting for preintervention trends, well-conducted ITS studies permit stronger causal inference than do cross-sectional or simple prepost designs, but they may be subject to confounding by cointerventions or changes in population composition.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Interrupted Time Series
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even when such protocols are published, they do not provide a perfect safeguard against selective reporting of positive findings.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Causal inference can be strengthened in NE studies by the inclusion of additional design features alongside the principal method of effect estimation.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Good-quality natural experimental studies typically use a range of approaches.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NE approaches to evaluation have become topical because they address researchers’ and policy makers’ interests in understanding the impact of large-scale population health interventions that, for practical, ethical, or political reasons, cannot be manipulated experimentally.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This restriction to a subset of the population may not matter if the intervention is expected to affect outcomes locally, as in the case of a minimum legal drinking age or if the substantive focus of the study is on the effect of a small change in the assignment rule. It is more serious when the outcome of interest is the effect on the whole population.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Discontinuity
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“IVs are widely used in econometric program evaluation and have attracted much recent interest in epidemiology, particularly in the context of Mendelian randomization studies, which use genetic variants as instruments for environmental exposures (25, 36, 48).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Instrumental Variables
What the sources say
- Not checked yetcites [25] source
- Not checked yetcites [36] 36. Holmes MV, Dale CE, Zuccolo L, Silverwood RJ, Guo Y, et al. 2014. Association between alcohol and cardiovascular disease: Mendelian randomisation analysis based on individual participant data. BMJ 349:g4164
- Not checked yetcites [48] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Reporting of NE studies of all kinds may also be improved by following established reporting guidelines such as STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) (74) or TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) (28).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Replication studies, which by definition retest a previously published hypothesis, are a valuable additional safeguard against retrospectively fitting hypotheses to known features of the data.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Most natural experimental studies are conducted retrospectively, using data collected before the study is planned.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Availability of data also constrains the choice of methods.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Such comparisons are particularly useful in early applications of novel methods whose strengths and weaknesses are not fully understood (49).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
What the sources say
- Not checked yetcites [49] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“NEs are not the answer to every evaluation question, and it is not always possible to conduct a good NE study whenever an RCT would be impractical.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“RD methods are widely regarded as the closest approximation of an observational study to an RCT (5), but their real value derives from their wide applicability to the evaluation of social programs for which eligibility is determined by a score on some form of continuous scale and also from their reliance on relatively weak, directly testable assumptions.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Discontinuity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To work in this way, an IV must be associated with exposure to the intervention, must have no association with any other factors associated with exposure, and must be associated with outcomes only through its association with exposure to the intervention (Figure 1).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Instrumental Variables
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“One key shortcoming is that restricting the bandwidth to reduce bias results in a loss of precision (46, 73), and estimates that may hold over only a small segment of the whole population exposed to the intervention.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Methods for Evaluating Natural Experiments > Regression Discontinuity
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Smaller numbers of observations yield less precise estimates.
None of the sources cited for this claim have been checked yet.
In the paper
“Recent investments in infrastructure to link data across policy sectors for research purposes are a welcome move that should increase opportunities to evaluate NEs (12, 26, 37).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Conclusions
What the sources say
- Not checked yetcites [12] source
- Not checked yetcites [26] Farr Inst. Health Inf. Res., 2016
- Not checked yetcites [37] House Commons Sci. Technol. Comm, 2016
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Qualitative research can strengthen the design of RCTs of complex public health interventions (10, 56), and this argument applies equally to NEs (38).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
What the sources say
- Not checked yetcites [10] source
- Not checked yetcites [56] source
- Not checked yetcites [38] 38. Humphreys DK, Panter J, Sahlqvist S, Goodman A, Ogilvie D. 2016. Changing the environment to improve population health: a framework for considering exposure in natural experimental studies. J. Epidemiol. Community Health. doi: 10.1136/jech-2015-206381 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“It can also improve understanding of the processes that determine exposure, factors associated with intervention delivery and compliance, mechanisms by which outcomes are realized, and the strengths and limitations of routinely collected measures of exposures and outcomes (13).”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
What the sources say
- Not checked yetcites [13] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if the observed effects are large and rapidly follow implementation, confidence in attributing them to the intervention can be markedly improved by a detailed consideration of alternative explanations.”
§ Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research > Strengthening Inference in Natural Experimental Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
Age, income, and other continuous variables are often used to determine entitlement to social programs such as means-tested welfare benefits.
“Age, income, and other continuous variables are often used to determine entitlement to social programs, such as means-tested welfare benefits.”
The paper relies on this without support.
Smaller numbers of observations yield less precise estimates.
“One drawback is that smaller numbers of observations will yield less precise estimates, so the choice involves a trade-off between bias and precision.”
The paper relies on this without support.
Most natural experimental studies are conducted retrospectively, using data collected before the study is planned.
“Most NE studies are conducted retrospectively, using data collected before the study is planned.”
The paper relies on this without support.
Good-quality natural experimental studies typically use a range of approaches.
“Good-quality NE studies typically use a range of approaches.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“noting only that what they call ‘qualitative’ techniques (the cited references for these are all QCA studies) require “further studies … to establish their validity and usefulness” [2].”
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.008 · Cited by 17,893
(2002) · Cited by 13,211
Austin, P. C. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.568786 · Cited by 12,242
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204
Abadie, A., Diamond, A., & Hainmueller, J. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08746 · Cited by 5,513
Regression discontinuity designs in economics
Lee, D. S., & Lemieux, T. (2010) · Journal of Economic Literature · doi:10.1257/jel.48.2.281 · Cited by 4,534
The economic costs of conflict: a case study of the Basque Country
Abadie, A., & Gardeazabal, J. (2003) · American Economic Review · doi:10.1257/000282803321455188 · Cited by 4,517
Segmented regression analysis of interrupted time series studies in medication use research
Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross-Degnan, D. (2002) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1046/j.1365-2710.2002.00430.x · Cited by 3,279 · cited 2×
“Lags in the effect of the intervention can be accounted for by omitting outcome values that occur during the lag period or by modeling the lag period as a separate segment (75).”
BackgroundMethod
Research works printed but not linked: 4
- Acad. Med. Sci., 2007, Identifying the Environmental Causes of Disease: How Should We Decide What to Believe and When to Take Action.
- Farr Inst. Health Inf. Res., 2016, Environmental and public health research.
- Hernán M, 2017, Causal Inference
- House Commons Sci. Technol. Comm, 2016, The Big Data Dilemma. Fourth Report of Session 2015–16.
Cited by
Sorted by citations
Designing Difference in Difference Studies: Best Practices for Public Health Policy Research
Wing, C., Simon, K., & Bello-Gomez, R. A. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-013507 · Cited by 1,470
Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.012 · Cited by 911
Self-citationSkivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K., Craig, N. J., French, D., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 585
Self-citationSelecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research
Handley, M. A., Lyles, C. R., McCulloch, C., & Cattamanchi, A. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014128 · Cited by 428
The Handbook of Behavior Change
Hagger, M. S., Cameron, L. D., Hamilton, K., Hankonen, N. E., & Lintunen, T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108677318 · Cited by 446
Metadata
- Authors
- Craig, P., Katikireddi, S. V., Leyland, A., & Popham, F.
- Affiliations
- MRC/CSO Social and Public Health Sciences Unit (GB)
- Venue
- Annual Review of Public Health
- Publisher
- Annual Reviews
- Published
- 2017-03-20
- Volume
- 38
- Issue
- 1
- Pages
- 39-56
- ISSN
- 0163-7525 (ISSN-L)
- 1545-2093
- PMC
- PMC6485604
- OpenAlex
- W2516352831
- Semantic Scholar
- 6a9707b998af100fd91830ef34d5aab862721860
- Cited by
- 570
- References
- 76
- Citations by year
- 2026: 48 · 2025: 53 · 2024: 53 · 2023: 70 · 2022: 67 · 2021: 83 · 2020: 126 · 2019: 31 · 2018: 29 · 2017: 8
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Annual Review of Public Health (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion, cc-by-sa)
- PubMed Central (acceptedVersion, cc-by-sa)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
- Funders
- Scottish Government (SPHSU13, SCAF/15/02, SPHSU15)
- Medical Research Council (MC_UU_12017–13 & MC_UU_12017–15, MC_UU_12017/13, MC_UU_12017/15, MC_UU_12017, SPHSU15, SPHSU13)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- natural experiment, population health interventions, causal inference, administrative data, data linkage, health disparities, public health evidence
- MeSH terms
- Decision Making, Health Priorities, Humans, Public Health, Research, Socioeconomic Factors, Healthcare Disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
