The Promise and Pitfalls of Differences-in-Differences: Reflections on 16 and Pregnant and Other Applications
Open accessAbstract
We use the exchange between Kearney/Levine and Jaeger/Joyce/Kaestner on 16 and Pregnant to reexamine the use of DiD as a response to the failure of nature to properly design an experiment for us. We argue that (1) any DiD paper should address why the original levels of the experimental and control groups differed, and why this would not impact trends, (2) the parallel trends argument requires a justification of the chosen functional form and that the use of the interaction coefficients in probit and logit may be justified in some cases, and (3) parallel trends in the period prior to treatment is suggestive of counterfactual parallel trends, but parallel pre-trends is neither necessary nor sufficient for the parallel counterfactual trends condition to hold. Importantly, the purely statistical approach uses pretesting and thus, generates the wrong standard errors. Moreover, we underline the dangers of implicitly or explicitly accepting the null hypothesis when failing to reject the absence of a differential pre-trend.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The wider use of DiD has coincided with major methodological advances over the past decade that improved the method while making it harder to implement and interpret.
“Broader DiD adoption has been accompanied by significant methodological advancements over the past decade, which have refined the approach but rendered its implementation and interpretation more challenging [17-21].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [20]
Section 4, “Testing for Preexisting Trends” (page unavailable), says DiD has grown in popularity and that researchers are now effectively required to test for pre-trends when data permit. The paper supports broader adoption and more demanding implementation, but it does not attribute those demands to major methodological advances over the past decade or say that such advances refined DiD while making interpretation harder. (LLM, conf=0.82)
Experimental — accuracy not yet measuredEven if there are pre-existing differences in trends, the tests described above may fail to reject owing to low power.
“A second issue is that even if there are pre-existing differences in trends, the tests described above may fail to reject owing to low power (Bilinski and Hatfield, 2018; Freyaldenhoven, Hansen a 2019; Kahn-Lang and Lang, 2020; Roth, 2022).”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Kahn-Lang and Lang, 2020
Section 4, “Testing for Preexisting Trends” (page unavailable), says that failing to reject parallel pre-treatment trends does not establish that the trends were parallel, and that passing a pre-trend test does not rule out significant pre-trends. It does not specifically attribute such a failure to reject to low power; its explicit discussion of reduced power concerns detecting a treatment effect after adding group-specific trends. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTraditional pre-trend tests can mislead because they may lack enough statistical power to detect a parallel trends violation even when non-parallel pre-intervention trends exist.
“However, as several papers have noted [20, 24, 76-77], these tests can be misleading because they may not have adequate statistical power to detect a violation, even if there are non-parallel pre-intervention trend differences between treatment and comparison groups [20, 24, 76-77].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [20]
Previous papers have noted that the parallel trends assumption may hold in logs but not levels or vice versa.
“Previous papers have noted that the parallel trends assumption may hold in logs but not levels or vice versa (e.g. Meyer, 1995; Athey and Imbens, 2006; Kahn-Lang and Lang, 2020).”
From When Is Parallel Trends Sensitive to Functional Form? (2023) · cites this paper as Kahn-Lang and Lang, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Interaction terms in logit and probit models
Ai, C., & Norton, E. C. (2003) · Economics Letters · doi:10.1016/s0165-1765(03)00032-6 · Cited by 5,466 · see citing works
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037 · see citing works
Identification and estimation of local average treatment effects
Imbens, G. W., & Angrist, J. D. (1994) · Econometrica · doi:10.2307/2951620 · Cited by 4,101 · see citing works
Natural and Quasi- Experiments in Economics
Meyer, B. (1994) · Journal of Business and Economic Statistics · doi:10.2307/1392369 · Cited by 1,806 · see citing works
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566 · see citing works
Identification and Inference in Nonlinear Difference-in-Differences Models
Athey, S., & Imbens, G. W. (2006) · Econometrica · doi:10.1111/j.1468-0262.2006.00668.x · Cited by 1,084 · see citing works
Mode of Communication of Cholera.
Parkes, E. A. (1855) · PubMed · Cited by 924 · see citing works
Revisiting Event Study Designs
Borusyak, K., & Jaravel, X. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2826228 · Cited by 463 · see citing works
Media Influences on Social Outcomes: The Impact of MTV’s <i>16 and Pregnant</i> on Teen Childbearing
Kearney, M. S., & Levine, P. B. (2015) · American Economic Review · doi:10.1257/aer.20140012 · Cited by 316 · see citing works
Alternative Approaches to Evaluation in Empirical Microeconomics
Blundell, R. W., & Dias, M. C. (2009) · The Journal of Human Resources · doi:10.3368/jhr.44.3.565 · Cited by 236 · see citing works
Ability Bias, Discount Rate Bias and the Return to Education
Lang, K. (1993) · Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) · Cited by 100 · see citing works
Self-citationJaeger, D. A., Joyce, T., & Kaestner, R. J. (2018) · Journal of Business and Economic Statistics · doi:10.1080/07350015.2018.1497510 · Cited by 70 · see citing works
Does Reality TV Induce Real Effects? A Response to Jaeger, Joyce, and Kaestner (2016)Preprint
Kearney, M. S., & Levine, P. B. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2864818 · Cited by 5 · see citing works
Should We Adjust for the Test for Pre-trends in Difference-in-Difference Designs?Preprint
Roth, J. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1804.01208 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027 · see citing works
A More Credible Approach to Parallel Trends
Rambachan, A., & Roth, J. (2023) · Review of Economic Studies · doi:10.1093/restud/rdad018 · Cited by 1,559 · see citing works
Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends
Roth, J. (2022) · American Economic Review: Insights · doi:10.1257/aeri.20210236 · Cited by 769 · see citing works
On the Use of Two-Way Fixed Effects Regression Models for Causal Inference with Panel Data
Imai, K., & Kim, I. S. (2021) · Political Analysis · doi:10.1017/pan.2020.33 · Cited by 623 · see citing works
“We emphasize that this assumption may not be credible in some settings (see, e.g., Bilinski and Hatfield 2018; Kahn-Lang and Lang 2019; Rambachan and Roth 2019).”
BackgroundTwo-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey
de Chaisemartin, C., & D’Haultfœuille, X. (2023) · The Econometrics Journal · doi:10.1093/ectj/utac017 · Cited by 522 · see citing works
COVID-19 doesn’t need lockdowns to destroy jobs: the effect of local outbreaks in Korea
Aum, S., Lee, S. Y. (., & Shin, Y. (2021) · Labour Economics · doi:10.1016/j.labeco.2021.101993 · Cited by 175 · see citing works
When Is Parallel Trends Sensitive to Functional Form?
Roth, J., & Sant'Anna, P. H. C. (2023) · Econometrica · doi:10.3982/ecta19402 · Cited by 142 · see citing works
Synthetic controls with staggered adoption.
Ben-Michael, E., Feller, A., & Rothstein, J. (2022) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12448 · Cited by 120 · see citing works
Pink work: Same-sex marriage, employment and discrimination
Sansone, D. (2019) · Journal of Public Economics · doi:10.1016/j.jpubeco.2019.104086 · Cited by 94 · see citing works
Two-Way Fixed Effects and Differences-in-Differences with Heterogeneous Treatment Effects: A Survey
de Chaisemartin, C., & D'Haultfoeuille, X. (2022) · National Bureau of Economic Research · doi:10.3386/w29734 · Cited by 70 · see citing works
The impact of upzoning on housing construction in Auckland
Greenaway-McGrevy, R., & Phillips, P. C. B. (2023) · Journal of Urban Economics · doi:10.1016/j.jue.2023.103555 · Cited by 70 · see citing works
Causal Panel Analysis under Parallel Trends: Lessons from a Large Reanalysis Study
CHIU, A., LAN, X., LIU, Z., & XU, Y. (2026) · American Political Science Review · doi:10.1017/s0003055425000243 · Cited by 48 · see citing works
Testing for Equivalence of Pre-Trends in Difference-in-Differences Estimation
Dette, H., & Schumann, M. (2024) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2024.2308121 · Cited by 30 · see citing works
Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods
Feng, S., Ganguli, I., Lee, Y., Poe, J., Ryan, A., & Bilinski, A. (2025) · Statistics in Medicine · doi:10.1002/sim.70247 · Cited by 16 · see citing works
- Authors
- Kahn-Lang, A., & Lang, K.
- Affiliations
- Harvard University (US)
- Boston University (US)
- ORCID
- Lang, Kevin
- Venue
- Journal of Business & Economic Statistics
- Publisher
- Informa UK Limited
- Published
- 2019-04-02
- Volume
- 38
- Issue
- 3
- Pages
- 613-620
- ISSN
- 0735-0015 (ISSN-L)
- 1537-2707
- OpenAlex
- W2884216135
- Cited by
- 367
- References
- 14
- Citations by year
- 2026: 37 · 2025: 70 · 2024: 60 · 2023: 57 · 2022: 71 · 2021: 47 · 2020: 17 · 2019: 7 · 2016: 1
- FWCI
- 27.80
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- Journal of Business and Economic Statistics (publishedVersion)
- OpenBU (Boston University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- difference-in-differences, parallel trends assumption, adolescent pregnancy, pretesting, standard errors, null hypothesis testing, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
