Inference with “Difference in Differences” with a Small Number of Policy Changes
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
In difference-in-differences applications, identification of the key parameter often arises from changes in policy by a small number of groups. In contrast, typical inference assumes that the number of groups changing policy is large. We present an alternative inference approach for a small (finite) number of policy changers, using information from a large sample of nonchanging groups. Treatment effect point estimators are not consistent, but we can consistently estimate their asymptotic distribution under any point null hypothesis about the treatment. Thus, treatment point estimators can be used as test statistics, and confidence intervals can be constructed using test statistic inversion.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Nevertheless, the assumption that all treated groups have the same distribution of errors is still strong, and will often be violated if either there is heterogeneity in treatment effects or in cluster sizes.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Conley and Taber (2011)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The key idea in Conley and Taber (2011) is that if we assume that the errors in treated states come from the same distribution as in control states, then we can learn the distribution of errors from the large number of control states and use that to construct standard errors.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Conley and Taber (2011)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Conley and Taber proposed an approach to estimate the distribution of errors in the treated clusters using that from the untreated clusters [40]. This approach may be powerful when there are only a few treated clusters but many untreated ones from which to estimate the error distribution.”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Conley and Taber (2011) introduce an alternative approach for inference that is able to relax the strong assumption of Gaussian errors in settings where there are many control clusters ( J 0 large) but few treated clusters ( J 1 small).”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Conley and Taber (2011)
The cited source says what the claim attributes to it.
In the citing paper
“A key advantage of this approach is that the distribution of errors is not assumed to be Gaussian, but rather is learned from the data.”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Conley and Taber (2011)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Filders, R. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90130-2 · Cited by 16,165
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641
Abadie, A., Diamond, A., & Hainmueller, J. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08746 · Cited by 5,513
Earnings Losses of Displaced Workers
Jacobson, L. S., LaLonde, R. J., & Sullivan, D. G. (1992) · doi:10.17848/wp92-11 · Cited by 2,895
GMM Estimation with Cross Sectional Dependence
Conley, T. G. (1999) · Journal of Econometrics · doi:10.1016/s0304-4076(98)00084-0 · Cited by 2,685
Self-citationAn Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units
Moulton, B. R. (1990) · The Review of Economics and Statistics · doi:10.2307/2109724 · Cited by 2,630
Analysis of Covariance with Qualitative Data
Chamberlain, G. (1980) · The Review of Economic Studies · doi:10.2307/2297110 · Cited by 2,629
Card, D., & Krueger, A. B. (2000) · American Economic Review · doi:10.1257/aer.90.5.1397 · Cited by 2,206
Natural and Quasi- Experiments in Economics
Meyer, B. (1994) · Journal of Business and Economic Statistics · doi:10.2307/1392369 · Cited by 1,792
The Impact of the Mariel Boatlift on the Miami Labor Market
Card, D. (1990) · ILR Review · doi:10.1177/001979399004300205 · Cited by 1,704
Empirical Strategies in Labor Economics
Angrist, J. D., & Krueger, A. B. (1999) · Handbook of Labor Economics · doi:10.1016/s1573-4463(99)03004-7 · Cited by 1,628
Inference with Difference-in-Differences and Other Panel Data
Donald, S. G., & Lang, K. (2007) · Review of Economics and Statistics · doi:10.1162/rest.89.2.221 · Cited by 1,324
The Geometry of Random Fields.
Gray, L. F., & Adler, R. J. (1982) · Journal of the American Statistical Association · doi:10.2307/2287334 · Cited by 1,285
Natural and Quasi-Experiments in Economics
Meyer, B. D. (1995) · Journal of Business & Economic Statistics · doi:10.1080/07350015.1995.10524589 · Cited by 1,100
Identification and Inference in Nonlinear Difference-in-Differences Models
Athey, S., & Imbens, G. W. (2006) · Econometrica · doi:10.1111/j.1468-0262.2006.00668.x · Cited by 1,084
Semi-Nonparametric Maximum Likelihood Estimation
Gallant, A. R., & Nychka, D. W. (1987) · Econometrica · doi:10.2307/1913241 · Cited by 913
Cluster-Sample Methods in Applied Econometrics
Wooldridge, J. M. (2003) · American Economic Review · doi:10.1257/000282803321946930 · Cited by 867
Classes of Nonseparable, Spatio-Temporal Stationary Covariance Functions
Cressie, N., & Huang, H.-C. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473885 · Cited by 679
Covariance Adjustment in Randomized Experiments and Observational Studies
Rosenbaum, P. R. (2002) · Statistical Science · doi:10.1214/ss/1042727942 · Cited by 477
Hope for Whom? Financial Aid for the Middle Class and Its Impact on College Attendance
Dynarski, S. (2000) · National Bureau of Economic Research · doi:10.3386/w7756 · Cited by 394
Hansen, C. B. (2007) · Journal of Econometrics · doi:10.1016/j.jeconom.2006.07.011 · Cited by 389
The Enrollment Effects of Merit‐Based Financial Aid: Evidence from Georgia’s HOPE Program
Cornwell, C., Mustard, D. B., & Sridhar, D. J. (2006) · Journal of Labor Economics · doi:10.1086/506485 · Cited by 374
Hope for Whom? Financial Aid for the Middle Class and Its Impact on College Attendance
Dynarski, S. (2000) · National Tax Journal · doi:10.17310/ntj.2000.3s.02 · Cited by 328
Semiparametric Analysis of Random Effects Linear Models from Binary Panel Data
Manski, C. F. (1987) · Econometrica · doi:10.2307/1913240 · Cited by 328
On characterizing the gamma and the normal distribution
Kotlarski, I. (1967) · Pacific Journal of Mathematics · doi:10.2140/pjm.1967.20.69 · Cited by 267
Abortion Legalization and Child Living Circumstances: Who is the "Marginal Child"?
Gruber, J., Levine, P., & Staiger, D. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399556007 · Cited by 263
Hotz, V. J., Mullin, C. H., & Sanders, S. G. (1997) · The Review of Economic Studies · doi:10.2307/2971732 · Cited by 255
Semiparametric Analysis Of Random Effects Linear Models From Binary Panel DataPreprint
Manski, C. F., & Manski, C. F. (1985) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.292669 · Cited by 249
The effects of the unemployment insurance payroll tax on wages, employment, claims and denials
Anderson, P. M., & Meyer, B. D. (2000) · Journal of Public Economics · doi:10.1016/s0047-2727(99)00112-7 · Cited by 231
End-of-Sample Instability Tests
Andrews, D. W. K. (2003) · Econometrica · doi:10.1111/1468-0262.00466 · Cited by 196
Efficient Estimators for Regressing Regression Coefficients
Hanushek, E. A. (1974) · The American Statistician · doi:10.1080/00031305.1974.10479073 · Cited by 184
Central limit theorems and uniform laws of large numbers for arrays of random fields
Jenish, N., & Prucha, I. R. (2009) · Journal of Econometrics · doi:10.1016/j.jeconom.2009.02.009 · Cited by 176
Restrictions of economic theory in nonparametric methodsPreprint
Matzkin, R. L. (1986) · RePEc: Research Papers in Economics · Cited by 171
A Spatial Analysis of Sectoral Complementarity
Conley, T. G., & Dupor, B. (2003) · Journal of Political Economy · doi:10.1086/367681 · Cited by 165
Self-citationIdentification and Inference in Nonlinear Difference-in-Differences ModelsPreprint
Athey, S. C., & Imbens, G. W. (2002) · SSRN Electronic Journal · doi:10.2139/ssrn.311920 · Cited by 143
Dynarski, S. (2004) · National Bureau of Economic Research · doi:10.7208/9780226355375-004 · Cited by 142
Paying for grades: Impact of merit‐based financial aid on educational quality
Henry, G. T., & Rubenstein, R. (2002) · Journal of Policy Analysis and Management · doi:10.1002/pam.1042 · Cited by 121
A Note on a Random Coefficients Model
Amemiya, T. (1978) · International Economic Review · doi:10.2307/2526342 · Cited by 114
Spatio-temporal stationary covariance models
Ma, C. (2003) · Journal of Multivariate Analysis · doi:10.1016/s0047-259x(02)00014-3 · Cited by 74
Identifiability in Stochastic Models
(1992) · Elsevier eBooks · doi:10.1016/c2009-0-29097-0 · Cited by 52 · cited 1×
“1 in Prakasa Rao (1992) which he attributes to Kotlarski (1967) as Theorem 2.”
BackgroundGeneralized Predictive Tests and Structural Change Analysis in Econometrics
Dufour, J.-M., Ghysels, E., & Hall, A. (1994) · International Economic Review · doi:10.2307/2527098 · Cited by 51
Efficient Estimators for Regressing Regression Coefficients
Hanushek, E. A. (1974) · The American Statistician · doi:10.2307/2683605 · Cited by 48
The Enrollment Effects of Merit-Based Financial Aid: Evidence from Georgia's Hope ScholarshipPreprint
Cornwell, C. M., Mustard, D. B., & Sridhar, D. J. (2001) · SSRN Electronic Journal · doi:10.2139/ssrn.277313 · Cited by 43
The Effects of Merit-Based Financial Aid on Course Enrollment, Withdrawal and Completion in College
Cornwell, C. M., Lee, K. H., & Mustard, D. B. (2003) · SSRN Electronic Journal · doi:10.2139/ssrn.434543 · Cited by 32
Inference with "Difference in Differences" with a Small Number of Policy Changes
Conley, T. G., & Taber, C. R. (2005) · National Bureau of Economic Research · doi:10.3386/t0312 · Cited by 24
Self-citationThe effects of merit-based financial aid on academic choices in college
Lee, K. H. (2004) · Cited by 7
(1986) · RePEc: Research Papers in Economics
Research works printed but not linked: 1
- Bertrand Marianne, 2002, NBER working paper, 8841
Cited by
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Abadie, A., Diamond, A., & Hainmueller, J. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08746 · Cited by 5,513
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
“The key idea in Conley and Taber (2011) is that if we assume that the errors in treated states come from the same distribution as in control states, then we can learn the distribution of errors from the large number of control states and use that to construct standard errors.”
Designing Difference in Difference Studies: Best Practices for Public Health Policy Research
Wing, C., Simon, K., & Bello-Gomez, R. A. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-013507 · Cited by 1,470
Metadata
- Authors
- Conley, T. G., & Taber, C. R.
- Affiliations
- University of Chicago (US)
- University of Wisconsin–Madison (US)
- ORCID
- Conley, Timothy G.
- Venue
- Review of Economics and Statistics
- Publisher
- MIT Press
- Published
- 2011-02
- Volume
- 93
- Issue
- 1
- Pages
- 113-125
- ISSN
- 0034-6535 (ISSN-L)
- 1530-9142
- OpenAlex
- W2097636528
- Cited by
- 633
- References
- 44
- Citations by year
- 2026: 79 · 2025: 36 · 2024: 45 · 2023: 39 · 2022: 52 · 2021: 50 · 2020: 39 · 2019: 50 · 2018: 49 · 2017: 40 · 2016: 36 · 2015: 36 · 2014: 18 · 2013: 24 · 2012: 14
- FWCI
- 14.89
- Type
- journal article
- Language
- en
- Open access
- green
- www.nber.org
- Locations
- The Review of Economics and Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- difference-in-differences, treatment effect estimation, asymptotic distribution, point null hypothesis, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.