The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
How Much Should We Trust Differences-In-Differences Estimates?
Open accessAbstract
Most papers that employ Differences-in-Differences estimation (DD) use many years of data and focus on serially correlated outcomes but ignore that the resulting standard errors are inconsistent. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its “effect” as well as the standard error of this estimate. These conventional DD standard errors severely understate the standard deviation of the estimators: we find an “effect” significant at the 5 percent level for up to 45 percent of the placebo interventions. We use Monte Carlo simulations to investigate how well existing methods help solve this problem. Econometric corrections that place a specific parametric form on the time-series process do not perform well. Bootstrap (taking into account the autocorrelation of the data) works well when the number of states is large enough. Two corrections based on asymptotic approximation of the variance-covariance matrix work well for moderate numbers of states and one correction that collapses the time series information into a “pre”- and “post”-period and explicitly takes into account the effective sample size works well even for small numbers of states.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The regressions are weighted by live births in state s and year t, and the standard errors are corrected for clustering at the state level (Bertrand, Duflo, and Mullainathan 2004).”
From Occupational Licensing and Maternal Health: Evidence from Early Midwifery Laws (2020) · cites this paper as Bertrand, Duflo, and Mullainathan 2004
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The regressions are weighted by the female population of state s in year t , and the standard errors are corrected for clustering at the state level (Bertrand et al. 2004).”
From The Effect of Occupational Licensing on Consumer Welfare: Early Midwifery Laws and Maternal Mortality (2016) · cites this paper as Bertrand et al. 2004
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The relevant variance estimators are the Liang-Zeger clustered variance estimator and the clustered bootstrap (Bertrand et al. [2004], Liang and Zeger [1986]). Both have large sample justifications under random sampling from a large population, so they are in general not equal to the variance estimator here.”
From Design-based analysis in Difference-In-Differences settings with staggered adoption (2022) · cites this paper as Bertrand et al. [2004]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from a wide variety of contexts indicates that the asymptotic limiting distribution of robust Wald statistics can be a poor approximation when the number of clusters is small, even if corrections such as CR2 or CR3 are employed (Bell and McCaffrey 2002; Bertrand, Duflo, and Mullainathan 2004; Cameron, Gelbach, and Miller 2008).”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Bertrand, Duflo, and Mullainathan 2004
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Assumption 8 allows for serial correlation of the treatments and outcomes within each group, which are important features to account for in DID studies (Bertrand et al., 2004). Assumption 8 is also weaker than the common assumption that ( D g , Y g ) g ≥ 1 are i.i.d. or even simply independent. In particular, it allows for any form of dependence between groups' treatments ( D g ) g ≥ 1 .”
From Difference-in-Differences Estimators of Intertemporal Treatment Effects (2026) · cites this paper as Bertrand et al., 2004
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2001) · RePEc: Research Papers in Economics · Cited by 27,577
Newey, W. K., & West, K. D. (1987) · Econometrica · doi:10.2307/1913610 · Cited by 16,936
A Simple, Positive Semi-Definite, Heteroskedasticity and AutocorrelationConsistent Covariance Matrix
Newey, W., & West, K. (1986) · National Bureau of Economic Research · doi:10.3386/t0055 · Cited by 12,278
Biases in dynamic models with fixed effects
Nickell, S. (1981) · Econometrica · doi:10.2307/1911408 · Cited by 8,735
Bound, J., Jaeger, D. A., & Baker, R. M. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476536 · Cited by 3,618
An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units
Moulton, B. R. (1990) · The Review of Economics and Statistics · doi:10.2307/2109724 · Cited by 2,630
Semiparametric Difference-in-Differences Estimators
Abadie, A. (2005) · The Review of Economic Studies · doi:10.1111/0034-6527.00321 · Cited by 2,286
MacKinnon, J. G., & White, H. (1985) · Journal of Econometrics · doi:10.1016/0304-4076(85)90158-7 · Cited by 1,879
Natural and Quasi- Experiments in Economics
Meyer, B. (1994) · Journal of Business and Economic Statistics · doi:10.2307/1392369 · Cited by 1,792
The Review of Economics and Statistics
Dorfman, R. (1997) · Economic Theory and Public Decisions · doi:10.4337/9781035334919.00011 · Cited by 1,540
Research works printed but not linked: 4
- Athey Susan, 2002, National Bureau of Economic Research Technical Working Paper No., T0280
- Bertrand Marianne, 2002, National Bureau of Economic Research Working Paper No., 8841
- Greene, William H., Econometric Analysis (New York, NY: Prentice Hall, 2002).
- Solon Gary, 1984, National Bureau of Economic Research Technical Working Paper No., 32
Cited by
Estimating Standard Errors in Finance Panel Data Sets: Comparing Approaches
Petersen, M. A. (2009) · Review of Financial Studies · doi:10.1093/rfs/hhn053 · Cited by 10,746
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Difference-in-differences with variation in treatment timing
Goodman-Bacon, A. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2021.03.014 · Cited by 8,046
The Skill Content of Recent Technological Change: An Empirical Exploration
Autor, D. H., Levy, F., & Murnane, R. J. (2003) · The Quarterly Journal of Economics · doi:10.1162/003355303322552801 · Cited by 6,205
Abadie, A., Diamond, A., & Hainmueller, J. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08746 · Cited by 5,513
Metadata
- Authors
- Bertrand, M., Duflo, E., & Mullainathan, S.
- Affiliations
- National Bureau of Economic Research (US)
- Venue
- The Quarterly Journal of Economics
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-02-01
- Volume
- 119
- Issue
- 1
- Pages
- 249-275
- ISSN
- 0033-5533 (ISSN-L)
- 1531-4650
- OpenAlex
- W2096684396
- Cited by
- 10,641
- References
- 25
- Citations by year
- 2026: 804 · 2025: 672 · 2024: 651 · 2023: 638 · 2022: 698 · 2021: 774 · 2020: 709 · 2019: 578 · 2018: 568 · 2017: 600 · 2016: 524 · 2015: 502 · 2014: 475 · 2013: 435 · 2012: 413
- FWCI
- 157.34
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc
- Locations
- The Quarterly Journal of Economics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Social Sciences › Economics, Econometrics and Finance › General Economics, Econometrics and Finance › Monetary Policy and Economic Impact
- Keywords
- difference-in-differences, standard errors, serial correlation, placebo tests, bootstrapping, covariance matrix, asymptotic variance, effective sample size, women's wages, state-level policy effects, Monte Carlo simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Much Should We Trust Differences-In-Differences Estimates? https://app.cochat.ai/papers/10.1162/003355304772839588
- The paper
- Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-In-Differences Estimates? The Quarterly Journal of Economics. https://doi.org/10.1162/003355304772839588
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.