Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

How Much Should We Trust Differences-In-Differences Estimates?

Open access
Bertrand, M., Duflo, E., Mullainathan, S. · 2004 · The Quarterly Journal of Economics

Abstract

Most papers that employ Differences-in-Differences estimation (DD) use many years of data and focus on serially correlated outcomes but ignore that the resulting standard errors are inconsistent. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its “effect” as well as the standard error of this estimate. These conventional DD standard errors severely understate the standard deviation of the estimators: we find an “effect” significant at the 5 percent level for up to 45 percent of the placebo interventions. We use Monte Carlo simulations to investigate how well existing methods help solve this problem. Econometric corrections that place a specific parametric form on the time-series process do not perform well. Bootstrap (taking into account the autocorrelation of the data) works well when the number of states is large enough. Two corrections based on asymptotic approximation of the variance-covariance matrix work well for moderate numbers of states and one correction that collapses the time series information into a “pre”- and “post”-period and explicitly takes into account the effective sample size works well even for small numbers of states.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/003355304772839588

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 4

  1. Athey Susan, 2002, National Bureau of Economic Research Technical Working Paper No., T0280
  2. Bertrand Marianne, 2002, National Bureau of Economic Research Working Paper No., 8841
  3. Greene, William H., Econometric Analysis (New York, NY: Prentice Hall, 2002).
  4. Solon Gary, 1984, National Bureau of Economic Research Technical Working Paper No., 32

Cited by

See all 10,641

Metadata

Authors
Bertrand, M., Duflo, E., & Mullainathan, S.
Affiliations
National Bureau of Economic Research (US)
Venue
The Quarterly Journal of Economics
Published
2004-02-01
Volume
119
Issue
1
Pages
249-275
ISSN
  • 0033-5533 (ISSN-L)
  • 1531-4650
OpenAlex
W2096684396
Cited by
10,641
References
25
Citations by year
2026: 804 · 2025: 672 · 2024: 651 · 2023: 638 · 2022: 698 · 2021: 774 · 2020: 709 · 2019: 578 · 2018: 568 · 2017: 600 · 2016: 524 · 2015: 502 · 2014: 475 · 2013: 435 · 2012: 413
FWCI
157.34
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • The Quarterly Journal of Economics (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
  • Social Sciences › Economics, Econometrics and Finance › General Economics, Econometrics and Finance › Monetary Policy and Economic Impact
Keywords
difference-in-differences, standard errors, serial correlation, placebo tests, bootstrapping, covariance matrix, asymptotic variance, effective sample size, women's wages, state-level policy effects, Monte Carlo simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: How Much Should We Trust Differences-In-Differences Estimates? https://app.cochat.ai/papers/10.1162/003355304772839588
The paper
Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-In-Differences Estimates? The Quarterly Journal of Economics. https://doi.org/10.1162/003355304772839588

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.