Characterizing Selection Bias Using Experimental Data
Open accessThe DOI matches its Crossref record.
Abstract
This paper develops and applies semiparametric econometric methods to estimate the form of selection bias that arises from using nonexperimental comparison groups to evaluate social programs and to test the identifying assumptions that justify three widely-used classes of estimators and our extensions of them: (a) the method of matching; (b) the classical econometric selection model which represents the bias solely as a function of the probability of participation; and (c) the method of difference-in-differences. Using data from an experiment on a prototypical social program combined with unusually rich data from a nonexperimental comparison group, we reject the assumptions justifying matching and our extensions of that method but find evidence in support of the index-sufficient selection bias model and the assumptions that justify application of a conditional semiparametric version of the method of difference-in-difference. Fa comparable people and to appropriately weight participants and nonparticipants a sources of selection bias as conveniently measured. We present a rigorous defin bias and find that in our data it is a small component of conventially meausred it is still substantial when compared with experimentally-estimated program impa matching participants to comparison group members in the same labor market, givi same questionnaire, and making sure they have comparable characteristics substan the performance of any econometric program evaluation estimator. We show how t analysis to estimate the impact of treatment on the treated using ordinary obser
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These fall into two main categories, which we denote trajectory modeling [28, 29, 86, 88, 95] and propensity score weighting [17, 30].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [28]
Citations
Cites
Still being checked.
Cited by
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
Pager, D. (2003) · American Journal of Sociology · doi:10.1086/374403 · Cited by 2,840
What's trending in difference-in-differences? A synthesis of the recent econometrics literature
Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.008 · Cited by 2,027
A More Credible Approach to Parallel Trends
Rambachan, A., & Roth, J. (2023) · Review of Economic Studies · doi:10.1093/restud/rdad018 · Cited by 1,559
Using Randomization in Development Economics Research: A Toolkit
Duflo, E., Glennerster, R., & Kremer, M. (2006) · National Bureau of Economic Research · doi:10.3386/t0333 · Cited by 1,191
Doubly robust difference-in-differences estimators
Sant’Anna, P. H. C., & Zhao, J. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.06.003 · Cited by 1,047
Electoral Rules and Corruption
Persson, T., Tabellini, G., & Trebbi, F. (2003) · Journal of the European Economic Association · doi:10.1162/154247603322493203 · Cited by 782
Does Diversification Cause the 'Diversification Discount'?
Villalonga, B. (2000) · SSRN Electronic Journal · doi:10.2139/ssrn.227828 · Cited by 737
Decomposition Methods in Economics
Fortin, N., Lemieux, T., & Firpo, S. (2010) · National Bureau of Economic Research · doi:10.3386/w16045 · Cited by 736
Inference on Counterfactual Distributions
(2013) · Econometrica · doi:10.3982/ecta10582 · Cited by 601
Recent Developments in the Econometrics of Program Evaluation
Imbens, G., & Wooldridge, J. (2008) · National Bureau of Economic Research · doi:10.3386/w14251 · Cited by 533
Evaluating the impact of R&D tax credits on innovation: A microeconometric study on Canadian firms
Czarnitzki, D., Hanel, P., & Rosa, J. M. (2011) · Research Policy · doi:10.1016/j.respol.2010.09.017 · Cited by 522
Evaluating Anti-Poverty Programs
Ravallion, M. (2005) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-3625 · Cited by 507
Identification and Estimation of Triangular Simultaneous Equations Models Without Additivity
(2009) · Econometrica · doi:10.3982/ecta7108 · Cited by 488
Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score
Hirano, K., Imbens, G. W., & Ridder, G. (2002) · SSRN Electronic Journal · doi:10.2139/ssrn.324940 · Cited by 480
Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review
Imbens, G. (2003) · National Bureau of Economic Research · doi:10.3386/t0294 · Cited by 446
Statistical methods with varying coefficient models
Fan, J., & Zhang, W. (2008) · Statistics and Its Interface · doi:10.4310/sii.2008.v1.n1.a15 · Cited by 417
Dechezleprêtre, A., Nachtigall, D., & Venmans, F. (2023) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2022.102758 · Cited by 416
Fifty Years of Mincer Earnings Regressions
Heckman, J., Lochner, L., & Todd, P. (2003) · National Bureau of Economic Research · doi:10.3386/w9732 · Cited by 345
Self-citationSimple and Bias-Corrected Matching Estimators for Average Treatment Effects
Abadie, A., & Imbens, G. (2002) · National Bureau of Economic Research · doi:10.3386/t0283 · Cited by 342
Earnings Functions, Rates of Return and Treatment Effects: The Mincer Equation and Beyond
Heckman, J., Lochner, L., & Todd, P. (2005) · National Bureau of Economic Research · doi:10.3386/w11544 · Cited by 335
Self-citationUnderstanding Instrumental Variables in Models with Essential Heterogeneity
Heckman, J., Urzua, S., & Vytlacil, E. (2006) · National Bureau of Economic Research · doi:10.3386/w12574 · Cited by 329
Self-citationStructural Equations, Treatment Effects and Econometric Policy Evaluation
Heckman, J., & Vytlacil, E. (2005) · National Bureau of Economic Research · doi:10.3386/w11259 · Cited by 326
Self-citation(2010) · Quantitative Economics · doi:10.3982/qe8 · Cited by 304
Self-citationManchanda, P., Packard, G., & Pattabhiramaiah, A. (2015) · Marketing Science · doi:10.1287/mksc.2014.0890 · Cited by 280
The impact of entrepreneurship education in high school on long-term entrepreneurial performance
Elert, N., Andersson, F. W., & Wennberg, K. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2014.12.020 · Cited by 290
Hotz, V. J., Mullin, C. H., & Sanders, S. G. (1997) · The Review of Economic Studies · doi:10.2307/2971732 · Cited by 255
The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics
Marcus, M., & Sant’Anna, P. H. C. (2021) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/711509 · Cited by 192
What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations
Card, D., Kluve, J., & Weber, A. (2015) · doi:10.3386/w21431 · Cited by 136
The Returns to College(s): Relative Value-Added and Match Effects in Higher Education
Mountjoy, J., & Hickman, B. (2020) · doi:10.2139/ssrn.3537773 · Cited by 41
Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods
Feng, S., Ganguli, I., Lee, Y., Poe, J., Ryan, A., & Bilinski, A. (2025) · Statistics in Medicine · doi:10.1002/sim.70247 · Cited by 16
Galdo, V., & Briceño, B. (2005) · doi:10.18235/0011133 · Cited by 13
Metadata
- Authors
- Heckman, J., Ichimura, H., Smith, J., & Todd, P.
- Affiliations
- University of Chicago (US)
- The University of Tokyo (JP)
- National Bureau of Economic Research (US)
- Venue
- National Bureau of Economic Research
- Publisher
- National Bureau of Economic Research
- Published
- 1998-08
- OpenAlex
- W2167047802
- Cited by
- 696
- References
- 0
- Citations by year
- 2026: 9 · 2025: 25 · 2024: 30 · 2023: 23 · 2022: 30 · 2021: 47 · 2020: 48 · 2019: 23 · 2018: 27 · 2017: 29 · 2016: 28 · 2015: 37 · 2014: 30 · 2013: 27 · 2012: 39
- Type
- report
- Language
- en
- Open access
- green
- doi.org
- Locations
- National Bureau of Economic Research (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › School Choice and Performance
- Keywords
- selection bias, program evaluation, matching estimators, difference-in-differences, selection model, social programs, treatment effect estimation, semiparametric estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.