Characterizing Selection Bias Using Experimental Data
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
This paper develops and applies semiparametric econometric methods to estimate the form of selection bias that arises from using nonexperimental comparison groups to evaluate social programs and to test the identifying assumptions that justify three widely-used classes of estimators and our extensions of them: (a) the method of matching; (b) the classical econometric selection model which represents the bias solely as a function of the probability of participation; and (c) the method of difference-in-differences. Using data from an experiment on a prototypical social program combined with unusually rich data from a nonexperimental comparison group, we reject the assumptions justifying matching and our extensions of that method but find evidence in support of the index-sufficient selection bias model and the assumptions that justify application of a conditional semiparametric version of the method of difference-in-difference. Fa comparable people and to appropriately weight participants and nonparticipants a sources of selection bias as conveniently measured. We present a rigorous defin bias and find that in our data it is a small component of conventially meausred it is still substantial when compared with experimentally-estimated program impa matching participants to comparison group members in the same labor market, givi same questionnaire, and making sure they have comparable characteristics substan the performance of any econometric program evaluation estimator. We show how t analysis to estimate the impact of treatment on the treated using ordinary obser
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Model selection is conducted through an iterative process, with the primary goal of achieving covariate balance afterwards (Heckman, Ichimura, Smith, & Todd, 1998; Rosenbaum & Rubin, 1984), as opposed to explicit modeling of the selection process.”
From Differential Effects of Adult Court Transfer on Juvenile Offender Recidivism (2010) · cites this paper as Heckman, Ichimura, Smith, & Todd, 1998
In the citing paper
“Inference can be done using the delta-method for parametric models, and there are also several methods available for semi-/non-parametric models (under some additional regularity conditions), including the bootstrap, as described in Heckman et al. (1998).”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Heckman et al. (1998)
In the citing paper
“The identification strategy for the effect of the ETS on innovation has been matching-adjusted DID that proceeded in two steps44,45, used for estimation of both the main policy effects and spillovers.”
From Low-carbon innovation induced by emissions trading in China (2019) · cites this paper as [44]
In the citing paper
“The Achilles heel of the survey methodology is its inability to escape from the glaring problems of selection that plague research in this field (see Winship and Morgan 1999; Rubin 1990; Heckman et al. 1998).”
From The Mark of a Criminal Record (2003) · cites this paper as Heckman et al. 1998
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 101–200 of 1,003
Brock, W. A., & Durlauf, S. N. (2001) · Handbook of Econometrics · doi:10.1016/s1573-4412(01)05007-3 · Cited by 389
Heckman, J. J., & Smith, J. A. (1999) · The Economic Journal · doi:10.1111/1468-0297.00451 · Cited by 386
Self-citationNatural resource abundance and human capital accumulation
Stijns, J.-P. (2006) · World Development · doi:10.1016/j.worlddev.2005.11.005 · Cited by 368
Practical propensity score matching: a reply to Smith and Todd
Dehejia, R. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.012 · Cited by 390
Too Much Ado about Propensity Score Models? Comparing Methods of Propensity Score Matching
Baser, O. (2006) · Value in Health · doi:10.1111/j.1524-4733.2006.00130.x · Cited by 372
Using the Propensity Score Method to Estimate Causal Effects: A Review and Practical Guide
Li, M. (2013) · Organizational Research Methods · doi:10.1177/1094428112447816 · Cited by 366
Heckman, J. J., & Vytlacil, E. J. (2007) · Handbook of Econometrics · doi:10.1016/s1573-4412(07)06071-0 · Cited by 369
Self-citationEndogeneity in nonparametric and semiparametric regression models
Powell, J. L., & Blundell, R. (2001) · Working Paper Series · doi:10.1920/wp.cem.2001.0901 · Cited by 362
LONG-RUN EFFECTS OF PUBLIC SECTOR SPONSORED TRAINING IN WEST GERMANY
Lechner, M., Miquel, R., & Wunsch, C. (2011) · Journal of the European Economic Association · doi:10.1111/j.1542-4774.2011.01029.x · Cited by 371
Hussinger, K. (2008) · Journal of Applied Econometrics · doi:10.1002/jae.1016 · Cited by 370
Behrman, J. R., Cheng, Y., & Todd, P. E. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023714 · Cited by 361
Self-citationNonexperimental Versus Experimental Estimates of Earnings Impacts
Glazerman, S., Levy, D. M., & Myers, D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254879 · Cited by 352
Best Practices in Quasi–Experimental Designs: Matching Methods for Causal Inference
Stuart, E. A., & Rubin, D. B. (2008) · Best Practices in Quantitative Methods · doi:10.4135/9781412995627.d14 · Cited by 335
On young highly innovative companies: why they matter and how (not) to policy support them
Schneider, C., & Veugelers, R. (2010) · Industrial and Corporate Change · doi:10.1093/icc/dtp052 · Cited by 349
Lechner, M. (2002) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/1467-985x.0asp2 · Cited by 369
Eliason, M., & Storrie, D. (2009) · Journal of Human Resources · doi:10.1353/jhr.2009.0020 · Cited by 342
Simple and Bias-Corrected Matching Estimators for Average Treatment Effects
Abadie, A., & Imbens, G. (2002) · National Bureau of Economic Research · doi:10.3386/t0283 · Cited by 342
Carrasco, M., Florens, J.-P., & Renault, E. (2007) · Handbook of Econometrics · doi:10.1016/s1573-4412(07)06077-1 · Cited by 299
Blundell, R., Dearden, L., & Sianesi, B. (2005) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2004.00360.x · Cited by 343
Chapter 7 Earnings Functions, Rates of Return and Treatment Effects: The Mincer Equation and Beyond
Heckman, J. J., Lochner, L. J., & Todd, P. E. (2006) · Handbook of the Economics of Education · doi:10.1016/s1574-0692(06)01007-5 · Cited by 328
Self-citationThe performance of estimators based on the propensity score
Huber, M., Lechner, M., & Wunsch, C. (2013) · Journal of Econometrics · doi:10.1016/j.jeconom.2012.11.006 · Cited by 311
Chapter 16 Schools, Teachers, and Education Outcomes in Developing Countries
Glewwe, P., & Kremer, M. (2006) · Handbook of the Economics of Education · doi:10.1016/s1574-0692(06)02016-2 · Cited by 323
Micro-Econometrics for Policy, Program and Treatment Effects
Lee, M.-J. (2005) · doi:10.1093/0199267693.001.0001 · Cited by 339
Earnings Functions, Rates of Return and Treatment Effects: The Mincer Equation and Beyond
Heckman, J., Lochner, L., & Todd, P. (2005) · National Bureau of Economic Research · doi:10.3386/w11544 · Cited by 335
Self-citationAttanasio, O., Kugler, A., & Meghir, C. (2011) · American Economic Journal: Applied Economics · doi:10.1257/app.3.3.188 · Cited by 332
Structural Equations, Treatment Effects and Econometric Policy Evaluation
Heckman, J., & Vytlacil, E. (2005) · National Bureau of Economic Research · doi:10.3386/w11259 · Cited by 326
Self-citationReconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods
Smith, J. A., & Todd, P. E. (2001) · American Economic Review · doi:10.1257/aer.91.2.112 · Cited by 331
Self-citationThe Real Effects of Credit Ratings: The Sovereign Ceiling Channel
ALMEIDA, H., CUNHA, I., FERREIRA, M. A., & RESTREPO, F. (2017) · The Journal of Finance · doi:10.1111/jofi.12434 · Cited by 336
Recent developments in the econometrics of program evaluation
Wooldridge, J. M., & Imbens, G. (2008) · doi:10.1920/wp.cem.2008.2408 · Cited by 283
The use of propensity score matching in the evaluation of active labour market policies
Bryson, A., Dorsett, R., & Purdon, S. (2002) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · Cited by 320
Ito, J., Bao, Z., & Su, Q. (2012) · Food Policy · doi:10.1016/j.foodpol.2012.07.009 · Cited by 287
Opiates for the Matches: Matching Methods for Causal Inference
Sekhon, J. S. (2009) · Annual Review of Political Science · doi:10.1146/annurev.polisci.11.060606.135444 · Cited by 319
Fowlie, M., Holland, S. P., & Mansur, E. T. (2012) · American Economic Review · doi:10.1257/aer.102.2.965 · Cited by 272
Exploring the relationship between HRM and firm performance: A meta-analysis of longitudinal studies
Saridakis, G., Lai, Y., & Cooper, C. L. (2017) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.09.005 · Cited by 293
(2010) · Quantitative Economics · doi:10.3982/qe8 · Cited by 304
Self-citationLASSEN, D. D., & SERRITZLEW, S. (2011) · American Political Science Review · doi:10.1017/s000305541100013x · Cited by 305
Start-up subsidies for the unemployed: Long-term evidence and effect heterogeneity
Caliendo, M., & Künn, S. (2011) · Journal of Public Economics · doi:10.1016/j.jpubeco.2010.11.003 · Cited by 310
Volpe Martincus, C., & Carballo, J. (2008) · Journal of International Economics · doi:10.1016/j.jinteco.2008.05.002 · Cited by 296
International Institutions and Issue Linkage: Building Support for Agricultural Trade Liberalization
DAVIS, C. L. (2004) · American Political Science Review · doi:10.1017/s0003055404001066 · Cited by 303
Abid, M., Schneider, U. A., & Scheffran, J. (2016) · Journal of Rural Studies · doi:10.1016/j.jrurstud.2016.08.005 · Cited by 296
The Role of the Propensity Score in Estimating Dose-Response Functions
Imbens, G. (1999) · National Bureau of Economic Research · doi:10.3386/t0237 · Cited by 290
Bounding a Matching Estimator: The Case of a Norwegian Training Program
Aakvik, A. (2001) · Oxford Bulletin of Economics and Statistics · doi:10.1111/1468-0084.00211 · Cited by 296
Lasting or Latent Scars? Swedish Evidence on the Long‐Term Effects of Job Displacement
Eliason, M., & Storrie, D. (2006) · Journal of Labor Economics · doi:10.1086/506487 · Cited by 287
(2011) · The University of Bath Online Publications Store (The University of Bath) · Cited by 284
Lin, X. (2010) · Journal of Labor Economics · doi:10.1086/653506 · Cited by 291
How to do a good systematic review of effects in international development: a tool kit
Waddington, H., White, H., Snilstveit, B., Hombrados, J. G., Vojtkova, M., Davies, P., Bhavsar, A., Eyers, J., Koehlmoos, T. P., Petticrew, M., et al. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711765 · Cited by 278
Manchanda, P., Packard, G., & Pattabhiramaiah, A. (2015) · Marketing Science · doi:10.1287/mksc.2014.0890 · Cited by 280
The impact of entrepreneurship education in high school on long-term entrepreneurial performance
Elert, N., Andersson, F. W., & Wennberg, K. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2014.12.020 · Cited by 290
ABATE, G. T., FRANCESCONI, G. N., & GETNET, K. (2014) · Annals of Public and Cooperative Economics · doi:10.1111/apce.12035 · Cited by 257
The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation
Ravallion, M. (2001) · The World Bank Economic Review · doi:10.1093/wber/15.1.115 · Cited by 258
The Returns to Higher Education in Britain: Evidence From a British Cohort
Blundell, R., Dearden, L., Goodman, A., & Reed, H. (2000) · The Economic Journal · doi:10.1111/1468-0297.00508 · Cited by 270
Aakvik, A., Heckman, J. J., & Vytlacil, E. J. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.002 · Cited by 274
Self-citationHosanagar, K., Fleder, D., Lee, D., & Buja, A. (2014) · Management Science · doi:10.1287/mnsc.2013.1808 · Cited by 264
Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment
Heckman, J., Hohmann, N., Smith, J., & Khoo, M. (2000) · The Quarterly Journal of Economics · doi:10.1162/003355300554764 · Cited by 266
Self-citationQuantifying the Life-Cycle Benefits of an Influential Early-Childhood Program
García, J. L., Heckman, J. J., Leaf, D. E., & Prados, M. J. (2020) · Journal of Political Economy · doi:10.1086/705718 · Cited by 247
Self-citationDoes knowledge diffusion between university and industry increase innovativeness?
Lööf, H., & Broström, A. (2008) · The Journal of Technology Transfer · doi:10.1007/s10961-006-9001-3 · Cited by 264
Cook, T. D. (2002) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737024003175 · Cited by 257
When Can History Be Our Guide? The Pitfalls of Counterfactual Inference
KING, G., & ZENG, L. (2007) · International Studies Quarterly · doi:10.1111/j.1468-2478.2007.00445.x · Cited by 256
de Brauw, A., & Hoddinott, J. (2011) · Journal of Development Economics · doi:10.1016/j.jdeveco.2010.08.014 · Cited by 210
Propensity scores: From naïve enthusiasm to intuitive understanding
Williamson, E., Morley, R., Lucas, A., & Carpenter, J. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280210394483 · Cited by 254
Nonparametric Density and Regression Estimation
DiNardo, J., & Tobias, J. L. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.11 · Cited by 257
Building Programme Evaluation into the Design of Public Research-Support Programmes
Jaffe, A. B. (2002) · Oxford Review of Economic Policy · doi:10.1093/oxrep/18.1.22 · Cited by 239
Opportunity Counts: Teams and the Effectiveness of Production Incentives
Boning, B., Ichniowski, C., & Shaw, K. (2007) · Journal of Labor Economics · doi:10.1086/519539 · Cited by 243
The Constraining Power of International Treaties: Theory and Methods
SIMMONS, B. A., & HOPKINS, D. J. (2005) · American Political Science Review · doi:10.1017/s0003055405051920 · Cited by 237
Chen, Y., & Jin, G. Z. (2012) · Journal of Health Economics · doi:10.1016/j.jhealeco.2011.11.001 · Cited by 231
Does eco-certification have environmental benefits? Organic coffee in Costa Rica
Blackman, A., & Naranjo, M. A. (2012) · Ecological Economics · doi:10.1016/j.ecolecon.2012.08.001 · Cited by 234
Remittances and natural disasters: ex-post response and contribution to ex-ante preparedness
Mohapatra, S., Joseph, G., & Ratha, D. (2012) · Environment, Development and Sustainability · doi:10.1007/s10668-011-9330-8 · Cited by 229
Does high involvement management improve worker wellbeing?
Böckerman, P., Bryson, A., & Ilmakunnas, P. (2012) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2012.09.005 · Cited by 226
Bailey, M. J., & Goodman-Bacon, A. (2015) · American Economic Review · doi:10.1257/aer.20120070 · Cited by 227
The Labor Market Impacts of Youth Training in the Dominican Republic
Card, D., Ibarrarán, P., Regalia, F., Rosas-Shady, D., & Soares, Y. (2011) · Journal of Labor Economics · doi:10.1086/658090 · Cited by 225
“…U.S. case, particularly credible evidence is available from randomized evaluations of JTPA (see General Accounting Office 1996; Bloom et al. 1997; Heckman et al. 1998; Heckman, Ichimura, and Todd 1998), Job Corps (Schochet, McConnell, and Burghardt 2003), and a series of programs for welfare…”
BackgroundInfluentialPropensity score matching and variations on the balancing test
Lee, W.-S. (2013) · Empirical Economics · doi:10.1007/s00181-011-0481-0 · Cited by 224
Gender Wage Disparities among the Highly Educated
Black, D. A., Haviland, A. M., Sanders, S. G., & Taylor, L. J. (2008) · Journal of Human Resources · doi:10.1353/jhr.2008.0030 · Cited by 228
Raising the Bar: Curricular Intensity and Academic Performance
Attewell, P., & Domina, T. (2008) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707313409 · Cited by 223
(International) R&D collaboration and SMEs: The effectiveness of targeted public R&D support schemes
Hottenrott, H., & Lopes-Bento, C. (2014) · Research Policy · doi:10.1016/j.respol.2014.01.004 · Cited by 216
LOUGHRAN, T. A., MULVEY, E. P., SCHUBERT, C. A., FAGAN, J., PIQUERO, A. R., & LOSOYA, S. H. (2009) · Criminology · doi:10.1111/j.1745-9125.2009.00165.x · Cited by 218
IMAI, K. (2005) · American Political Science Review · doi:10.1017/s0003055405051658 · Cited by 215
Program evaluation as a decision problem
Dehejia, R. H. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.006 · Cited by 214
How Well Do Automated Linking Methods Perform? Lessons from US Historical Data
Bailey, M. J., Cole, C., Henderson, M., & Massey, C. (2020) · Journal of Economic Literature · doi:10.1257/jel.20191526 · Cited by 175
Corporate divestitures and family control
Feldman, E. R., Amit, R., & Villalonga, B. (2016) · Strategic Management Journal · doi:10.1002/smj.2329 · Cited by 203
Abate, G. T., Rashid, S., Borzaga, C., & Getnet, K. (2016) · World Development · doi:10.1016/j.worlddev.2016.03.003 · Cited by 199
Value for money? New microeconometric evidence on public R&D grants in Flanders
Czarnitzki, D., & Lopes-Bento, C. (2013) · Research Policy · doi:10.1016/j.respol.2012.04.008 · Cited by 202
Czarnitzki, D., & Lopes-Bento, C. (2014) · Industry and Innovation · doi:10.1080/13662716.2014.973246 · Cited by 206
Criminal Recidivism after Prison and Electronic Monitoring
Di Tella, R., & Schargrodsky, E. (2013) · Journal of Political Economy · doi:10.1086/669786 · Cited by 206
Gilligan, D. O., & Hoddinott, J. (2007) · American Journal of Agricultural Economics · doi:10.1111/j.1467-8276.2007.00992.x · Cited by 190
Voluntary environmental regulation in developing countries: Mexico’s Clean Industry Program
Blackman, A., Lahiri, B., Pizer, W., Rivera Planter, M., & Muñoz Piña, C. (2010) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2010.05.006 · Cited by 196
Heckman, J. J., & Smith, J. A. (2004) · Journal of Labor Economics · doi:10.1086/381250 · Cited by 197
Self-citationDiaz, J. J., & Handa, S. (2006) · Journal of Human Resources · doi:10.3368/jhr.xli.2.319 · Cited by 196
Michalopoulos, C., Bloom, H. S., & Hill, C. J. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023732 · Cited by 199
Darendeli, A., Fiechter, P., Hitz, J.-M., & Lehmann, N. (2022) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2022.101525 · Cited by 193
When Natural Experiments Are Neither Natural nor Experiments
SEKHON, J. S., & TITIUNIK, R. (2012) · American Political Science Review · doi:10.1017/s0003055411000542 · Cited by 201
Environmental Policy and Directed Technological Change: Evidence from the European Carbon Market
Calel, R., & Dechezleprêtre, A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2041147 · Cited by 191
Panel Data Analysis - Advantages and Challenges
Hsiao, C. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.902657 · Cited by 186
The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics
Marcus, M., & Sant’Anna, P. H. C. (2021) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/711509 · Cited by 192
Estimating returns to soil conservation adoption in the northern Ethiopian highlands
Kassie, M., Pender, J., Yesuf, M., Kohlin, G., Bluffstone, R., & Mulugeta, E. (2008) · Agricultural Economics · doi:10.1111/j.1574-0862.2007.00295.x · Cited by 193
Predicting the efficacy of future training programs using past experiences at other locations
Joseph Hotz, V., Imbens, G. W., & Mortimer, J. H. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.009 · Cited by 190
Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects
(2018) · doi:10.48550/arxiv.1804.05785 · Cited by 189
Colombo, M. G., & Piva, E. (2012) · Research Policy · doi:10.1016/j.respol.2011.08.010 · Cited by 190
The impact of firm participation in R&D programmes on R&D partnerships
Busom, I., & Fernández-Ribas, A. (2008) · Research Policy · doi:10.1016/j.respol.2007.11.002 · Cited by 185
Lupu, Y. (2013) · International Organization · doi:10.1017/s002081831300012x · Cited by 186
Winners or Losers? The Effects of Banking Consolidation on Corporate Borrowers
DI PATTI, E. B., & GOBBI, G. (2007) · The Journal of Finance · doi:10.1111/j.1540-6261.2007.01220.x · Cited by 191
Metadata
- Authors
- Heckman, J., Ichimura, H., Smith, J., & Todd, P.
- Venue
- Econometrica
- Publisher
- JSTOR
- Published
- 1998-09
- Volume
- 66
- Issue
- 5
- Pages
- 1017
- ISSN
- 0012-9682 (ISSN-L)
- OpenAlex
- W3021176487
- Cited by
- 2,286
- References
- 0
- Citations by year
- 2026: 27 · 2025: 49 · 2024: 51 · 2023: 68 · 2022: 95 · 2021: 70 · 2020: 89 · 2019: 72 · 2018: 84 · 2017: 98 · 2016: 90 · 2015: 120 · 2014: 116 · 2013: 120 · 2012: 133
- FWCI
- 30.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › School Choice and Performance
- Keywords
- selection bias, program evaluation, matching estimators, difference-in-differences, selection model, social programs, treatment effect estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.