Characterizing Selection Bias Using Experimental Data
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
This paper develops and applies semiparametric econometric methods to estimate the form of selection bias that arises from using nonexperimental comparison groups to evaluate social programs and to test the identifying assumptions that justify three widely-used classes of estimators and our extensions of them: (a) the method of matching; (b) the classical econometric selection model which represents the bias solely as a function of the probability of participation; and (c) the method of difference-in-differences. Using data from an experiment on a prototypical social program combined with unusually rich data from a nonexperimental comparison group, we reject the assumptions justifying matching and our extensions of that method but find evidence in support of the index-sufficient selection bias model and the assumptions that justify application of a conditional semiparametric version of the method of difference-in-difference. Fa comparable people and to appropriately weight participants and nonparticipants a sources of selection bias as conveniently measured. We present a rigorous defin bias and find that in our data it is a small component of conventially meausred it is still substantial when compared with experimentally-estimated program impa matching participants to comparison group members in the same labor market, givi same questionnaire, and making sure they have comparable characteristics substan the performance of any econometric program evaluation estimator. We show how t analysis to estimate the impact of treatment on the treated using ordinary obser
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Model selection is conducted through an iterative process, with the primary goal of achieving covariate balance afterwards (Heckman, Ichimura, Smith, & Todd, 1998; Rosenbaum & Rubin, 1984), as opposed to explicit modeling of the selection process.”
From Differential Effects of Adult Court Transfer on Juvenile Offender Recidivism (2010) · cites this paper as Heckman, Ichimura, Smith, & Todd, 1998
In the citing paper
“Inference can be done using the delta-method for parametric models, and there are also several methods available for semi-/non-parametric models (under some additional regularity conditions), including the bootstrap, as described in Heckman et al. (1998).”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Heckman et al. (1998)
In the citing paper
“The identification strategy for the effect of the ETS on innovation has been matching-adjusted DID that proceeded in two steps44,45, used for estimation of both the main policy effects and spillovers.”
From Low-carbon innovation induced by emissions trading in China (2019) · cites this paper as [44]
In the citing paper
“The Achilles heel of the survey methodology is its inability to escape from the glaring problems of selection that plague research in this field (see Winship and Morgan 1999; Rubin 1990; Heckman et al. 1998).”
From The Mark of a Criminal Record (2003) · cites this paper as Heckman et al. 1998
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 901–1,000 of 1,003
Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic
Dehejia, R. (2015) · Journal of Globalization and Development · doi:10.1515/jgd-2014-0005 · Cited by 36
Tyler, J. H. (2004) · Industrial and Labor Relations Review · doi:10.1177/001979390405700406 · Cited by 40
Employment effects of a payroll-tax cut - evidence from a regional tax exemption experimentPreprint
Korkeamäki, O., & Uusitalo, R. (2006) · RePEc: Research Papers in Economics · Cited by 32
Asymptotic expansions for some semiparametric program evaluation estimators
Linton, O., & Ichimura, H. (2001) · Cemmap working papers · doi:10.1920/wp.cem.2001.0401 · Cited by 24
Self-citationStatistical treatment choice: an application to active labour market programmes
Frölich, M. (2006) · Cemmap working papers · doi:10.1920/wp.cem.2006.2406 · Cited by 24
The effect of education policy on crime: an intergenerational perspective
Meghir, C., Palme, M. O., & Schnabel, M. (2011) · Working paper series - Institute for Fiscal Studies/Working papers · doi:10.1920/wp.ifs.2011.1111 · Cited by 26
Höfler, R., Elston, J. A., & Lee, J. (2004) · RePEc: Research Papers in Economics · Cited by 24
Horizontal Mismatch in the Labour Market of Graduates: The Role of SignallingPreprint
Domadenik, P., Farčnik, D., & Pastore, F. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2314822 · Cited by 24
Macura, B., Secco, L., Pisani, E., Pullin, A. S., & Reyes‐García, V. (2016) · Regional Environmental Change · doi:10.1007/s10113-016-0978-3 · Cited by 23
The Influence of High Schools on Developing Public Service Motivation
Holt, S. (2018) · International Public Management Journal · doi:10.1080/10967494.2018.1470120 · Cited by 24
Liu, R., Gao, Z., Nayga, R. M., Shi, L., Oxley, L., & Ma, H. (2020) · Agribusiness · doi:10.1002/agr.21641 · Cited by 24
A Return-on-Investment Analysis of the Health Promotion Program At the University of Minnesota
Nyman, J. A., Barleen, N. A., & Dowd, B. E. (2009) · Journal of Occupational and Environmental Medicine · doi:10.1097/jom.0b013e31818aab8d · Cited by 24
da Cunha, D. A., Coelho, A. B., Féres, J. G., Braga, M. J., & de Souza, E. C. (2013) · Revista de Economia e Sociologia Rural · doi:10.1590/s0103-20032013000200009 · Cited by 23
Start‐up subsidies in East Germany: finally, a policy that works?
Caliendo, M. (2009) · International Journal of Manpower · doi:10.1108/01437720910997326 · Cited by 31
Białowolski, P., & Węziak‐Białowolska, D. (2016) · Social Indicators Research · doi:10.1007/s11205-016-1363-9 · Cited by 26
Statin Use and Gastrointestinal Hemorrhage: A Large Retrospective Cohort Study
Martinez, A. I., Freeman, P. R., & Moga, D. C. (2018) · American Journal of Cardiovascular Drugs · doi:10.1007/s40256-018-0301-4 · Cited by 26
Demographic responses to a political transformation: Evidence of women’s empowerment from Nepal
Paudel, J., & de Araújo, P. (2016) · Journal of Comparative Economics · doi:10.1016/j.jce.2016.07.003 · Cited by 26
Job creation through public start-up assistance?
Almus, M. (2004) · Applied Economics · doi:10.1080/0003684042000201802 · Cited by 32
Using propensity matching estimators to evaluate the impact of privatization on wages
Monteiro, N. P. (2008) · Applied Economics · doi:10.1080/00036840701721281 · Cited by 24
Plesca, M., & Smith, J. A. (2007) · Empirical Economics · doi:10.1007/s00181-006-0095-0 · Cited by 32
Self-citationHAS INFLATION TARGETING CHANGED THE CONDUCT OF MONETARY POLICY?
Creel, J., & Hubert, P. (2013) · Macroeconomic Dynamics · doi:10.1017/s1365100513000199 · Cited by 23
Effect of input credit on smallholder farmers’ output and income: Evidence from Northern Ghana
Iddrisu, A., Ansah, I. G. K., & Nkegbe, P. K. (2017) · Agricultural Finance Review · doi:10.1108/afr-05-2017-0032 · Cited by 28
Fitzenberger, B., & Speckesser, S. (2003) · RePEc: Research Papers in Economics · Cited by 23
Nepotism, incentives and the academic success of college students
Gevrek, D., & Gevrek, Z. E. (2009) · Labour Economics · doi:10.1016/j.labeco.2009.11.001 · Cited by 25
Employment effects of the provision of specific professional skills and techniques in Germany
Fitzenberger, B., & Speckesser, S. (2008) · Physica-Verlag eBooks · doi:10.1007/978-3-7908-2022-5_14 · Cited by 23
Tey, Y. S., Brindal, M., Hadi, A. H. I. A., & Darham, S. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2021.12.020 · Cited by 30
Nonvide, G. M. A. (2022) · Economics of Innovation and New Technology · doi:10.1080/10438599.2022.2144846 · Cited by 22
Impact Evaluation of Chile Solidario: Lessons and Policy Recommendations
Larrañaga, O., Contreras, D., & Ruiz‐Tagle, J. (2012) · Journal of Latin American Studies · doi:10.1017/s0022216x12000053 · Cited by 22
Programs for Homeless Children and Youth: A Critical Review of Evidence
Herbers, J. E., & Cutuli, J. J. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8718-0_10 · Cited by 24
Do the equity holding and soundness of bank underwriters affect issue costs of SEOs?
Suzuki, K. (2009) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2009.10.011 · Cited by 22
Sensitivity analysis of job-training effects on reemployment for Korean women
Lee, M., & Lee, S. J. (2008) · Empirical Economics · doi:10.1007/s00181-008-0188-z · Cited by 22
Faulkner, W. N. (2014) · Evaluation · doi:10.1177/1356389014528602 · Cited by 28
The role of the housing market in workers′ resilience to job displacement after firm bankruptcy
Meekes, J., & Hassink, W. (2018) · Journal of Urban Economics · doi:10.1016/j.jue.2018.11.005 · Cited by 29
Breidenbach, P., & Mitze, T. (2021) · Econometrics Journal · doi:10.1093/ectj/utab021 · Cited by 26
Wynen, J., Boon, J., & Verlinden, S. (2022) · Public Performance & Management Review · doi:10.1080/15309576.2022.2037003 · Cited by 34
Distance functions for matching in small samples
Dettmann, E., Becker, C., & Schmeißer, C. (2010) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.11.022 · Cited by 26
Chapter 62 Evaluating Conditional Schooling and Health Programs
Parker, S. W., Rubalcava, L., & Teruel, G. (2007) · Handbook of development economics · doi:10.1016/s1573-4471(07)04062-4 · Cited by 25
Who Benefits from R&D Tax Policy?
Martínez-Azúa, B. C., & Ros, E. M. (2010) · Cuadernos de Economía y Dirección de la Empresa · doi:10.1016/s1138-5758(10)70027-x · Cited by 42
Do reductions of standard hours affect employment transitions?: Evidence from Chile
Sánchez, R. M. (2012) · Labour Economics · doi:10.1016/j.labeco.2012.10.001 · Cited by 22
Does Group Affiliation Improve Firm Performance? The Case of Chinese State-Owned Firms
Yu, H., van Ees, H., & Lensink, R. (2009) · The Journal of Development Studies · doi:10.1080/00220380902952365 · Cited by 22
Donegan, S., Maluccio, J. A., Myers, C. K., Menon, P., Ruel, M. T., & Habicht, J. (2010) · Journal of Nutrition · doi:10.3945/jn.109.114272 · Cited by 27
Gutiérrez, J. P., García-Saisó, S., & Aracena-Genao, B. (2018) · PLoS ONE · doi:10.1371/journal.pone.0201333 · Cited by 22
Gamoran, A., Collares, A. C. M., & Barfels, S. E. (2016) · American Journal of Sociology · doi:10.1086/683605 · Cited by 26
Does 'Work for the Dole' Work?Preprint
Borland, J., & Tseng, Y. (2004) · RePEc: Research Papers in Economics · Cited by 51
An Evaluation of the Adult Education Initiative Relative Labor Market TrainingPreprint
Stenberg, A. (2003) · RePEc: Research Papers in Economics · Cited by 22
How to Examine External Validity Within an Experiment
Kowalski, A. (2018) · National Bureau of Economic Research · doi:10.3386/w24834 · Cited by 28
Start-Up Subsidies for the Unemployed: Long-Term Evidence and Effect HeterogeneityPreprint
Caliendo, M., & Künn, S. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1633837 · Cited by 28
Innovation in immigrant-owned firms
Ostrovsky, Y., & Picot, G. (2021) · Small Business Economics · doi:10.1007/s11187-020-00376-2 · Cited by 21
Treatment Effect Heterogeneity
Smith, J. A. (2022) · Evaluation Review · doi:10.1177/0193841x221090731 · Cited by 30
Self-citationLong Live Friendship? The Long-Term Impact of Soviet Aid on Sino-Russian Trade
Hu, Z.-A., Li, J., & Nie, Z. (2021) · Journal of Development Economics · doi:10.1016/j.jdeveco.2023.103117 · Cited by 26
Lokshin, M. M., & Yemtsov, R. (2004) · Journal of Human Development · doi:10.1080/1464988042000225168 · Cited by 21
Colaresi, M. P. (2014) · Journal of Peace Research · doi:10.1177/0022343313512851 · Cited by 24
Impact of Farmer Mentorship Project on Farm Efficiency and Income in Rural Ghana
Martey, E., Dogbe, W., Etwire, P. M., & Wiredu, A. N. (2015) · Journal of Agricultural Science · doi:10.5539/jas.v7n10p79 · Cited by 28
From pay-per-bag to pay-per-kg: The case of Flanders revisited
Jaeger, S. D., & Eyckmans, J. (2015) · Waste Management & Research The Journal for a Sustainable Circular Economy · doi:10.1177/0734242x15610422 · Cited by 21
No student left behind? Evidence from the Programme for School Guidance in Spain
Pérez, J. I. G., & Hidalgo‐Hidalgo, M. (2017) · Economics of Education Review · doi:10.1016/j.econedurev.2017.08.006 · Cited by 26
Re-evaluating Microfinance: Evidence from Propensity Score Matching
Cintina, I., & Love, I. (2016) · The World Bank Economic Review · doi:10.1093/wber/lhw069 · Cited by 24
Zeray, N. G., Legesse, B., Mohamed, J. H., & Ketema, M. (2017) · Pastoralism Research Policy and Practice · doi:10.1186/s13570-017-0079-z · Cited by 21
Becoming self-employed from inactivity: an in-depth analysis of satisfaction
Justo, R., Congregado, E., & Román, C. (2019) · Small Business Economics · doi:10.1007/s11187-019-00212-2 · Cited by 28
Financing constraints and exports: evidence from India
Padmaja, M., & Sasidharan, S. (2020) · Journal of Economics and Finance · doi:10.1007/s12197-020-09532-6 · Cited by 23
Impact of business innovation on future employment in Nigeria
Osabohien, R. A., Worgwu, H., Rafi, S. K., Adediran, O. S., Matthew, O. A., & Aderounmu, B. (2022) · Managerial and Decision Economics · doi:10.1002/mde.3629 · Cited by 21
Améstica-Rivas, L. R., Llinas-Audet, X., & Sánchez, I. (2014) · Formación universitaria · doi:10.4067/s0718-50062014000300004 · Cited by 24
Wong, H., Guo, S., Chen, M., & Ip, W.-C. (2009) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2009.01.016 · Cited by 21
Does the information content of payout initiations and omissions influence firm risks?
von Eije, H., Goyal, A., & Muckley, C. B. (2014) · Journal of Econometrics · doi:10.1016/j.jeconom.2014.05.012 · Cited by 21
Temporary/Permanent Workers' Wage Gap: A Brand‐new Form of Wage Inequality?
Elia, L. (2010) · Labour · doi:10.1111/j.1467-9914.2010.00478.x · Cited by 21
Treatment Evaluation With Selective Participation and Ineligibles
Dias, M. C., Ichimura, H., & van den Berg, G. J. (2013) · Journal of the American Statistical Association · doi:10.1080/01621459.2013.795447 · Cited by 27
Self-citationHartley, R. D. (2014) · Wiley eBooks · doi:10.1002/9781118517390.wbetc148 · Cited by 21
Lokshin, M. M., & Yemtsov, R. (2003) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-3155 · Cited by 21
López, F., & Maffioli, A. (2008) · doi:10.18235/0011179 · Cited by 21
The Echo of Job DisplacementPreprint
Eliason, M., & Storrie, D. (2003) · SSRN Electronic Journal · doi:10.2139/ssrn.455420 · Cited by 21
Doubly Robust Difference-in-Differences EstimatorsPreprint
Sant’Anna, P. H. C., & Zhao, J. B. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3293315 · Cited by 28
Brown, J. D., Earle, J. S., & Lup, D. (2004) · Upjohn Research (W.E. Upjohn Institute for Employment Research) · doi:10.17848/wp03-94 · Cited by 21
Augsburg, B. (2006) · doi:10.1920/re.ifs.2024.0736 · Cited by 21
From Giving Birth to Paid Labor: The Effects of Adult Education for Prime-Aged MothersPreprint
Bergemann, A., & van den Berg, G. J. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1184010 · Cited by 21
Jing, K., & Liao, W. (2023) · Regional Science and Urban Economics · doi:10.1016/j.regsciurbeco.2023.103897 · Cited by 22
Impacts of improved agricultural technologies on food security and child nutrition in rural Ethiopia
Habtewold, T. M., & Heshmati, A. (2023) · Cogent Food & Agriculture · doi:10.1080/23311932.2023.2276565 · Cited by 22
Socioeconomic impacts of solar home systems in rural Ethiopia
Beyene, A. D., Mekonnen, A., Jeuland, M., & Czakon, S. (2023) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2023.114197 · Cited by 24
The performance of sample selection estimators to control for attrition bias
Grasdal, A. L. (2001) · Health Economics · doi:10.1002/hec.628 · Cited by 25
Fernández‐Ribas, A., & Shapira, P. (2009) · International Journal of Technology Management · doi:10.1504/ijtm.2009.026690 · Cited by 27
Stepping-stones, dead-ends, or both? An analysis of Swedish replacement contracts
Hartman, L., Liljeberg, L., & Skans, O. N. (2009) · Empirical Economics · doi:10.1007/s00181-009-0283-9 · Cited by 24
(2003) · Cited by 20
Making Valid Causal Inferences About Corrective Actions by Parents From Longitudinal Data
Larzelere, R. E., & Cox, R. B. (2013) · Journal of Family Theory & Review · doi:10.1111/jftr.12020 · Cited by 22
Olarinde, L. O., Oduol, J., Binam, J., Diagne, A., & Njuki, J. (2011) · Middle-East Journal of Scientific Research · Cited by 20
Lean manufacturing and productivity changes: the moderating role of R&D
Shi, Y., Wang, X., & Zhu, X. (2019) · International Journal of Productivity and Performance Management · doi:10.1108/ijppm-03-2018-0117 · Cited by 29
Integrated Value Chain Development: Evidence from Bangladesh
Kuijpers, R. (2020) · Food Policy · doi:10.1016/j.foodpol.2020.101916 · Cited by 25
Zhong, H., Li, W., & Boarnet, M. G. (2020) · Environment and Planning B Urban Analytics and City Science · doi:10.1177/2399808320982305 · Cited by 21
Dey, M. M., Kambewa, P., Prein, M., Jamu, D. M., Paraguas, F. J., Briones, R., & Pemsl, D. E. (2007) · CABI Publishing eBooks · doi:10.1079/9781845932831.0118 · Cited by 20
Testing Conditional Independence Restrictions
Linton, O., & Gozalo, P. L. (2013) · Econometric Reviews · doi:10.1080/07474938.2013.825135 · Cited by 20
Evaluation of the Reggio approach to early education
Biroli, P., Boca, D. D., Heckman, J. J., Heckman, L. P., Koh, Y. K., Kuperman, S., Moktan, S., Pronzato, C., & Ziff, A. L. (2017) · Research in Economics · doi:10.1016/j.rie.2017.05.006 · Cited by 36
Self-citationEvaluating active labour market programmes in New Zealand
Perry, G., & Maloney, T. (2007) · International Journal of Manpower · doi:10.1108/01437720710733447 · Cited by 23
Group-specific Effects of Inter‐regional Mobility on Earnings – A Microdata Analysis for Germany
Lehmer, F., & Möller, J. (2008) · Regional Studies · doi:10.1080/00343400701543686 · Cited by 28
Effectiveness of One-Euro-Jobs: do programme characteristics matter?
Hohmeyer, K. (2011) · Applied Economics · doi:10.1080/00036846.2011.591734 · Cited by 22
Evaluating the impact of a mandatory job search program: evidence from a large longitudinal dataset
Centeno, L., Centeno, M. J. G. D. F., & Novo, Á. A. (2004) · Cited by 20
Hujer, R., & Wellner, M. (2003) · RePEc: Research Papers in Economics · Cited by 20
Identification and Estimation of 'Irregular' Correlated Random Coefficient Models
Graham, B. S., & Powell, J. (2008) · National Bureau of Economic Research · doi:10.3386/w14469 · Cited by 20
Propensity Score Matching, a Distance-Based Measure of Migration, and the Wages of Young Men ∗Preprint
Ham, J. C., Li, X., & Reagan, P. B. (2006) · RePEc: Research Papers in Economics · Cited by 22
Card, D., & de la Rica, S. (2005) · National Bureau of Economic Research · doi:10.3386/w11829 · Cited by 24
In sight, in mind: Spatial proximity to protest sites and changes in peoples' political attitudes
Xu, D., & Guo, J. (2023) · British Journal of Sociology · doi:10.1111/1468-4446.12988 · Cited by 24
Kirksey, J. J. (2024) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737241288838 · Cited by 23
Jaeger, S. D., & Eyckmans, J. (2007) · Waste Management · doi:10.1016/j.wasman.2007.08.007 · Cited by 22
Harknett, K. (2006) · Evaluation Review · doi:10.1177/0193841x06293411 · Cited by 19
Metadata
- Authors
- Heckman, J., Ichimura, H., Smith, J., & Todd, P.
- Venue
- Econometrica
- Publisher
- JSTOR
- Published
- 1998-09
- Volume
- 66
- Issue
- 5
- Pages
- 1017
- ISSN
- 0012-9682 (ISSN-L)
- OpenAlex
- W3021176487
- Cited by
- 2,286
- References
- 0
- Citations by year
- 2026: 27 · 2025: 49 · 2024: 51 · 2023: 68 · 2022: 95 · 2021: 70 · 2020: 89 · 2019: 72 · 2018: 84 · 2017: 98 · 2016: 90 · 2015: 120 · 2014: 116 · 2013: 120 · 2012: 133
- FWCI
- 30.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › School Choice and Performance
- Keywords
- selection bias, program evaluation, matching estimators, difference-in-differences, selection model, social programs, treatment effect estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.