Cited by 2,395Advanced Causal Inference TechniquesEvaluation and Performance AssessmentSchool Choice and Performance

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.

Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs

Dehejia, Rajeev H., Wahba, Sadek·1999·Journal of the American Statistical Association

DOI 10.1080/01621459.1999.10473858

Open this paper in CoChat — free account

Abstract

This article uses propensity score methods to estimate the treatment impact of the National Supported Work (NSW) Demonstration, a labor training program, on postintervention earnings. We use data from Lalonde's evaluation of nonexperimental methods that combine the treated units from a randomized evaluation of the NSW with nonexperimental comparison units drawn from survey datasets. We apply propensity score methods to this composite dataset and demonstrate that, relative to the estimators that Lalonde evaluates, propensity score estimates of the treatment impact are much closer to the experimental benchmark estimate. Propensity score methods assume that the variables associated with assignment to treatment are observed (referred to as ignorable treatment assignment, or selection on observables). Even under this assumption, it is difficult to control for differences between the treatment and comparison groups when they are dissimilar and when there are many preintervention variables. The estimated propensity score (the probability of assignment to treatment, conditional on preintervention variables) summarizes the preintervention variables. This offers a diagnostic on the comparability of the treatment and comparison groups, because one has only to compare the estimated propensity score across the two groups. We discuss several methods (such as stratification and matching) that use the propensity score to estimate the treatment impact. When the range of estimated propensity scores of the treatment and comparison groups overlap, these methods can estimate the treatment impact for the treatment group. A sensitivity analysis shows that our estimates are not sensitive to the specification of the estimated propensity score, but are sensitive to the assumption of selection on observables. We conclude that when the treatment and comparison groups overlap, and when the variables determining assignment to treatment are observed, these methods provide a means to estimate the treatment impact. Even though propensity score methods are not always applicable, they offer a diagnostic on the quality of nonexperimental comparison groups in terms of observable preintervention variables.

Integrity signals

Registry

Crossref · 10.1080/01621459.1999.10473858

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1985) · The American Statistician · doi:10.2307/2683903 · Cited by 6,472
Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997) · The Review of Economic Studies · doi:10.2307/2971733 · Cited by 5,486
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293
Design of Experiments
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Matching As An Econometric Evaluation Estimator
Heckman, J. J., Ichimura, H., & Todd, P. (1998) · Review of Economic Studies · doi:10.1111/1467-937x.00044 · Cited by 3,807
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082

Show all 44 references

Research works printed but not linked: 6

Angrist J., 1990, American Economic Review, 80, 313
Dehejia R. H., 1998, “Matching Methods for Estimating Causal Effects in Non-Experimental Studies,”
10.1080/01621459.1989.10478848 · DOI
Lalonde R., 1986, American Economic Review, 76, 604
1983, Summary and Findings of the National Supported Work Demonstration
Rubin D., 1977, Journal of Educational Statistics, 2, 1

Sorted by citations

Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
SOME PRACTICAL GUIDANCE FOR THE IMPLEMENTATION OF PROPENSITY SCORE MATCHING
Caliendo, M., & Kopeinig, S. (2008) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.2007.00527.x · Cited by 6,238
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
Unobservable Selection and Coefficient Stability: Theory and Evidence
Oster, E. (2019) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2016.1227711 · Cited by 5,367
Recent Developments in the Econometrics of Program Evaluation
Imbens, G. W., & Wooldridge, J. M. (2009) · Journal of Economic Literature · doi:10.1257/jel.47.1.5 · Cited by 4,926
MatchIt : Nonparametric Preprocessing for Parametric Causal Inference
Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2011) · Journal of Statistical Software · doi:10.18637/jss.v042.i08 · Cited by 4,385
Causal Inference without Balance Checking: Coarsened Exact Matching
Iacus, S. M., King, G., & Porro, G. (2012) · Political Analysis · doi:10.1093/pan/mpr013 · Cited by 3,563
Does matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review
Imbens, G. W. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023651 · Cited by 2,994
Social norms and energy conservation
Allcott, H. (2011) · Journal of Public Economics · doi:10.1016/j.jpubeco.2011.03.003 · Cited by 2,802
Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation
Rubin, D. B. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020363010465 · Cited by 2,228
A tutorial on propensity score estimation for multiple treatments using generalized boosted models
McCaffrey, D. F., Griffin, B. A., Almirall, D., Slaughter, M. E., Ramchand, R., & Burgette, L. F. (2013) · Statistics in Medicine · doi:10.1002/sim.5753 · Cited by 1,450
Bias-Corrected Matching Estimators for Average Treatment Effects
Abadie, A., & Imbens, G. W. (2011) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2009.07333 · Cited by 1,416
Propensity score estimation with boosted regression for evaluating causal effects in observational studies
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.403 · Cited by 1,317
Environmental Policy and Directed Technological Change: Evidence from the European Carbon Market
Calel, R., & Dechezleprêtre, A. (2016) · Review of Economics and Statistics · doi:10.1162/rest_a_00470 · Cited by 1,147
Identification and Inference in Nonlinear Difference-in-Differences Models
Athey, S., & Imbens, G. W. (2006) · Econometrica · doi:10.1111/j.1468-0262.2006.00668.x · Cited by 1,084
Doubly robust difference-in-differences estimators
Sant’Anna, P. H. C., & Zhao, J. (2020) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.06.003 · Cited by 1,047
Inverse probability weighted estimation for general missing data problems
Wooldridge, J. M. (2007) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.02.002 · Cited by 1,029
Dealing with limited overlap in estimation of average treatment effects
Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009) · Biometrika · doi:10.1093/biomet/asn055 · Cited by 1,007
Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research.
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010) · Psychological Methods · doi:10.1037/a0019623 · Cited by 802
Using matching, instrumental variables and control functions to estimate economic choice models
Heckman, J., & Navarro-Lozano, S. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023660 · Cited by 768
Causal Effects in Clinical and Epidemiological Studies Via Potential Outcomes: Concepts and Analytical Approaches
Little, R. J., & Rubin, D. B. (2000) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.21.1.121 · Cited by 721
Optimal Full Matching and Related Designs via Network Flows
Hansen, B. B., & Klopfer, S. O. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x137047 · Cited by 703
Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604
Diagnosing and responding to violations in the positivity assumption
Petersen, M. L., Porter, K. E., Gruber, S., Wang, Y., & van der Laan, M. J. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280210386207 · Cited by 548
Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations
Slavin, R. E. (2008) · Educational Researcher · doi:10.3102/0013189x08314117 · Cited by 560
Neonatal Outcomes After Prenatal Exposure to Selective Serotonin Reuptake Inhibitor Antidepressants and Maternal Depression Using Population-Based Linked Health Data
Oberlander, T. F., Warburton, W., Misri, S., Aghajanian, J., & Hertzman, C. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.8.898 · Cited by 551
The Labor Market Consequences of Incarceration
Western, B., Kling, J. R., & Weiman, D. F. (2001) · Crime & Delinquency · doi:10.1177/0011128701047003007 · Cited by 499
Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479
The importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490
Site Selection Bias in Program Evaluation *
Allcott, H. (2015) · The Quarterly Journal of Economics · doi:10.1093/qje/qjv015 · Cited by 426
New evidence on the finite sample properties of propensity score reweighting and matching estimators
Busso, M., DiNardo, J., & McCrary, J. (2014) · Review of Economics and Statistics · doi:10.1162/rest_a_00431 · Cited by 428
The joint impact of the European Union emissions trading system on carbon emissions and economic performance
Dechezleprêtre, A., Nachtigall, D., & Venmans, F. (2023) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2022.102758 · Cited by 416
A Survey of Learning Causality with Data: Problems and Methods
Guo, R., Cheng, L., Li, J., Hahn, P. R., & Liu, H. (2021) · ACM Computing Surveys · doi:10.1145/3397269 · Cited by 362
Nonexperimental Versus Experimental Estimates of Earnings Impacts
Glazerman, S., Levy, D. M., & Myers, D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254879 · Cited by 352
Drawing conclusions about causes from systematic reviews of risk factors: The Cambridge Quality Checklists
Murray, J., Farrington, D. P., & Eisner, M. P. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-008-9066-0 · Cited by 280
The impact of entrepreneurship education in high school on long-term entrepreneurial performance
Elert, N., Andersson, F. W., & Wennberg, K. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2014.12.020 · Cited by 290
An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator: Evidence from Mexico’s PROGRESA Program
Diaz, J. J., & Handa, S. (2006) · Journal of Human Resources · doi:10.3368/jhr.xli.2.319 · Cited by 196
Growing Existing Customers’ Revenue Streams through Customer Referral Programs
Garnefeld, I., Eggert, A., Helm, S. V., & Tax, S. S. (2013) · Journal of Marketing · doi:10.1509/jm.11.0423 · Cited by 171
Robust post-matching inference
Abadie, A., & Spiess, J. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1840383 · Cited by 169
Transitions in romantic relationships and development of self-esteem.
Luciano, E. C., & Orth, U. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000109 · Cited by 149
Does the EU Emissions Trading System induce investment leakage? Evidence from German multinational firms
Koch, N., & Basse Mama, H. (2019) · Energy Economics · doi:10.1016/j.eneco.2019.04.018 · Cited by 148
Frailty phenotypes and mortality after lung transplantation: a prospective cohort study
Singer, J. P., Diamond, J. M., Anderson, M. R., Katz, P. P., Covinsky, K., Oyster, M., Blue, T., Soong, A., Kalman, L., Shrestha, P., et al. (2018) · American Journal of Transplantation · doi:10.1111/ajt.14873 · Cited by 133
The Online STEM Classroom—Who Succeeds? An Exploration of the Impact of Ethnicity, Gender, and Non-traditional Student Characteristics in the Community College Context
Wladis, C., Conway, K. M., & Hachey, A. C. (2015) · Community College Review · doi:10.1177/0091552115571729 · Cited by 127
Interventions to improve the labour market outcomes of youth: A systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions
Kluve, J., Puerto, S., Robalino, D., Romero, J. M., Rother, F., Stöterau, J., Weidenkaff, F., & Witte, M. (2017) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2017.12 · Cited by 106
Quantile treatment effects in difference in differences models with panel data
Callaway, B., & Li, T. (2019) · Quantitative Economics · doi:10.3982/qe935 · Cited by 110
Positive Mental Health and Academic Achievement in Elementary School: New Evidence From a Matching Analysis
O’Connor, M., Cloney, D., Kvalsvig, A., & Goldfeld, S. (2019) · Educational Researcher · doi:10.3102/0013189x19848724 · Cited by 93
The negative impacts of starting middle school in sixth grade
Cook, P. J., MacCoun, R., Muschkin, C., & Vigdor, J. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20309 · Cited by 86
Impact of m‐health application used by community health volunteers on improving utilisation of maternal, new‐born and child health care services in a rural area of Uttar Pradesh, India
Prinja, S., Nimesh, R., Gupta, A., Bahuguna, P., Gupta, M., & Thakur, J. S. (2017) · Tropical Medicine & International Health · doi:10.1111/tmi.12895 · Cited by 81
What’s in a relationship? An examination of social capital, race and class in mentoring relationships
Gaddis, S. M. (2012) · Social Forces · doi:10.1093/sf/sos003 · Cited by 84
Adolescent Cannabis Problems and Young Adult Depression: Male-Female Stratified Propensity Score Analyses
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn184 · Cited by 76
Financial inclusion and welfare in South Africa: Is there a gender gap?
Nanziri, E. L. (2016) · Journal of African Development · doi:10.5325/jafrideve.18.2.0109 · Cited by 65
Causal inference on human behaviour
Bailey, D. H., Jung, A. J., Beltz, A. M., Eronen, M. I., Gische, C., Hamaker, E. L., Kording, K. P., Lebel, C., Lindquist, M. A., Moeller, J., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01939-z · Cited by 56
Maternal breastfeeding and children’s cognitive development
Koh, K. (2017) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.06.012 · Cited by 52
Causal models for longitudinal and panel data: a survey
Arkhangelsky, D., & Imbens, G. (2024) · The Econometrics Journal · doi:10.1093/ectj/utae014 · Cited by 50
A decision support framework to implement optimal personalized marketing interventions
Guelman, L., Guillén, M., & Pérez-Marín, A. M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.01.010 · Cited by 48
The cost‐effectiveness of brief intervention versus brief treatment of Screening, Brief Intervention and Referral to Treatment (SBIRT) in the United States
Barbosa, C., Cowell, A., Dowd, W., Landwehr, J., Aldridge, A., & Bray, J. (2017) · Addiction · doi:10.1111/add.13658 · Cited by 46
Posttraumatic stress disorder and HIV risk among poor, inner-city women receiving care in an emergency department
El-Bassel, N., Gilbert, L., Vinocur, D., Chang, M., & Wu, E. (2011) · American Journal of Public Health · doi:10.2105/ajph.2009.181842 · Cited by 30
The relative impact of brief treatment versus brief intervention in primary health‐care screening programs for substance use disorders
Aldridge, A., Dowd, W., & Bray, J. (2017) · Addiction · doi:10.1111/add.13653 · Cited by 25
Do Internships Matter?: the Impact of Internship Participation on Employability
Choe, C., Kim, Y. B., & Choi, K. (2023) · The Singapore Economic Review · doi:10.1142/s0217590823500133 · Cited by 13
Expense preference behavior and management “outsourcing”: a comparison of adopters and non-adopters of contract management in U.S. hospitals.
Carey, K., & Dor, A. (2008) · Journal of Productivity Analysis · doi:10.1007/s11123-007-0065-3 · Cited by 9

Authors: Dehejia, R. H., & Wahba, S.
Affiliations: Columbia University (US)
Morgan Stanley (United States) (US)
ORCID: Dehejia, Rajeev H.
Venue: Journal of the American Statistical Association
Publisher: Informa UK Limited
Published: 1999-12
Volume: 94
Issue: 448
Pages: 1053-1062
ISSN: 0162-1459 (ISSN-L)
1537-274X
DOI: 10.1080/01621459.1999.10473858
OpenAlex: W3122948292
Cited by: 2,395
References: 44
Citations by year: 2026: 43 · 2025: 56 · 2024: 66 · 2023: 80 · 2022: 97 · 2021: 110 · 2020: 103 · 2019: 98 · 2018: 126 · 2017: 119 · 2016: 122 · 2015: 130 · 2014: 141 · 2013: 116 · 2012: 120
FWCI: 43.40
Type: journal article
Language: en
Open access: closed
Locations: Journal of the American Statistical Association (publishedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Social Sciences › Public Administration › Evaluation and Performance Assessment
Social Sciences › Education › School Choice and Performance
Keywords: propensity score methods, program evaluation, selection on observables, stratification, matching, sensitivity analysis, treatment effect estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks