The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
Nonexperimental Versus Experimental Estimates of Earnings Impacts
Abstract
To assess nonexperimental (NX) evaluation methods in the context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Direct comparisons of ordinary least squares and propensity score matching methods have not shown much of a difference to date (Glazerman et al. 2003; Bloom et al. 2005; Shadish et al. 2008; Cook et al. 2009)”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Glazerman et al. 2003
In the citing paper
“Early reviews of within-study comparisons (Glazerman et al. 2003; Bloom et al. 2005) concluded that estimates of effects from studies using common strategies for equating groups (e.g., matching, analysis of covariance, propensity scoring, selection modeling) are often wrong.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Glazerman et al. 2003
In the citing paper
“In nonexperimental studies, the choice of data analysis technique is not very important for reducing selection bias.”
From Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation (2015) · cites this paper as Glazerman et al. 2003
Citations
Cites
(1963) · Medical Entomology and Zoology · Cited by 15,085
(1979) · Medical Entomology and Zoology · Cited by 11,252
Langenberg, D. N. (2000) · Cited by 4,305
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395
Characterizing Selection Bias Using Experimental Data
Heckman, J., Ichimura, H., Smith, J., & Todd, P. (1998) · Econometrica · doi:10.2307/2999630 · Cited by 2,286
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Evaluating the Econometric Evaluations of Training Programs with Experimental DataPreprint
Lalonde, R. (1984) · RePEc: Research Papers in Economics · Cited by 839
Heckman, J. J., & Hotz, V. J. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478848 · Cited by 515
The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs
Fraker, T., & Maynard, R. (1987) · The Journal of Human Resources · doi:10.2307/145902 · Cited by 323
Making the most of summer school: a meta-analytic and narrative review.
Cooper, H., Charlton, K., Valentine, J. C., & Muhlenbruck, L. (2000) · PubMed · Cited by 315
Research works printed but not linked: 1
- Smith, Jeffrey C., and Petra Todd. Forthcoming. Does matching overcome Lalonde's critique of NX estimators? Journal of Econometrics.
Cited by
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference
Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007) · Political Analysis · doi:10.1093/pan/mpl013 · Cited by 4,205
“…and Rubin (1984), Rubin and Thomas (2000), and Imai and van Dyk (2004) who each recommend matching followed by a different form of parametric adjustment, as well as strategies used in applied research problems by Rosenbaum (1986) and others, as discussed in Glazerman, Levy, and Myers (2003).”
MethodWhy Propensity Scores Should Not Be Used for Matching
King, G., & Nielsen, R. (2019) · Political Analysis · doi:10.1017/pan.2019.11 · Cited by 1,706
Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination
Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Mościcki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005) · Prevention Science · doi:10.1007/s11121-005-5553-y · Cited by 1,338
“In other words, a few randomized trials will probably provide a more precise answer about whether or not an intervention works than many nonrandomized experiments using equating (e.g., Glazerman et al., 2003).”
Method
Metadata
- Authors
- Glazerman, S., Levy, D. M., & Myers, D.
- Affiliations
- Mathematica Policy Research (US)
- Centers for Disease Control and Prevention (US)
- ORCID
- Glazerman, Steven
- Venue
- The ANNALS of the American Academy of Political and Social Science
- Publisher
- SAGE Publications
- Published
- 2003-09
- Volume
- 589
- Issue
- 1
- Pages
- 63-93
- ISSN
- 0002-7162 (ISSN-L)
- 1552-3349
- OpenAlex
- W1967271316
- Cited by
- 352
- References
- 26
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 3 · 2023: 10 · 2022: 19 · 2021: 9 · 2020: 17 · 2019: 30 · 2018: 17 · 2017: 13 · 2016: 22 · 2015: 22 · 2014: 18 · 2013: 15 · 2012: 23
- FWCI
- 9.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- The Annals of the American Academy of Political and Social Science (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Labor market dynamics and wage inequality
- Keywords
- experimental evaluation, job training programs, social welfare programs, employment services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Nonexperimental Versus Experimental Estimates of Earnings Impacts. https://app.cochat.ai/papers/10.1177/0002716203254879
- The paper
- Glazerman, S., Levy, D. M., & Myers, D. (2003). Nonexperimental Versus Experimental Estimates of Earnings Impacts. The ANNALS of the American Academy of Political and Social Science. https://doi.org/10.1177/0002716203254879
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.