Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme
The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Abstract
This paper considers whether it is possible to devise a nonexperimental procedure for evaluating a prototypical job training programme. Using rich nonexperimental data, we examine the performance of a two-stage evaluation methodology that (a) estimates the probability that a person participates in a programme and (b) uses the estimated probability in extensions of the classical method of matching. We decompose the conventional measure of programme evaluation bias into several components and find that bias due to selection on unobservables, commonly called selection bias in econometrics, is empirically less important than other components, although it is still a sizeable fraction of the estimated programme impact. Matching methods applied to comparison groups located in the same labour markets as participants and administered the same questionnaire eliminate much of the bias as conventionally measured, but the remaining bias is a considerable fraction of experimentally-determined programme impact estimates. We test and reject the identifying assumptions that justify the classical method of matching. We present a nonparametric conditional difference-in-differences extension of the method of matching that is consistent with the classical index-sufficient sample selection model and is not rejected by our tests of identifying assumptions. This estimator is effective in eliminating bias, especially when it is due to temporally-invariant omitted variables.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For the OR approach, the consistency of the ATT estimator (2.2) relies on the estimators of md , t ( · ) , ̂ µ d , t ( · ) , being correctly specified, whereas the IPW estimator (2.4) relies on the propensity score estimator ̂ π ( · ) of p ( · ) being correctly specified.”
From Doubly robust difference-in-differences estimators (2020) · cites this paper as Heckman et al. (1997)
In the citing paper
“These fall into two main categories, which we denote trajectory modeling [28, 29, 86, 88, 95] and propensity score weighting [17, 30].”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [29]
In the citing paper
“Doubly robust (DR) estimators combine trajectory modeling and propensity score estimation and produce consistent estimates if either the trajectory model or the propensity score model is correctly specified: [29]”
From Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods (2025) · cites this paper as [29]
In the citing paper
“The OR, IPW and DR estimands respectively generalize Heckman et al. (1997), Abadie (2005) and Sant'Anna and Zhao (2020) estimands for the two groups, two periods DiD setup to the staggered DiD setup with multiple periods and multiple groups.”
From Difference-in-Differences with multiple time periods (2021) · cites this paper as Heckman et al. (1997)
In the citing paper
“CS also derives estimators that are consistent for AT T (g, τ ) under more general conditions, such as when the parallel-trends assumption holds conditionally on covariates, including an outcome-regression-based estimator”
From How Much Should We Trust Staggered Difference-in-Differences Estimates? (2022) · cites this paper as Heckman et al., 1997
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Density Estimation for Statistics and Data Analysis
Silverman, B. W. (2018) · Routledge eBooks · doi:10.1201/9781315140919 · Cited by 15,796
Rosenbaum, P. R., & Rubin, D. B. (1985) · The American Statistician · doi:10.2307/2683903 · Cited by 4,774
SOME THOUGHTS ON THE DISTRIBUTION OF EARNINGS 1
ROY, A. D. (1951) · Oxford Economic Papers · doi:10.1093/oxfordjournals.oep.a041827 · Cited by 2,958
Bayesian inference for causal effects: The role of randomization
Rubin, D. B. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344064 · Cited by 2,503
Estimating the Effect of Training Programs on Earnings
Ashenfelter, O. (1978) · The Review of Economics and Statistics · doi:10.2307/1924332 · Cited by 1,515
Design-adaptive Nonparametric Regression
Fan, J. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476255 · Cited by 1,268
The Estimation of Choice Probabilities from Choice Based Samples
Manski, C. F., & Lerman, S. R. (1977) · Econometrica · doi:10.2307/1914121 · Cited by 1,208
Cited by
Items 1,001–1,029 of 1,029
Succession and firm growth: results from a non-parametric matching approach
Diwisch, D. S., Voithofer, P., & Weiss, C. R. (2007) · Small Business Economics · doi:10.1007/s11187-007-9072-z · Cited by 59
“the empirical literature are discussed in more detail in Heckman et al. (1997) , Smith and Todd (2005) as well as Becker and Ichino (2002).”
BackgroundBinder, M., & Coad, A. (2013) · Social Science & Medicine · doi:10.1016/j.socscimed.2013.03.025 · Cited by 67
“On the one hand, it has been argued that omitting important variables can seriously increase bias in the resulting estimates (Heckman et al., 1997; Dehejia and Wahba, 1999).”
BackgroundThe Performance of Performance-Based Contracting in Human Services: A Quasi-Experiment
Lu, J. (2015) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muv002 · Cited by 60
“This is particularly a concern when using quasi-experiments in cross-state comparison (Heckman et al. 1997; Michalopoulos, Bloom, and Hill 2004).”
MethodThe “Peter Pan Syndrome” in Emerging Markets: The Productivity-Transparency Trade-off in IT Adoption
Sudhir, K., & Talukdar, D. (2015) · Marketing Science · doi:10.1287/mksc.2015.0921 · Cited by 59
“Specifically, we use kernel matching advocated in Heckman et al. (1997) to construct the weighting function based on the difference in propensity scores between firm i and j .”
MethodBarbosa, C., Cowell, A., Dowd, W., Landwehr, J., Aldridge, A., & Bray, J. (2017) · Addiction · doi:10.1111/add.13658 · Cited by 46
“kernel matching with the Epanechnikov kernel, which estimates the counterfactualmean for the comparisons using a local linear regression estimator [42].”
(2013) · MDRC · Cited by 45
Homeownership and household formation: No homeownership, no marriage?
Hu, M., & Wang, X. (2020) · Journal of Housing and the Built Environment · doi:10.1007/s10901-019-09724-5 · Cited by 37
Pan, X., & Yu, L. (2024) · Energy Economics · doi:10.1016/j.eneco.2024.107334 · Cited by 36
Partnerships to prevent deforestation in the Amazon
Jung, S., & Polasky, S. (2018) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2018.11.001 · Cited by 35
Assessing the Impact of Financial Incentives in Extension Programmes: Evidence From Ireland
Läpple, D., & Hennessy, T. (2015) · Journal of Agricultural Economics · doi:10.1111/1477-9552.12108 · Cited by 33
Policy evaluation during a pandemic
Callaway, B., & Li, T. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2023.03.009 · Cited by 38
Specification tests for the propensity score
Sant’Anna, P. H. C., & Song, X. (2019) · Journal of Econometrics · doi:10.1016/j.jeconom.2019.02.002 · Cited by 31
Almeida, R., Orr, L., & Robalino, D. (2014) · IZA Journal of Labor Policy · doi:10.1186/2193-9004-3-12 · Cited by 29
“The final U.S. wage subsidy, the on-the-job training subsidy under the Job Training Partnership Act (JTPA) has been evaluated by many authors (see Heckman et al. 1997; Bloom et al. 1997; andOrr et al. 1996).”
Comparing Outcomes for Youth Served in Treatment Foster Care and Treatment Group Care
Robst, J., Armstrong, M., & Dollard, N. (2011) · Journal of Child and Family Studies · doi:10.1007/s10826-011-9447-2 · Cited by 28
“Thus, when comparing individuals with similar estimated probabilities of receiving treatment (some receive a specific treatment and some do not), a difference-in-difference in outcomes among matched observations reflects treatment and not preexisting (observed or unobserved) factors ( Heckman et al. 1997 ).”
BackgroundBank failure prediction: corporate governance and financial indicators
Alzayed, N., Eskandari, R., & Yazdifar, H. (2023) · Review of Quantitative Finance and Accounting · doi:10.1007/s11156-023-01158-z · Cited by 23
“(2005) and Kao and Liu (2004), and PLS-DA, used by Serrano-Cinca and Gutiérrez-Nieto (2013).”
MethodThe psychology of job loss: using social media data to characterize and predict unemployment
Proserpio, D., Counts, S., & Jain, A. (2016) · Proceedings of the 8th ACM Conference on Web Science · doi:10.1145/2908131.2913008 · Cited by 23
“The question the ATT answers is the following: conditional on a user being fired (hired), what is the impact of this action on the psychological well-being of the user? While the ATE seems more informative, the ATT is more relevant in our setting given that the treatment is never administered randomly ([21], [22]).”
BackgroundLin, F., Tian, H., Zhao, J., & Chi, M. (2023) · Internet Research · doi:10.1108/intr-05-2021-0292 · Cited by 22
What Drives Risky Prescription Opioid Use? Evidence from Migration
Finkelstein, A., Gentzkow, M., Li, D., & Williams, H. (2022) · doi:10.3386/w30471 · Cited by 20
(2003) · Cited by 20
Coverage of infertility treatment and fertility outcomes
Machado, M. P., & Sanz-de-Galdeano, A. (2015) · SERIEs · doi:10.1007/s13209-015-0135-0 · Cited by 18
“…or attitudes, such as the abortion rate per 1000 women by state of residency; and (6) 28 These concerns have been raised in several studies (e.g., Heckman et al. 1997, 1998;Michalopoulos et al. 2004; Smith and Todd 2005), where it was argued that matching on observables alone would not…”
A matching estimator of household equivalence scales
Szulc, A. (2009) · Economics Letters · doi:10.1016/j.econlet.2009.01.027 · Cited by 18
Difference-in-Differences for Health Policy and Practice: A Review of Modern Methods
Feng, S., Ganguli, I., Lee, Y., Poe, J., Ryan, A., & Bilinski, A. (2025) · Statistics in Medicine · doi:10.1002/sim.70247 · Cited by 16
QCA is of questionable value for policy research
Tanner, S. (2014) · Policy and Society · doi:10.1016/j.polsoc.2014.08.003 · Cited by 16
MEDICAL REGULATION AND HEALTH OUTCOMES: THE EFFECT OF THE PHYSICIAN EXAMINATION REQUIREMENT
Cotet, A. M., & Benjamin, D. K. (2013) · Health Economics · doi:10.1002/hec.2807 · Cited by 13
Galdo, V., & Briceño, B. (2005) · doi:10.18235/0011133 · Cited by 13
Do Internships Matter?: the Impact of Internship Participation on Employability
Choe, C., Kim, Y. B., & Choi, K. (2023) · The Singapore Economic Review · doi:10.1142/s0217590823500133 · Cited by 13
Efficient Difference-in-Differences Estimation with High-Dimensional Common Trend Confounding
Zimmert, M. (2018) · arXiv · doi:10.48550/arxiv.1809.01643 · Cited by 9
Doubly robust estimation of policy-relevant causal effects under interference
Hettinger, G., Roberto, C., Lee, Y., & Mitra, N. (2025) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1093/jrsssc/qlae066 · Cited by 8
Adenuga, A. H., & Jack, C. (2025) · Ecological Indicators · doi:10.1016/j.ecolind.2025.113714 · Cited by 4
Metadata
- Authors
- Heckman, J. J., Ichimura, H., & Todd, P. E.
- Affiliations
- University of Chicago (US)
- University of Pittsburgh (US)
- University of Pennsylvania (US)
- Venue
- The Review of Economic Studies
- Publisher
- Oxford University Press (OUP)
- Published
- 1997-10-01
- Volume
- 64
- Issue
- 4
- Pages
- 605-654
- ISSN
- 0034-6527 (ISSN-L)
- 1467-937X
- OpenAlex
- W2066735350
- Semantic Scholar
- ec958b198d3b6712d1e0f3395a05ff2b0136a1bb
- Cited by
- 5,486
- References
- 32
- Citations by year
- 2026: 216 · 2025: 268 · 2024: 271 · 2023: 292 · 2022: 296 · 2021: 246 · 2020: 254 · 2019: 244 · 2018: 269 · 2017: 236 · 2016: 231 · 2015: 251 · 2014: 266 · 2013: 248 · 2012: 275
- FWCI
- 20.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Review of Economic Studies (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
- Keywords
- selection on unobservables, sample selection model, identifying assumptions, omitted variables
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.