2014 Rossi Award Lecture: Beyond Internal Validity
Abstract
Background: For much of the last 40 years, the evaluation profession has been consumed in a battle over internal validity. Today, that battle has been decided. Random assignment, while still far from universal in practice, is almost universally acknowledged as the preferred method for impact evaluation. It is time for the profession to shift its attention to the remaining major flaws in the “standard model” of evaluation: (i) external validity and (ii) the high cost and low hit rate of experimental evaluations as currently practiced. Recommendations: To raise the profession’s attention to external validity, the author recommends some simple, easy steps to be taken in every evaluation. The author makes two recommendations to increase the number of interventions found to be effective within existing resources: First, a two-stage evaluation strategy in which a cheap, streamlined Stage 1 evaluation is followed by a more intensive Stage 2 evaluation only for those interventions found to be effective in a Stage 1 trial and, second, use of random assignment to guide the myriad program management decisions that must be made in the course of routine program operations. This article is not intended as a solution to these issues: It is intended to stimulate the evaluation community to take these issues more seriously and to develop innovative solutions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The randomized experiment by Weisburd et al. (2011) may have low external validity.
“However, upon closer review of the study methodology, this randomized experiment may have low external validity, which scholars have noted in prior field experiments of policing (Eck, 2006) as well as in public policy research in general (Orr, 2015).”
From The Effect of Various Police Enforcement Actions on Violent Crime: Evidence From a Saturation Foot-Patrol Intervention (2018) · cites this paper as Orr, 2015
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Journal of economic literature
Armstrong, J. S. (1996) · International Journal of Forecasting · doi:10.1016/0169-2070(96)88197-0 · Cited by 9,926 · see citing works
Head Start Impact Study. Final Report.
Puma, M. J., Bell, S., Cook, R., Heid, C. A., Shapiro, G., Broene, P., Jenkins, F. J., Fletcher, P., Quinn, L., Friedman, J., et al. (2010) · Cited by 519 · see citing works
Does Job Corps Work? Impact Findings from the National Job Corps Study
Schochet, P. Z., Burghardt, J., & McConnell, S. (2008) · American Economic Review · doi:10.1257/aer.98.5.1864 · Cited by 447 · see citing works
Lessons from the U.S. Unemployment Insurance Experiments
Meyer, B. (2007) · RePEc: Research Papers in Economics · Cited by 428 · see citing works
Zia, M. I., Siu, L. L., Pond, G. R., & Chen, E. X. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.06.679 · Cited by 157 · see citing works
The digest of social experiments
(2005) · Choice Reviews Online · doi:10.5860/choice.42-3801 · Cited by 136 · see citing works
External Validity in Policy Evaluations That Choose Sites Purposively
Olsen, R. B., Orr, L. L., Bell, S. H., & Stuart, E. A. (2012) · Journal of Policy Analysis and Management · doi:10.1002/pam.21660 · Cited by 132 · see citing works
Self-citationBloom, H. S., & Weiland, C. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2594430 · Cited by 110 · see citing works
Fighting for reliable evidence
(2014) · Choice Reviews Online · doi:10.5860/choice.51-3910 · Cited by 73 · see citing works
An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations
Schochet, P. Z. (2009) · Evaluation Review · doi:10.1177/0193841x09350590 · Cited by 69 · see citing works
Diagnostic tests often fail to lead to changes in patient outcomes
Siontis, K. C., Siontis, G. C. M., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.008 · Cited by 67 · see citing works
The Evaluation of Charter School Impacts
(2010) · PsycEXTRA Dataset · doi:10.1037/e599002011-001 · Cited by 65 · see citing works
Research in Social Problems and Public Policy
(2010) · Transforming Government People Process and Policy · doi:10.1108/tg.2010.32604aaa.003 · Cited by 54 · see citing works
When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model
Epstein, D., & Klerman, J. A. (2012) · Evaluation Review · doi:10.1177/0193841x12474275 · Cited by 50 · see citing works
Miller, C. K., Bos, J. M., Porter, K. E., Tseng, F., & Abe, Y. (2005) · MDRC · Cited by 35 · see citing works
The Quantum Opportunity Program Demonstration: Implementation and Short-Term ImpactsPreprint
Maxfield, M., Schirm, A. L., & Rodríguez‐Planas, N. (2003) · RePEc: Research Papers in Economics · Cited by 29 · see citing works
Fighting for Reliable Evidence
Gueron, J. M., & Rolston, H. (2013) · Cited by 25 · see citing works
The National Evaluation of the Food Stamp Employment and Training Program
Puma, M. J., & Burstein, N. R. (1994) · Journal of Policy Analysis and Management · doi:10.2307/3325016 · Cited by 20 · see citing works
Besharov, D. J. (2009) · Journal of Policy Analysis and Management · doi:10.1002/pam.20423 · Cited by 15 · see citing works
Cody, S., & Asher, A. (2014) · RePEc: Research Papers in Economics · Cited by 11 · see citing works
BOND Implementation and Evaluation: Stage 2 Early Assessment ReportPreprint
Gubits, D., Cook, R., Bell, S. H., Derr, M. K., Berk, J., Person, A. E., Stapleton, D. C., Hoffman, D., & Wittenburg, D. (2013) · RePEc: Research Papers in Economics · Cited by 5 · see citing works
Replication issues in social experiments: lessons from US labor market programs
Barnow, B. S., & Greenberg, D. H. (2013) · Journal for Labour Market Research · doi:10.1007/s12651-013-0133-2 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 24 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Implications of Small Samples for Generalization: Adjustments and Rules of Thumb
Tipton, E., Hallberg, K., Hedges, L. V., & Chan, W. (2016) · Evaluation Review · doi:10.1177/0193841x16655665 · Cited by 149 · see citing works
Stuart, E. A., & Rhodes, A. (2016) · Evaluation Review · doi:10.1177/0193841x16660663 · Cited by 54 · see citing works
Piza, E. L. (2018) · Criminal Justice Policy Review · doi:10.1177/0887403417725370 · Cited by 32 · see citing works
“In an experiment pairing directed police patrol with proactive closed-circuit television monitoring, Piza et al. (2015) found that increased levels of enforcement in response to observed incidents of concern generated significant reductions in violent crime and social disorder.”
BackgroundA Design-Based Approach to Improve External Validity in Welfare Policy Evaluations
Tipton, E., & Peck, L. R. (2016) · Evaluation Review · doi:10.1177/0193841x16655656 · Cited by 24 · see citing works
Westbrook, T. R., Avellar, S., & Seftor, N. S. (2016) · Evaluation Review · doi:10.1177/0193841x16666463 · Cited by 21 · see citing works
Bridging to Action Requires Mixed Methods, Not Only Randomised Control Trials
Olsen, W. (2019) · European Journal of Development Research · doi:10.1057/s41287-019-00201-x · Cited by 18 · see citing works
“…must be both longitudinal (pre- and post-treatment measures, typically) and spread out far and wide to disallow people in the Control arm from discussing the treatment or its effects with those in the Treatment arm (an example in microfinance is McHugh, Biosca and Donaldson, 2017; see Orr, 2015).”
BackgroundThe key role of representativeness in evidence-based education
Joyce, K. E. (2019) · Educational Research and Evaluation · doi:10.1080/13803611.2019.1617989 · Cited by 16 · see citing works
“As Larry Orr (2015) defines it, “an externally valid evaluation provides unbiased estimates of the impact of an intervention on the population of interest for policy” (p. 169).”
BackgroundMethodInfluentialSeftor, N. S. (2016) · Evaluation Review · doi:10.1177/0193841x16665023 · Cited by 9 · see citing works
Vaganay, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160652 · Cited by 5 · see citing works
On the “When” of Social Experiments: The Tension Between Program Refinement and Abandonment
Epstein, D., & Klerman, J. A. (2016) · New Directions for Evaluation · doi:10.1002/ev.20213 · Cited by 3 · see citing works
Mercier, J., & Bouthillier, I. B. (2022) · UNESCO MGIEP eBooks · doi:10.56383/qhen7211 · Cited by 1 · see citing works
Roschelle, J., Tipton, E., Shechtman, N., & Vahey, P. (2018) · doi:10.13140/rg.2.2.31260.13442
Estimating Policy Effects in a Social Network with Independent Set SamplingPreprint
Ang, E. T. Y., Bhattacharya, P., & Lim, A. E. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4490708
Li, T., Chen, M., Lin, Y., & Chen, Z. (2025) · The Journal of Experimental Education · doi:10.1080/00220973.2025.2599809
- Authors
- Orr, L. L.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Orr, Larry L.
- ORCID
- Orr, Larry L.
- Venue
- Evaluation Review
- Publisher
- SAGE Publications
- Published
- 2015-03-24
- Volume
- 39
- Issue
- 2
- Pages
- 167-178
- ISSN
- 0145-4692 (ISSN-L)
- 0193-841X
- 1552-3926
- OpenAlex
- W2025278242
- Cited by
- 14
- References
- 24
- Citations by year
- 2025: 1 · 2023: 1 · 2022: 1 · 2019: 2 · 2018: 2 · 2016: 7
- FWCI
- 3.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Evaluation Review (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Economics, Econometrics and Finance › Finance › Community Development and Social Impact
- Keywords
- impact evaluation, generalizability, internal validity, random assignment, experimental evaluation, evaluation design
- MeSH terms
- Awards and Prizes, Evaluation Studies as Topic, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
