The Real Gold Standard: Measuring Counterfactual Worlds That Matter Most to Social Science and Policy
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
The randomized experiment has achieved the status of the gold standard for estimating causal effects in criminology and the other social sciences. Although causal identification is indeed important and observational data present numerous challenges to causal inference, we argue that conflating causality with the method used to identify it leads to a cognitive narrowing that diverts attention from what ultimately matters most—the difference between counterfactual worlds that emerge as a consequence of their being subjected to different treatment regimes applied to all eligible population members over a sustained period of time. To address this system-level and long-term challenge, we develop an analytic framework for integrating causality and policy inference that accepts the mandate of causal rigor but is conceptually rather than methodologically driven. We then apply our framework to two substantive areas that have generated high-visibility experimental research and that have considerable policy influence: ( a) hot-spots policing and ( b) the use of housing vouchers to reduce concentrated disadvantage and thereby crime. After reviewing the research in these two areas in light of our framework, we propose a research path forward and conclude with implications for the interplay of theory, data, and causal understanding in criminology and other social sciences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293
Chetty, R., Hendren, N., & Katz, L. F. (2016) · American Economic Review · doi:10.1257/aer.20150572 · Cited by 2,315
HOT SPOTS OF PREDATORY CRIME: ROUTINE ACTIVITIES AND THE CRIMINOLOGY OF PLACE*
SHERMAN, L. W., GARTIN, P. R., & BUERGER, M. E. (1989) · Criminology · doi:10.1111/j.1745-9125.1989.tb00862.x · Cited by 2,104
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841
Angrist, J. D., & Pischke, J.-S. (2010) · Journal of Economic Perspectives · doi:10.1257/jep.24.2.3 · Cited by 1,566
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2014) · Cambridge University Press · doi:10.1017/cbo9781107587991 · Cited by 1,037
General deterrent effects of police patrol in crime “hot spots”: A randomized, controlled trial
Sherman, L. W., & Weisburd, D. (1995) · Justice Quarterly · doi:10.1080/07418829500096221 · Cited by 1,029
Neighborhood Effects on the Long-Term Well-Being of Low-Income Adults
Ludwig, J., Duncan, G. J., Gennetian, L. A., Katz, L. F., Kessler, R. C., Kling, J. R., & Sanbonmatsu, L. (2012) · Science · doi:10.1126/science.1224648 · Cited by 747
Assessing the Gold Standard — Lessons from the History of RCTs
Bothwell, L. E., Greene, J. A., Podolsky, S. H., & Jones, D. S. (2016) · New England Journal of Medicine · doi:10.1056/nejmms1604593 · Cited by 664
Cited by
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221
“While crowd worker studies and randomized field experiments allow for controlled variation of different factors, even the best conceived randomized trials can fail to have field validity in social policy settings [36].”
BackgroundClustering of health, crime and social-welfare inequality in 4 million citizens from two nations
Richmond-Rakerd, L. S., D’Souza, S., Andersen, S. H., Hogan, S., Houts, R. M., Poulton, R., Ramrakha, S., Caspi, A., Milne, B. J., & Moffitt, T. E. (2020) · Nature Human Behaviour · doi:10.1038/s41562-019-0810-4 · Cited by 93
Ariel, B., Sherman, L. W., & Newton, M. (2020) · Criminology · doi:10.1111/1745-9125.12231 · Cited by 51
Building collective action at crime hot spots: Findings from a randomized field experiment
Weisburd, D., Gill, C., Wooditch, A., Barritt, W., & Murphy, J. (2021) · Journal of Experimental Criminology · doi:10.1007/s11292-019-09401-1 · Cited by 46
Ariel, B., Sutherland, A., & Sherman, L. W. (2019) · Journal of Experimental Criminology · doi:10.1007/s11292-018-9344-4 · Cited by 38
“The logical meaning of “what would have happened otherwise” collapses if the control group receives the treatment (Nagin and Sampson 2018).”
Background
Metadata
- Authors
- Nagin, D. S., & Sampson, R. J.
- Venue
- Annual Review of Criminology
- Publisher
- Annual Reviews
- Published
- 2019-01-13
- Volume
- 2
- Issue
- 1
- Pages
- 123-145
- ISSN
- 2572-4568 (ISSN-L)
- OpenAlex
- W2893064676
- Cited by
- 107
- References
- 16
- FWCI
- 2.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Real Gold Standard: Measuring Counterfactual Worlds That Matter Most to Social Science and Policy. https://app.cochat.ai/papers/10.1146/annurev-criminol-011518-024838
- The paper
- Nagin, D. S., & Sampson, R. J. (2019). The Real Gold Standard: Measuring Counterfactual Worlds That Matter Most to Social Science and Policy. Annual Review of Criminology. https://doi.org/10.1146/annurev-criminol-011518-024838
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.