The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note
Abstract
Despite the growing popularity of online opt-in samples in criminology, recent work shows that resultant findings often do not generalize. Not all opt-in samples are alike, however, and matching may improve data quality. Replicating and extending prior work, we compare the generalizability of relational inferences from unmatched and matched opt-in samples. Estimating identical models for four criminal justice outcomes, we compare multivariate regression results from national matched (YouGov) and unmatched (MTurk) opt-in samples to those from the General Social Survey (GSS). YouGov coefficients are almost always in the same direction as GSS coefficients, especially when statistically significant, and are mostly of a similar magnitude; less than 10% of the YouGov and GSS coefficients differ significantly. By contrast, MTurk coefficients are more likely to be in the wrong direction, more likely to be much larger or smaller, and are about three times as likely to differ significantly from GSS coefficients. Matched opt-in samples provide a relatively inexpensive data source for criminal justice researchers, compared to probability samples, and also appear to carry a smaller generalizability penalty than unmatched samples. Our study suggests relational inferences from matched opt-in samples are more likely to generalize than those from unmatched samples.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“YouGov uses a model-based sampling approach to produce matched, weighted samples that generalize extremely well to the US adult population”
From Rape or Homicide: Which Is Worse? (2024) · cites this paper as Graham et al., 2021
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
USING THE CORRECT STATISTICAL TEST FOR THE EQUALITY OF REGRESSION COEFFICIENTS
PATERNOSTER, R., BRAME, R., MAZEROLLE, P., & PIQUERO, A. (1998) · Criminology · doi:10.1111/j.1745-9125.1998.tb01268.x · Cited by 2,742
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 1×
“Respondents were limited to those who had completed over 500 HITs (human intelligence tasks), had a 95% or higher approval rating, and were aged 18 or older in order to capture high-quality respondents (Peer et al., 2014).”
BackgroundThe Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228 · cited 5×
“Recent research suggests that observational findings from such samples often fail to generalize (Thompson & Pickett, 2019), even though experimental results tend to be externally valid (Coppock, 2019; Mullinix et al., 2015).”
BackgroundMethodInfluentialLanders, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714 · cited 1×
“These are convenience samples (Landers & Behrend, 2015), but they provide a relatively inexpensive method of exploring many theories and ideas and have the potential to yield a diverse, nationwide group of respondents (Weinberg et al., 2014).”
MethodSummary Report of the AAPOR Task Force on Non-probability Sampling
Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smt008 · Cited by 652 · cited 3×
“, 2018), and never has complete or even near complete coverage—it only includes Internet users who are signed up on a specific survey platform (Baker et al., 2013; Callegaro et al., 2015).”
BackgroundWeinberg, J., Freese, J., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582 · cited 1×
“These are convenience samples (Landers & Behrend, 2015), but they provide a relatively inexpensive method of exploring many theories and ideas and have the potential to yield a diverse, nationwide group of respondents (Weinberg et al., 2014).”
MethodDoes Survey Mode Still Matter? Findings from a 2010 Multi-Mode Comparison
Ansolabehere, S., & Schaffner, B. F. (2014) · Political Analysis · doi:10.1093/pan/mpt025 · Cited by 573 · cited 1×
“…Current Population Survey (Simmons & Bobo, 2015).5 Other studies find that when compared against each other, there often are as many inconsistencies in results between different probability samples as between opt-in samples and probability samples (Ansolabehere & Schaffner, 2014; Bhutta, 2012).”
ResultGeneralizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · cited 2×
“Finally, we focused on observational relational inferences, but research shows that opt-in samples perform better in experiments (Coppock, 2019; Mullinix et al., 2015), because of low levels of effect heterogeneity (Coppock et al.”
BackgroundGeneralizability of heterogeneous treatment effect estimates across samples
Coppock, A., Leeper, T. J., & Mullinix, K. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1808083115 · Cited by 445 · cited 2×
“Finally, we focused on observational relational inferences, but research shows that opt-in samples perform better in experiments (Coppock, 2019; Mullinix et al., 2015), because of low levels of effect heterogeneity (Coppock et al., 2018).”
Background
Research works printed but not linked: 2
- Groves R. M., 2009, Survey methodology
- Mercer A., 2018, For weighting online opt-in samples, what matters most
Cited by
The American racial divide in fear of the police
Pickett, J. T., Graham, A., & Cullen, F. T. (2022) · Criminology · doi:10.1111/1745-9125.12298 · Cited by 174
“Studies have shown that YouGov’s sample-matching methodology produces high-quality data (Ansolabehere & Schaffner, 2014; Graham et al., 2021; Simmons & Bobo, 2015).”
Self-citationMetcalfe, C. F., & Pickett, J. T. (2021) · Criminology · doi:10.1111/1745-9125.12291 · Cited by 91
Self-citationMass support for proposals to reshape policing depends on the implications for crime and safety
Vaughn, P. E., Peyton, K., & Huber, G. A. (2022) · Criminology & Public Policy · doi:10.1111/1745-9133.12572 · Cited by 75
Are guns the new dog whistle? Gun control, racial resentment, and vote choice*
Schutten, N. M., Pickett, J. T., Burton, A. L., Jonson, C. L., Cullen, F. T., & Burton, V. S. (2021) · Criminology · doi:10.1111/1745-9125.12292 · Cited by 41
“…is evidence that YouGov’s sampling method works well for analyzing voting, evaluations of politicians, racial resentment, and criminal justice opinions— the topics of our study (Ansolabehere & Schaffner, 2014; Graham et al., 2020; Sanders et al., 2007; Simmons & Bobo, 2015; Vavreck & Rivers, 2008).”
Self-citationAn apple in one hand, a gun in the other: Public support for arming our nation's schools
Jonson, C. L., Burton, A. L., Cullen, F. T., Pickett, J. T., & Burton, V. S. (2021) · Criminology & Public Policy · doi:10.1111/1745-9133.12538 · Cited by 34
Self-citation
Metadata
- Authors
- Graham, A., Pickett, J. T., & Cullen, F. T.
- Affiliations
- Georgia Southern University (US)
- University at Albany, State University of New York (US)
- University of Cincinnati (US)
- Corresponding author
- Graham, Amanda
- Venue
- Crime & Delinquency
- Publisher
- SAGE Publications
- Published
- 2020-12-07
- Volume
- 67
- Issue
- 12
- Pages
- 1962-1981
- ISSN
- 0011-1287 (ISSN-L)
- 1552-387X
- OpenAlex
- W3111058860
- Semantic Scholar
- 7f893eb4a7b30c8a9740fc1c08f64d67219230f0
- Cited by
- 114
- References
- 46
- Citations by year
- 2026: 22 · 2025: 27 · 2024: 22 · 2023: 16 · 2022: 13 · 2021: 14
- FWCI
- 14.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Crime & Delinquency (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Keywords
- matched samples, generalizability, criminal justice attitudes, Amazon Mechanical Turk, General Social Survey, data quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note. https://app.cochat.ai/papers/10.1177/0011128720977439
- The paper
- Graham, A., Pickett, J. T., & Cullen, F. T. (2021). Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note. Crime & Delinquency. https://doi.org/10.1177/0011128720977439
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.