The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
The Generalizability of Survey Experiments
Open accessAbstract
Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Framing: toward clarification of a fractured paradigm
Entman, R. M. (1993) · Journal of Communication · doi:10.1111/j.1460-2466.1993.tb01304.x · Cited by 16,203
(2002) · Cited by 13,211
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach
Gamson, W. A., & Modigliani, A. (1989) · American Journal of Sociology · doi:10.1086/229213 · Cited by 4,971
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Is Anyone Responsible?: How Television Frames Political Issues
Iyengar, S. (1991) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226388533.001.0001 · Cited by 3,722
Chong, D., & Druckman, J. N. (2007) · Annual Review of Political Science · doi:10.1146/annurev.polisci.10.072805.103054 · Cited by 3,071
Self-citationData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 2×
“Recent work has sought to compare samples (e.g. Berinsky et al. 2012; Goodman et al. 2012; Horton et al. 2011; Krupnikov and Levine 2014; Paolacci et al. 2010; Weinberg et al. 2014).”
BackgroundSears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119
How Elite Partisan Polarization Affects Public Opinion Formation
DRUCKMAN, J. N., PETERSON, E., & SLOTHUUS, R. (2013) · American Political Science Review · doi:10.1017/s0003055412000500 · Cited by 1,569
Self-citationPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · cited 1×
“STUDY 1 For both studies, the source of our population-based sample is the National Science Foundation funded Time-sharing Experiments for the Social Sciences (TESS) program (http://tessexperiments.org/; also see Franco et al. 2014).”
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,529
Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance
Nelson, T. E., Clawson, R. A., & Oxley, Z. M. (1997) · American Political Science Review · doi:10.2307/2952075 · Cited by 1,422
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
The Oxford Handbook of Political Methodology
(2008) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199286546.001.0001 · Cited by 1,362
Field Experiments: Design, Analysis, and Interpretation
Green, D. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,320
The Implications of Framing Effects for Citizen Competence
Druckman, J. N. (2001) · Political Behavior · doi:10.1023/a:1015006907312 · Cited by 1,265
Self-citationInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Artifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
Framing Public Opinion in Competitive Democracies
CHONG, D., & DRUCKMAN, J. N. (2007) · American Political Science Review · doi:10.1017/s0003055407070554 · Cited by 1,120
Self-citationWho are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“Although not employed here, Berinsky et al. (2014) have suggested that screener questions can be used to address concerns about attention levels in Mturk.”
BackgroundChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903
Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey
Keeter, S., Kennedy, C., Dimock, M., Best, J., & Craighill, P. (2006) · Public Opinion Quarterly · doi:10.1093/poq/nfl035 · Cited by 902
Social heuristics shape intuitive cooperation
Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 720
Wilcox, J. B., Rossi, P. H., Wright, J. D., & Anderson, A. B. (1985) · Journal of Marketing Research · doi:10.2307/3151558 · Cited by 707
Population-Based Survey Experiments
Mutz, D. C. (2011) · doi:10.1515/9781400840489 · Cited by 674
Cambridge Handbook of Experimental Political Science
(2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921452 · Cited by 611
Research synthesis: AAPOR report on online panels
Prepared for the AAPOR Executive Council by a Task Force operating under the auspices of the AAPOR Standards Committee, W. M. I., Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., et al. (2010) · Public Opinion Quarterly · doi:10.1093/poq/nfq048 · Cited by 584
Weinberg, J., Freese, J., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582
Self-citationThe Affective Tipping Point: Do Motivated Reasoners Ever “Get It”?
Redlawsk, D. P., Civettini, A. J. W., & Emmerson, K. M. (2010) · Political Psychology · doi:10.1111/j.1467-9221.2010.00772.x · Cited by 571
Are Survey Experiments Externally Valid?
BARABAS, J., & JERIT, J. (2010) · American Political Science Review · doi:10.1017/s0003055410000092 · Cited by 481
Reconciling conflicting results derived from experimental and survey studies of attitude change.
Hovland, C. I. (1959) · American Psychologist · doi:10.1037/h0042210 · Cited by 457
Students as Experimental Participants: A Defense of the “Narrow Data Base”
Druckman, J. N., & Kam, C. D. (2011) · Cambridge Handbook of Experimental Political Science · doi:10.1017/cbo9780511921452.004 · Cited by 454
Self-citationAttributing Blame: The Public's Response to Hurricane Katrina
Malhotra, N., & Kuo, A. G. (2008) · The Journal of Politics · doi:10.1017/s0022381607080097 · Cited by 440
Humans display a ‘cooperative phenotype’ that is domain general and temporally stable
Peysakhovich, A., Nowak, M. A., & Rand, D. G. (2014) · Nature Communications · doi:10.1038/ncomms5939 · Cited by 426
Social Science Methodology: A Unified Framework
Gerring, J. (2012) · Cited by 404
The Growth and Development of Experimental Research in Political Science
DRUCKMAN, J. N., GREEN, D. P., KUKLINSKI, J. H., & LUPIA, A. (2006) · American Political Science Review · doi:10.1017/s0003055406062514 · Cited by 374
Self-citationForecasting elections with non-representative polls
Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.06.001 · Cited by 371
Students as Experimental Participants: A Defense of the 'Narrow Data Base'
Druckman, J. N., & Kam, C. D. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1498843 · Cited by 350
Self-citationThe Strategy of Rhetoric: Campaigning for the American Constitution
Larkin, E., Riker, W., Calvert, R. L., Mueller, J., & Wilson, R. K. (1997) · Journal of the Early Republic · doi:10.2307/3124651 · Cited by 298
Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data
BENOIT, K., CONWAY, D., LAUDERDALE, B. E., LAVER, M., & MIKHAYLOV, S. (2016) · American Political Science Review · doi:10.1017/s0003055416000058 · Cited by 295
Self-Fulfilling Misperceptions of Public Polarization
Ahler, D. J. (2014) · The Journal of Politics · doi:10.1017/s0022381614000085 · Cited by 276
HUBER, G. A., HILL, S. J., & LENZ, G. S. (2012) · American Political Science Review · doi:10.1017/s0003055412000391 · Cited by 252
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251
Green, D. P., & Kern, H. L. (2012) · Public Opinion Quarterly · doi:10.1093/poq/nfs036 · Cited by 250
Priming the Vote: Campaign Effects in a U.S. Senate Election
Druckman, J. N. (2004) · Political Psychology · doi:10.1111/j.1467-9221.2004.00388.x · Cited by 238
Self-citationThe Influence of Competing Identity Primes on Political Preferences
Klar, S. (2013) · The Journal of Politics · doi:10.1017/s0022381613000698 · Cited by 229
Curses—Winner's and Otherwise—in Genetic Epidemiology
Kraft, P. (2008) · Epidemiology · doi:10.1097/ede.0b013e318181b865 · Cited by 225
Madansky, A., Rossi, P. H., Wright, J. D., & Anderson, A. B. (1985) · Journal of the American Statistical Association · doi:10.2307/2288504 · Cited by 215
Clifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 206
Experimental Methodology in Political Science
McDermott, R. (2002) · Political Analysis · doi:10.1093/pan/10.4.325 · Cited by 188
Online Panel Research: Data Quality Perspective, A
(2014) · doi:10.1002/9781118763520 · Cited by 184
Social science methodology: A unified framework
Dassen, A., & Gemenis, K. (2012) · Acta Politica · doi:10.1057/ap.2012.17 · Cited by 184
Broockman, D., & GREEN, D. P. (2013) · Political Behavior · doi:10.1007/s11109-013-9239-z · Cited by 159
Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental Research
Kam, C. D., Wilking, J. R., & Zechmeister, E. J. (2007) · Political Behavior · doi:10.1007/s11109-007-9037-6 · Cited by 158
Bohannon, J. (2011) · Science · doi:10.1126/science.334.6054.307 · Cited by 140
Professional respondents in nonprobability online panels
Hillygus, D. S., Jackson, N., & Young, M. (2014) · Wiley eBooks · doi:10.1002/9781118763520.ch10 · Cited by 139
The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation
Sniderman, P. M. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921452.008 · Cited by 113
The Political Consequences of Uninformed Voters
Fowler, A., & Margolis, M. F. (2011) · Electoral Studies · doi:10.1016/j.electstud.2013.09.009 · Cited by 113
Field Experiments and Natural Experiments
Gerber, A. S., & GREEN, D. P. (2009) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199286546.003.0015 · Cited by 109
Comparing Contemporaneous Laboratory and Field Experiments on Media Effects
Jerit, J., Barabas, J., & Clifford, S. (2013) · Public Opinion Quarterly · doi:10.1093/poq/nft005 · Cited by 89
Contributions of Survey Research to Political Science
Brady, H. E. (2000) · PS Political Science & Politics · doi:10.2307/420775 · Cited by 88
Sorting through the garbage can: under what conditions do governments adopt policy programs?
Sager, F., & Rielle, Y. (2012) · Policy Sciences · doi:10.1007/s11077-012-9165-7 · Cited by 75
Valentino, N. A., Traugott, M. W., & Hutchings, V. L. (2002) · Political Communication · doi:10.1080/105846002317246470 · Cited by 54
Coppock, A., & GREEN, D. P. (2014) · Political Science Research and Methods · doi:10.1017/psrm.2014.10 · Cited by 54
Combining forecasts for elections: Accurate, relevant, and timely
Rothschild, D. M. (2014) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.08.006 · Cited by 41
Online panel research: History, concepts, applications and a look at the future
Callegaro, M., Baker, R., Bethlehem, J. G., et al. (2014) · Cited by 38
Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments
Cassese, E. C., Huddy, L., Hartman, T. K., Mason, L., & Weber, C. R. (2013) · PS Political Science & Politics · doi:10.1017/s1049096513001029 · Cited by 37
What Is the Value of Social Science? Challenges for Researchers and Government Funders
Lupia, A. (2013) · PS Political Science & Politics · doi:10.1017/s1049096513001613 · Cited by 36
Druckman, J. N., & Lupia, A. (2012) · Science · doi:10.1126/science.1207808 · Cited by 33
Self-citationKlar, S., Robison, J., & Druckman, J. N. (2013) · Cited by 19
Self-citationDefending the Decimals: How Foolishly False Precision Might Strengthen Social Science
Jeremy Freese, A. (2014) · Sociological Science · doi:10.15195/v1.a29 · Cited by 4
Cited by
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
Prior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268
Jardina, A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108645157 · Cited by 1,189
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Metadata
- Authors
- Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J.
- Venue
- Journal of Experimental Political Science
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015
- Volume
- 2
- Issue
- 2
- Pages
- 109-138
- ISSN
- 2052-2630 (ISSN-L)
- 2052-2649
- OpenAlex
- W2279053323
- Cited by
- 1,228
- References
- 78
- FWCI
- 222.58
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.