The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
The Generalizability of Survey Experiments
Open accessAbstract
Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Framing: toward clarification of a fractured paradigm
Entman, R. M. (1993) · Journal of Communication · doi:10.1111/j.1460-2466.1993.tb01304.x · Cited by 16,203
(2002) · Cited by 13,211
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach
Gamson, W. A., & Modigliani, A. (1989) · American Journal of Sociology · doi:10.1086/229213 · Cited by 4,971
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Is Anyone Responsible?: How Television Frames Political Issues
Iyengar, S. (1991) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226388533.001.0001 · Cited by 3,722
Cited by
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
Prior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268
Jardina, A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108645157 · Cited by 1,189
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Coppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987
Pennycook, G., Bear, A., Collins, E. T., & Rand, D. G. (2020) · Management Science · doi:10.1287/mnsc.2019.3478 · Cited by 714
The demographic and political composition of Mechanical Turk samples
Levay, K. E., Freese, J., & Druckman, J. N. (2016) · Sage Open · doi:10.1177/2158244016636433 · Cited by 676
“…& Levine, 2014; Paolacci & Chandler, 2014).3 Similarly, differences in political identity appear to be mostly stable, with MTurk samples frequently containing many more Democrats and liberals than populationbased samples (Berinsky et al., 2012; Krupnikov & Levine, 2014; Mullinix et al., 2015).”
BackgroundMethodInfluentialSelf-citationOnline panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Clayton, K., Blair, S., Busam, J. A., Forstner, S., Glance, J., Green, G., Kawata, A., Kovvuri, A., Martin, J., Morgan, E., et al. (2020) · Political Behavior · doi:10.1007/s11109-019-09533-0 · Cited by 658
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Boas, T. C., Christenson, D. P., & Glick, D. M. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2018.28 · Cited by 570
Demand Effects in Survey Experiments: An Empirical Assessment
MUMMOLO, J., & PETERSON, E. (2019) · American Political Science Review · doi:10.1017/s0003055418000837 · Cited by 548
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
“None of the seven I replicated were in the set of TESS studies replicated Mullinix et al. (2015).”
BackgroundMethodInfluentialThe shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
“In fact, MTurk samples generally provide higher quality data than student samples, community samples, and even some high-quality national samples (Hauser and Schwarz, 2015; Mullinix et al ., 2015; Thomas and Clifford, 2017; Anson, 2018).”
BackgroundProcessing political misinformation: comprehending the Trump phenomenon
Swire, B., Berinsky, A. J., Lewandowsky, S., & Ecker, U. K. H. (2017) · Royal Society Open Science · doi:10.1098/rsos.160802 · Cited by 498
Consequences of Routine Work-Schedule Instability for Worker Health and Well-Being
Schneider, D., & Harknett, K. (2019) · American Sociological Review · doi:10.1177/0003122418823184 · Cited by 496
Reliance on emotion promotes belief in fake news
Martel, C., Pennycook, G., & Rand, D. G. (2020) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-020-00252-3 · Cited by 470
Generalizability of heterogeneous treatment effect estimates across samples
Coppock, A., Leeper, T. J., & Mullinix, K. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1808083115 · Cited by 445
Self-citationSampling in software engineering research: a critical review and guidelines
Baltes, S., & Ralph, P. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10072-8 · Cited by 382
Frenemies: How Social Media Polarizes America
Settle, J. E. (2018) · doi:10.1017/9781108560573 · Cited by 377
Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments
Guess, A., & Coppock, A. (2020) · British Journal of Political Science · doi:10.1017/s0007123418000327 · Cited by 372
The Very Efficient Assessment of Need for Cognition: Developing a Six-Item Version
Lins de Holanda Coelho, G., H. P. Hanel, P., & J. Wolf, L. (2020) · Assessment · doi:10.1177/1073191118793208 · Cited by 358
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2018) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 340
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies
Gaddis, S. (2017) · Sociological Science · doi:10.15195/v4.a19 · Cited by 285
Does Conjoint Analysis Mitigate Social Desirability Bias?
Horiuchi, Y., Markovich, Z., & Yamamoto, T. (2022) · Political Analysis · doi:10.1017/pan.2021.30 · Cited by 277
No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments
Kane, J. V., & Barabas, J. (2019) · American Journal of Political Science · doi:10.1111/ajps.12396 · Cited by 276
Addressing climate change with behavioral science: A global intervention tournament in 63 countries
Vlasceanu, M., Doell, K. C., Bak-Coleman, J. B., Todorova, B., Berkebile-Weinberg, M. M., Grayson, S. J., Patel, Y., Goldwert, D., Pei, Y., Chakroff, A., et al. (2024) · Science Advances · doi:10.1126/sciadv.adj5778 · Cited by 269
Newman, B., Merolla, J. L., Shah, S., Lemi, D. C., Collingwood, L., & Ramakrishnan, S. K. (2021) · British Journal of Political Science · doi:10.1017/s0007123419000590 · Cited by 267
“While our study is not nationally representative, experimental studies on MTurk often replicate those done on representative samples (Berinsky, Huber and Lenz 2012; Mullinix et al. 2015; Weinberg, Freese and McElhattan 2014), and panel studies using national samples are often cost prohibitive.”
ResultThe potential of mobility as a service bundles as a mobility management tool
Matyas, M., & Kamargianni, M. (2019) · Transportation · doi:10.1007/s11116-018-9913-4 · Cited by 266
Income inequality influences perceptions of legitimate income differences
Trump, K.-S. (2018) · British Journal of Political Science · doi:10.1017/s0007123416000326 · Cited by 236
Elite rhetoric can undermine democratic norms
Clayton, K., Davis, N. T., Nyhan, B., Porter, E., Ryan, T. J., & Wood, T. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2024125118 · Cited by 226
Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity
Voelkel, J. G., Stagnaro, M., Chu, J., Pink, S. L., Mernyk, J. S., Redekopp, C., Ghezae, I., Cashman, M., Adjodah, D., Allen, L., et al. (2023) · doi:10.31219/osf.io/y79u5 · Cited by 174
Self-citationMegastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity
Voelkel, J. G., Stagnaro, M. N., Chu, J. Y., Pink, S. L., Mernyk, J. S., Redekopp, C., Ghezae, I., Cashman, M., Adjodah, D., Allen, L. G., et al. (2024) · Science · doi:10.1126/science.adh4764 · Cited by 174
Self-citationLearning Together Slowly: Bayesian Learning about Political Facts
Hill, S. J. (2017) · The Journal of Politics · doi:10.1086/692739 · Cited by 159
Correcting inaccurate metaperceptions reduces Americans’ support for partisan violence
Mernyk, J. S., Pink, S. L., Druckman, J. N., & Willer, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2116851119 · Cited by 146
Self-citationEmphasizing publishers does not effectively reduce susceptibility to misinformation on social media
Dias, N., Pennycook, G., & Rand, D. G. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-001 · Cited by 145
Bayesian or biased? Analytic thinking and political belief updating
Tappin, B. M., Pennycook, G., & Rand, D. G. (2020) · Cognition · doi:10.1016/j.cognition.2020.104375 · Cited by 127
Representative Bureaucracy and Attitudes Toward Automated Decision Making
Miller, S. M., & Keiser, L. R. (2021) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muaa019 · Cited by 127
Flynn, A., Powell, A., Scott, A. J., & Cama, E. (2022) · The British Journal of Criminology · doi:10.1093/bjc/azab111 · Cited by 120
Graham, A., Pickett, J. T., & Cullen, F. T. (2021) · Crime & Delinquency · doi:10.1177/0011128720977439 · Cited by 114
“Recent research suggests that observational findings from such samples often fail to generalize (Thompson & Pickett, 2019), even though experimental results tend to be externally valid (Coppock, 2019; Mullinix et al., 2015).”
BackgroundMethodInfluentialKnudsen, E. (2021) · Scandinavian Political Studies · doi:10.1111/1467-9477.12186 · Cited by 106
What does the public want police to do during pandemics? A national experiment
Nix, J., Ivanov, S., & Pickett, J. T. (2021) · Criminology & Public Policy · doi:10.1111/1745-9133.12535 · Cited by 99
Through the Grapevine: Informational Consequences of Interpersonal Political Communication
CARLSON, T. N. (2019) · American Political Science Review · doi:10.1017/s000305541900008x · Cited by 95
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94
Ong, C. W., Pierce, B. G., Petersen, J. M., Barney, J. L., Fruge, J. E., Levin, M. E., & Twohig, M. P. (2020) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2020.08.007 · Cited by 87
The Past, Present, and Future of Experimental Methods in the Social Sciences
Mize, T. D., & Manago, B. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102799 · Cited by 76
Representativeness and Response Validity Across Nine Opt-In Online Samples
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2024) · doi:10.31234/osf.io/h9j2d · Cited by 68
Self-citationResistance to Position Change, Motivated Reasoning, and Polarization
Stanley, M. L., Henne, P., Yang, B. W., & De Brigard, F. (2020) · Political Behavior · doi:10.1007/s11109-019-09526-z · Cited by 58
Reliance on emotion promotes belief in fake news
Martel, C., Pennycook, G., & Rand, D. G. (2019) · doi:10.31234/osf.io/a2ydw · Cited by 49
Drones and Support for the Use of Force
Walsh, J., & Schulzke, M. (2018) · doi:10.3998/mpub.9946611 · Cited by 49
Wulff, J. N., & Villadsen, A. R. (2020) · European Management Review · doi:10.1111/emre.12342 · Cited by 46
Jeon, Y. A. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.03.028 · Cited by 39
Becker, B. (2020) · Social Justice Research · doi:10.1007/s11211-019-00343-7 · Cited by 37
Like, Share, Comment, and Repeat: Far-right Messages, Emotions, and Amplification in Social Media
Doroshenko, L., & Tu, F. (2023) · Journal of Information Technology & Politics · doi:10.1080/19331681.2022.2097358 · Cited by 25
Motives Don't Matter? Motive Attribution and Counterterrorism Policy
Canetti, D., Gubler, J., & Zeitzoff, T. (2021) · Political Psychology · doi:10.1111/pops.12708 · Cited by 6
Kang, B., & Wu, S. (2023) · Journal of Experimental Criminology · doi:10.1007/s11292-022-09512-2 · Cited by 6
An audit of social science survey experiments
Rauf, T., Voelkel, J. G., Druckman, J., & Freese, J. (2025) · Public Opinion Quarterly · doi:10.1093/poq/nfaf052 · Cited by 4
Self-citation
Metadata
- Authors
- Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J.
- Venue
- Journal of Experimental Political Science
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015
- Volume
- 2
- Issue
- 2
- Pages
- 109-138
- ISSN
- 2052-2630 (ISSN-L)
- 2052-2649
- OpenAlex
- W2279053323
- Cited by
- 1,228
- References
- 78
- FWCI
- 222.58
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.