Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents
Open accessAbstract
Researchers have increasingly turned to online convenience samples as sources of survey responses that are easy and inexpensive to collect. As reliance on these sources has grown, so too have concerns about the use of convenience samples in general and Amazon’s Mechanical Turk in particular. We distinguish between “external validity” and theoretical relevance, with the latter being the more important justification for any data collection strategy. We explore an alternative source of online convenience samples, the Lucid Fulcrum Exchange, and assess its suitability for online survey experimental research. Our point of departure is the 2012 study by Berinsky, Huber, and Lenz that compares Amazon’s Mechanical Turk to US national probability samples in terms of respondent characteristics and treatment effect estimates. We replicate these same analyses using a large sample of survey responses on the Lucid platform. Our results indicate that demographic and experimental findings on Lucid track well with US national benchmarks, with the exception of experimental treatments that aim to dispel the “death panel” rumor regarding the Affordable Care Act. We conclude that subjects recruited from the Lucid platform constitute a sample that is suitable for evaluating many social scientific theories, and can serve as a drop-in replacement for many scholars currently conducting research on Mechanical Turk or other similar platforms.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Citizen journalists can positively impact health-related behaviors during a pandemic when the credibility of government and its performance is under question.
“It was also noted that the perception of independent sources can have a positive impact on trust among critics of government and individuals without predispositions towards the source, implying that citizen journalists can positively impact health-related behaviors during a pandemic when the credibility of government and its performance is under question and is being challenged.”
From Understanding media empowerment: citizen journalism in Palestine (2023) · cites this paper as Coppock and McClellan, 2019
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 17,604 · cited 1× · see citing works
“Tversky and Kahneman (1981) show that people take the riskier option when in a “loss frame” rather than a “gain frame.””
BackgroundThe weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1× · see citing works
“Stewart et al. (2015) estimate the pool of active MTurk respondents for a given lab to be approximately 7300 subjects at any one time.”
BackgroundUsing Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715 · see citing works
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532 · see citing works
Shifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155 · see citing works
Fighting misinformation on social media using crowdsourced judgments of news source quality
Pennycook, G., & Rand, D. G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806781116 · Cited by 947 · see citing works
“In the second study, we tested the generalizability of our findings by surveying an additional n = 970 Americans recruited from Lucid, providing a subject pool that is nationally representative on age, gender, ethnicity, and geography (27).”
ResultThe Elusive Backfire Effect: Mass Attitudes’ Steadfast Factual Adherence
Wood, T., & Porter, E. (2019) · Political Behavior · doi:10.1007/s11109-018-9443-y · Cited by 904 · see citing works
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · see citing works
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668 · see citing works
McKnight-Eily, L. R., Okoro, C. A., Strine, T. W., Verlenden, J., Hollis, N. D., Njai, R., Mitchell, E. W., Board, A., Puddy, R., & Thomas, C. (2021) · MMWR. Morbidity and Mortality Weekly Report · doi:10.15585/mmwr.mm7005a3 · Cited by 455 · see citing works
Reliance on emotion promotes belief in fake news
Martel, C., Pennycook, G., & Rand, D. G. (2020) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-020-00252-3 · Cited by 470 · see citing works
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · doi:10.31234/osf.io/uhbk9 · Cited by 390 · see citing works
Kreps, S. E., & Kriner, D. L. (2020) · Science Advances · doi:10.1126/sciadv.abd4563 · Cited by 328 · see citing works
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342 · see citing works
Mental distress among U.S. adults during the COVID-19 pandemic
Twenge, J. M., & Joiner, T. E. (2020) · Journal of Clinical Psychology · doi:10.1002/jclp.23064 · Cited by 330 · see citing works
Emphasizing publishers does not effectively reduce susceptibility to misinformation on social media
Dias, N., Pennycook, G., & Rand, D. G. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-001 · Cited by 145 · see citing works
“We then assigned participants from MTurk and the sampling company Lucid (Coppock & McClellan, 2019) to either rate the trustworthiness of a random subset of the 60 sources (N=251 from MTurk in Study 4, N=253 from Lucid in Study 5) or rate the accuracy of a random subset of the 600 headlines…”
MethodInfluentialIdeology, Not Affect: What Americans Want from Political Representation
Costa, M. (2021) · American Journal of Political Science · doi:10.1111/ajps.12571 · Cited by 118 · see citing works
The social media context interferes with truth discernment
Epstein, Z., Sirlin, N., Arechar, A., Pennycook, G., & Rand, D. (2023) · Science Advances · doi:10.1126/sciadv.abo6169 · Cited by 100 · see citing works
Quantifying the potential persuasive returns to political microtargeting
Tappin, B. M., Wittenberg, C., Hewitt, L. B., Berinsky, A. J., & Rand, D. G. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2216261120 · Cited by 83 · see citing works
Epstein, Z., Pennycook, G., & Rand, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376232 · Cited by 67 · see citing works
Representativeness and Response Validity Across Nine Opt-In Online Samples
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2024) · doi:10.31234/osf.io/h9j2d · Cited by 68 · see citing works
Mental distress among U.S. adults during the COVID-19 pandemic
Twenge, J., & Joiner, T. (2020) · doi:10.31234/osf.io/wc8ud · Cited by 330 · see citing works
Reliance on emotion promotes belief in fake news
Martel, C., Pennycook, G., & Rand, D. G. (2019) · doi:10.31234/osf.io/a2ydw · Cited by 49 · see citing works
Ternovski, J., Kalla, J., & Aronow, P. M. (2021) · doi:10.31219/osf.io/dta97 · Cited by 37 · see citing works
Understanding media empowerment: citizen journalism in Palestine
Horoub, I. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01526-z · Cited by 14 · see citing works
“vaccinations, statistics, and information regarding the virus) (Blair et al., 2017; Coppock and McClellan, 2019; Liao et al., 2010; Raza et al., 2021).”
BackgroundThe Liar’s Dividend: Can Politicians Use Deepfakes and Fake News to Evade Accountability?
Schiff, K. J., Schiff, D. S., & Bueno, N. (2022) · doi:10.31235/osf.io/q6mwn · Cited by 6 · see citing works
Hamrak, B. (2025) · Electoral Studies · doi:10.1016/j.electstud.2025.102967 · Cited by 5 · see citing works
- Authors
- Coppock, A., & McClellan, O. A.
- Affiliations
- Yale University (US)
- Columbia University (US)
- Corresponding author
- Coppock, Alexander
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 6
- Issue
- 1
- ISSN
- 2053-1680 (ISSN-L)
- OpenAlex
- W2911522314
- Cited by
- 987
- References
- 7
- Citations by year
- 2026: 69 · 2025: 142 · 2024: 170 · 2023: 134 · 2022: 207 · 2021: 127 · 2020: 115 · 2019: 17 · 2018: 4 · 2016: 1
- FWCI
- 234.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Research & Politics (publishedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Political Science and International Relations › Electoral Systems and Political Participation
- Keywords
- Amazon Mechanical Turk, generalizability, replication, treatment effect estimates
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
