The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Turking in the time of COVID
Open accessAbstract
On March 16, 2020, the US Government introduced strict social distancing protocols for the United States in an effort to stem the spread of the COVID-19 pandemic. This had an immediate major effect on the job market, with millions of Americans forced to find alternative ways to make a living from home. As online labor markets like Amazon Mechanical Turk (MTurk) play a major role in social science research, concerns have been raised that the pandemic may be reducing the diversity of subjects participating in experiments. Here, we investigate this possibility empirically. Specifically, we look at 15,539 responses gathered in 23 studies run on MTurk between February and July 2020, examining the distribution of gender, age, ethnicity, political preference, and analytic cognitive style. We find notable changes on some of the measures following the imposition of nationwide social distancing: participants are more likely to be less reflective (as measured by the Cognitive Reflection Test), and somewhat less likely to be white, Democrats (traditionally over-represented on MTurk), and experienced with MTurk. Most of these differences are explained by an influx of new participants into the MTurk subject pool who are more diverse and representative - but also less attentive - than previous MTurkers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is likely due in large part to participants on MTurk being quite a bit more attentive than participants on Lucid50”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715
Self-citationReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Evans, D. C., Schott, R. K., Larson, D. W., Brown, C. M., & Ryan, M. J. (2014) · Nature Communications · doi:10.1038/ncomms4289 · Cited by 942
Intuitive Cooperation and the Social Heuristics Hypothesis: Evidence from 15 Time Constraint Studies
Rand, D. G., Peysakhovich, A., Kraft‐Todd, G., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2013) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 772
Self-citationThe Online Laboratory: Conducting Experiments in a Real Labor Market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2010) · National Bureau of Economic Research · doi:10.3386/w15961 · Cited by 402
Self-citationRobinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203
Participant carelessness and fraud: Consequences for clinical research and potential solutions.
Chandler, J., Sisso, I., & Shapiro, D. N. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000479 · Cited by 124
No Participant Left Behind: Conducting Science During COVID-19
Lourenco, S. F., & Tasimi, A. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.05.003 · Cited by 111
Demographic Stability on Mechanical Turk Despite COVID-19
Moss, A. J., Rosenzweig, C., Robinson, J., & Litman, L. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.05.014 · Cited by 87
The persistence of pay inequality: The gender pay gap in an anonymous online labor market
Litman, L., Robinson, J., Rosen, Z., Rosenzweig, C., Waxman, J., & Bates, L. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229383 · Cited by 74
The online laboratory: conducting experiments in a real labor market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2011) · Experimental Economics · doi:10.1007/s10683-011-9273-9 · Cited by 9
Self-citation
Cited by
Sorted by citations
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021) · Information Systems Research · doi:10.1287/isre.2021.1079 · Cited by 625
Pennycook, G., & Rand, D. G. (2022) · Nature Communications · doi:10.1038/s41467-022-30073-5 · Cited by 270
Self-citationThe Generalizability of Online Experiments Conducted During the COVID-19 Pandemic
Peyton, K., Huber, G. A., & Coppock, A. (2021) · Journal of Experimental Political Science · doi:10.1017/xps.2021.17 · Cited by 174
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166
Metadata
- Authors
- Arechar, A. A., & Rand, D. G.
- Affiliations
- Centro de Investigación y Docencia Económicas (MX)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Arechar, Antonio A.
- Venue
- Behavior Research Methods
- Published
- 2021-05-07
- Volume
- 53
- Issue
- 6
- Pages
- 2591-2595
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC8103881
- OpenAlex
- W3157385754
- Cited by
- 105
- References
- 14
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 14 · 2023: 29 · 2022: 27 · 2021: 19
- FWCI
- 19.04
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- Amazon Mechanical Turk, COVID-19 pandemic, social distancing, digital labour markets, participant diversity, demographic representation, cognitive reflection, analytic cognitive style, gender, ethnicity, political preferences
- MeSH terms
- COVID-19, SARS-CoV-2, Attention, Humans, Social Sciences, United States, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
