Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Online experiments allow researchers to collect datasets at times not typical of laboratory studies. We recruit 2336 participants from Amazon Mechanical Turk to examine if participant characteristics and behaviors differ depending on whether the experiment is conducted during the day versus night, and on weekdays versus weekends. Participants make incentivized decisions involving prosociality, punishment, and discounting, and complete a demographic and personality survey. We find no time or day differences in behavior, but do find that participants at nights and on weekends are less experienced with online studies; on weekends are less reflective; and at night are less conscientious and more neurotic. These results are largely robust to finer-grained measures of time and day. We also find that those who participated earlier in the course of the study are more experienced, reflective, and agreeable, but less charitable than later participants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 1×
“Since the appearance of Amazon Mechanical Turk (MTurk), a plethora of studies have validated their use by successfully replicating classic findings from economics and psychology (Paolacci et al. 2010; Horton et al. 2011; Amir et al. 2012; Berinsky et al. 2012; Rand 2012; Arechar et al. 2016).”
BackgroundHeroin addicts have higher discount rates for delayed rewards than non-drug-using controls.
Kirby, K. N., Petry, N. M., & Bickel, W. K. (1999) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.128.1.78 · Cited by 2,171
Divine intuition: Cognitive style influences belief in God.
Shenhav, A., Rand, D. G., & Greene, J. D. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0025391 · Cited by 557
Self-citationHumans display a ‘cooperative phenotype’ that is domain general and temporally stable
Peysakhovich, A., Nowak, M. A., & Rand, D. G. (2014) · Nature Communications · doi:10.1038/ncomms5939 · Cited by 426 · cited 2×
“…is typical for experimental economics, which could have led to more noise; however, we did screen for comprehension of the game payoffs, and prior work with the same short instructions has successfully observed correlations between game play and various other factors (Peysakhovich et al. 2014).”
ResultSelf-citationEconomic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364 · cited 2×
“Since the appearance of Amazon Mechanical Turk (MTurk), a plethora of studies have validated their use by successfully replicating classic findings from economics and psychology (Paolacci et al. 2010; Horton et al. 2011; Amir et al. 2012; Berinsky et al. 2012; Rand 2012; Arechar et al. 2016).”
BackgroundSelf-citationUsing Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · cited 3×
“…or consciousness seem likely to impact task performance (or, more importantly, interact with treatment effects for a given study, e.g. as in Rand et al. (2014) and Chandler et al. (2015)), researchers should use full randomization across treatments and be mindful of when they launch online studies.”
Background
Cited by
Pennycook, G., & Rand, D. G. (2020) · Journal of Personality · doi:10.1111/jopy.12476 · Cited by 879
Self-citationPersonality and prosocial behavior: A theoretical framework and meta-analysis.
Thielmann, I., Spadaro, G., & Balliet, D. (2020) · Psychological Bulletin · doi:10.1037/bul0000217 · Cited by 689
Testing the Waters: Behavior across Participant Pools
Snowberg, E., & Yariv, L. (2021) · American Economic Review · doi:10.1257/aer.20181065 · Cited by 273
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94
“Weekends also permit greater flexibility as weekend workers tend to take a longer time to complete tasks than frequent/professional workers (Arechar et al., 2017).”
BackgroundAssessing and dissociating virtues from the ‘bottom up’: A case study of generosity vs. fairness
Kraft-Todd, G. T., Kleiman-Weiner, M., & Young, L. (2023) · The Journal of Positive Psychology · doi:10.1080/17439760.2022.2154254 · Cited by 7
Self-citation
Metadata
- Authors
- Arechar, A. A., Kraft-Todd, G. T., & Rand, D. G.
- Venue
- Journal of the Economic Science Association
- Publisher
- Cambridge University Press (CUP)
- Published
- 2017-07
- Volume
- 3
- Issue
- 1
- Pages
- 1-11
- ISSN
- 2199-6776 (ISSN-L)
- 2199-6784
- OpenAlex
- W3123190558
- Cited by
- 90
- References
- 9
- FWCI
- 23.79
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. https://app.cochat.ai/papers/10.1007/s40881-017-0035-0
- The paper
- Arechar, A. A., Kraft-Todd, G. T., & Rand, D. G. (2017). Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. Journal of the Economic Science Association. https://doi.org/10.1007/s40881-017-0035-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.