Cited by 950Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseExperimental Behavioral Economics Studies

Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

Open access
Chandler, Jesse, Mueller, Pam, Paolacci, Gabriele·2014·Behavior Research Methods

Abstract

Crowdsourcing services--particularly Amazon Mechanical Turk--have made it easy for behavioral scientists to recruit research participants. However, researchers have overlooked crucial differences between crowdsourcing and traditional recruitment methods that provide unique opportunities and challenges. We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection. Using these techniques can minimize the effects of previously ignored drawbacks and expand the scope of crowdsourcing as a tool for psychological research.

Integrity signals

Registry

Crossref · 10.3758/s13428-013-0365-7

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Chandler, J., Mueller, P., & Paolacci, G.
Affiliations
  • Princeton University (US)
  • Erasmus University Rotterdam (NL)
Corresponding author
Chandler, Jesse
Venue
Behavior Research Methods
Published
2013-07-09
Volume
46
Issue
1
Pages
112-130
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
OpenAlex
W1975450585
Cited by
950
References
57
Citations by year
2026: 18 · 2025: 29 · 2024: 50 · 2023: 54 · 2022: 61 · 2021: 84 · 2020: 120 · 2019: 116 · 2018: 101 · 2017: 111 · 2016: 85 · 2015: 79 · 2014: 32 · 2013: 7 · 2012: 1
FWCI
102.24
Type
journal article
Language
en
Open access
Locations
  • Behavior Research Methods (publishedVersion)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Safety Research › Experimental Behavioral Economics Studies
Keywords
Amazon Mechanical Turk, crowdsourcing, participant recruitment, prescreening, longitudinal data collection, behavioral research
MeSH terms
Analysis of Variance, Data Collection, Humans, Longitudinal Studies, Male, Population Surveillance, Practice, Psychological, Research Personnel, Task Performance and Analysis, Video Games, Internet, Information Dissemination, Behavioral Research, Research Subjects, Health Literacy, Crowdsourcing, Healthy Volunteers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review