CoChat
Sign Up
Cited by 1,617Mobile Crowdsensing and CrowdsourcingNeural and Behavioral Psychology StudiesExperimental Behavioral Economics Studies

The DOI matches its Crossref record; 50 of the 52 references it cites were found in a registry or an index.

Evaluating amazon’s mechanical turk as a tool for experimental behavioral research

Open access
Crump, Matthew J. C., McDonnell, John V., Gureckis, Todd M.·2013·PLoS ONE
DOI 10.1371/journal.pone.0057410PMC PMC3596391
Open this paper in CoChat — free accountOpen PDF

Abstract

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0057410

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Congruent and Incongruent RTs, Error Rates and Individual Stroop Scores by Mean RT. A. Mean RTs and error rates for congruent and incongruent Stroop items with standard error bars. B. Individual subject Stroop difference scores (incongruent-congruent) plotted as a function of individual subject mean RTs.
Figure 1 Congruent and Incongruent RTs, Error Rates and Individual Stroop Scores by Mean RT. A. Mean RTs and error rates for congruent and incongruent Stroop items with standard error bars. B. Individual subject Stroop difference scores (incongruent-congruent) plotted as a function of individual subject mean RTs.CC BY · PubMed Central
Figure 2 Repeat and Switch RTs, Error Rates and Individual Switch costs by mean RT. A. Mean RTs and error rates for task repeat and switch trials with standard error bars. B. Individual subject switch costs (switch-repeat) plotted as a function of individual subject mean RTs.
Figure 2 Repeat and Switch RTs, Error Rates and Individual Switch costs by mean RT. A. Mean RTs and error rates for task repeat and switch trials with standard error bars. B. Individual subject switch costs (switch-repeat) plotted as a function of individual subject mean RTs.CC BY · PubMed Central
Figure 3 Compatible and Incompatible RTs, Error Rates and Individual Flanker Scores by Mean RT. A. Mean RTs and error rates for compatible and incompatible flanker items with standard error bars. B. Individual subject flanker scores (incompatible-compatible) plotted as a function of individual subject mean RTs.
Figure 3 Compatible and Incompatible RTs, Error Rates and Individual Flanker Scores by Mean RT. A. Mean RTs and error rates for compatible and incompatible flanker items with standard error bars. B. Individual subject flanker scores (incompatible-compatible) plotted as a function of individual subject mean RTs.CC BY · PubMed Central
Figure 4 Compatible and Incompatible RTs, Error Rates, and Individual Simon Scores by Mean RT. A. Mean RTs and error rates for compatible and incompatible Simon trials with standard error bars. B. Individual subject Simon scores (incompatible-compatible) plotted as a function of individual subject mean RTs.
Figure 4 Compatible and Incompatible RTs, Error Rates, and Individual Simon Scores by Mean RT. A. Mean RTs and error rates for compatible and incompatible Simon trials with standard error bars. B. Individual subject Simon scores (incompatible-compatible) plotted as a function of individual subject mean RTs.CC BY · PubMed Central
Figure 5 Visual Cuing: Cued and Uncued Mean RTs as a function of CSTOA. Mean RTs for cued and uncued trials as a function of cue-target stimulus onset asynchrony with standard error bars.
Figure 5 Visual Cuing: Cued and Uncued Mean RTs as a function of CSTOA. Mean RTs for cued and uncued trials as a function of cue-target stimulus onset asynchrony with standard error bars.CC BY · PubMed Central
Figure 6 Attentional Blink: Mean T2 Proportion Correct as a function of T1–T2 Lag. Mean T2 (second target) proportion correct as a function of T1–T2 lag with standard error bars.
Figure 6 Attentional Blink: Mean T2 Proportion Correct as a function of T1–T2 Lag. Mean T2 (second target) proportion correct as a function of T1–T2 lag with standard error bars.CC BY · PubMed Central
Figure 7 Masked Priming: Compatible and Incompatible Mean RTs and Error Rates Across Prime Durations. Mean RTs and error rates for compatible and incompatible masked prime trials as a function of prime duration with standard error bars.
Figure 7 Masked Priming: Compatible and Incompatible Mean RTs and Error Rates Across Prime Durations. Mean RTs and error rates for compatible and incompatible masked prime trials as a function of prime duration with standard error bars.CC BY · PubMed Central
Figure 8 Cognitive Learning: A comparison between the learning curves reported in Nosfosky et al. (1994) data and the AMT replication data in Experiment 8. The probability of classification error as a function of training block. The top panel shows the learning curves estimated by Nosfosky et al. [38] using 120 participants (40 per learning problem) who each performed two randomly selected problems. The right panel shows our AMT data with 228 participants, each who performed only one problem (38 per condition). We ended the experiment after 15 blocks, although Nosofsky et al. stopped after 25. Thus, the Nosofsky et al. data have been truncated to facilitate visual comparison.
Figure 8 Cognitive Learning: A comparison between the learning curves reported in Nosfosky et al. (1994) data and the AMT replication data in Experiment 8. The probability of classification error as a function of training block. The top panel shows the learning curves estimated by Nosfosky et al. [38] using 120 participants (40 per learning problem) who each performed two randomly selected problems. The right panel shows our AMT data with 228 participants, each who performed only one problem (38 per condition). We ended the experiment after 15 blocks, although Nosofsky et al. stopped after 25. Thus, the Nosofsky et al. data have been truncated to facilitate visual comparison.CC BY · PubMed Central
Figure 9 Cognitive Learning: The average number of block to criterion for each problem, an index of problem difficulty. The average number of blocks it took participants to reach criterion (2 blocks of 16 trials in a row with no mistakes) in each problem. The white bars show the estimated average number of blocks to criterion reported by Nosofsky et al. [38].
Figure 9 Cognitive Learning: The average number of block to criterion for each problem, an index of problem difficulty. The average number of blocks it took participants to reach criterion (2 blocks of 16 trials in a row with no mistakes) in each problem. The white bars show the estimated average number of blocks to criterion reported by Nosofsky et al. [38].CC BY · PubMed Central
Figure 10 Cognitive Learning: The learning curves for Shepard et al. Type II and IV problems based on task incentives. The probability of classification error as a function of training block, learning problem and incentive for Experiment 9. The incentive structure had little impact on performance within each problem.
Figure 10 Cognitive Learning: The learning curves for Shepard et al. Type II and IV problems based on task incentives. The probability of classification error as a function of training block, learning problem and incentive for Experiment 9. The incentive structure had little impact on performance within each problem.CC BY · PubMed Central
Figure 11 Cognitive Learning: The learning curves for Shepard et al. problems I, II, IV, and VI in Experiment 10. The top panel compared the results of Nosfosky et al. [38] to the results of Experiment 10. The bottom compares the results of Lewandowsky [40] to the results of Experiment 10 giving two different views of the relationship between the online and laboratory based data. Overall, the Type II problem seems more difficult than in previous report (as is the Type VI). However, in general, the instruction manipulation increased the congruence between the online and laboratory data.
Figure 11 Cognitive Learning: The learning curves for Shepard et al. problems I, II, IV, and VI in Experiment 10. The top panel compared the results of Nosfosky et al. [38] to the results of Experiment 10. The bottom compares the results of Lewandowsky [40] to the results of Experiment 10 giving two different views of the relationship between the online and laboratory based data. Overall, the Type II problem seems more difficult than in previous report (as is the Type VI). However, in general, the instruction manipulation increased the congruence between the online and laboratory data.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 52 references

Research works printed but not linked: 1

  1. Ipeirotis PG (2010) Demographics of Mechanical Turk (Tech. Rep. No. CeDER-10-01). New York: New York University. Available: http://hdl.handle.net/2451/29585. Accessed 2012 Jun 1.

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M.
Affiliations
  • City University of New York (US)
  • New York University (US)
Corresponding author
Crump, Matthew J. C.
ORCID
Venue
PLoS ONE
Publisher
Public Library of Science (PLoS)
Published
2013-03-13
Volume
8
Issue
3
Pages
e57410
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0057410
PMC
PMC3596391
OpenAlex
W2080211902
Semantic Scholar
d59101076dd96b6b39f58c3d3e3747e5cf5f98ab
Cited by
1,617
References
52
Citations by year
2026: 25 · 2025: 63 · 2024: 83 · 2023: 108 · 2022: 162 · 2021: 171 · 2020: 164 · 2019: 138 · 2018: 164 · 2017: 155 · 2016: 146 · 2015: 129 · 2014: 86 · 2013: 20
FWCI
192.44
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • CUNY Academic Works (City University of New York) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
Keywords
Amazon Mechanical Turk, online crowdsourcing, replication, Stroop task, attentional blink, subliminal priming, category learning, Simon task, flanker task, stimulus presentation
MeSH terms
Attention, Cues, Humans, Learning, Reaction Time, Internet, Behavioral Research, Learning Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review