The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index, and 2 are not checked yet.
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Open accessAbstract
Using capture-recapture analysis we estimate the effective size of the active Amazon Mechanical Turk (MTurk) population that a typical laboratory can access to be about 7,300 workers. We also estimate that the time taken for half of the workers to leave the MTurk pool and be replaced is about 7 months. Each laboratory has its own population pool which overlaps, often extensively, with the hundreds of other laboratories using MTurk. Our estimate is based on a sample of 114,460 completed sessions from 33,408 unique participants and 689 sessions across seven laboratories in the US, Europe, and Australia from January 2012 to March 2015.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The standard undergraduate subject pools are not nearly large or diverse enough to achieve this, nor are crowdsourcing platforms like Amazon Mechanical Turk (Stewart et al., 2015).”
From A critical period for second language acquisition: Evidence from 2/3 million English speakers (2018) · cites this paper as Stewart et al., 2015
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · cited 1×
“Buhrmester et al. (2011) found that increasing pay rates increased the rate at which workers were recruited but did not affect data quality.”
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041 · cited 1×
“Participants often conform to demand characteristics (Orne, 1962), and MTurk workers may infer demands, correctly or otherwise, from debriefings from earlier experiments.”
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · cited 3×
“MTurk workers are used in psychology, economics, and political science, with classic findings replicated in all three domains (Berinsky et al., 2012; Goodman, Cryder, & Cheema, 2013; Horton, Rand, We thank Jon Baron, Leif Nelson, Stian Reimers, and two anonymous reviewers for suggestions.”
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 2×
“…at large, certainly more representative than an undergraduate sample, and produces reliable results at low cost (Behrend, Sharek, Meade, & Wiebe, 2011; Berinsky, Hu-ber, & Lenz, 2012; Buhrmester, Kwang, & Gosling, 2011; Paolacci, Chandler, & Ipeirotis, 2010; Woods, Velasco, A., X., & Spence, 2015).”
Self-citationThe Estimation of Animal Abundance and Related Parameters.
Cormack, R. M., & Seber, G. A. F. (1984) · Biometrics · doi:10.2307/2531423 · Cited by 3,289
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Analysis and Management of Animal Populations
Grand, J. B., Williams, B. K., Nichols, J. D., & Conroy, M. J. (2003) · The Journal of Wildlife Management · doi:10.2307/3802722 · Cited by 2,671
Statistical inference from capture data on closed animal populations
Otis, D. L., Burnham, K. P., White, G. C., & Anderson, D. R. (1978) · Wildlife Monographs · Cited by 2,442
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 1×
“MTurk workers are used in psychology, economics, and political science, with classic findings replicated in all three domains (Berinsky et al., 2012; Goodman, Cryder, & Cheema, 2013; Horton, Rand, We thank Jon Baron, Leif Nelson, Stian Reimers, and two anonymous reviewers for suggestions.”
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 1×
“Many factors differ across our laboratories—such as the description of tasks, duration, posting time, requester reputation, or even just randomness in early update of HITs (Salganik, Dodds, & Watts, 2006)— and so experimental manipulation of these factors is required to make causal claims.”
Research works printed but not linked: 4
- Simonsohn, 2013, Evaluating replication results.
- Casey, 2015, Demographic characteristics of a large sample of us workers.
- Mullinix, 2014, The generalizability of survey experiments.
- Woods, 2015, Conducting perception research over the internet: A tutorial review.
Cited by
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
“…from an effective population size of only around 7000 participants (and not 500 K, as MTurk advertises), because a small number of MTurk workers are highly active, and consequently usually complete most HITs before other, less activeworkers have had a chance to see them (Stewart et al., 2015).”
BackgroundConducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
“The number of active users is unknown but is likely close to 15,000 individual US workers at any given time (Stewart et al. 2015; see also Fort et al. 2011), with perhaps a similar number of Indian users (see Indian Workers 56 Chandler · Shapiro A nn u.”
BackgroundResultInfluentialSelf-citationAn MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
“…years have seen an exponential increase in psychological studies being conducted on MTurk (Buhrmester, Talaifar, & Gosling, 2018), likely because MTurk is an efficient method for collecting large amounts of data (Buhrmester, Kwang, & Gosling, 2011; Buhrmester et al., 2018; Stewart et al., 2015).”
MethodMTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
“This helps identify MTurkers who attempt self-misrepre-sentation to qualify for a particular study (Stewart et al., 2015).”
Background
Metadata
- Authors
- Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J.
- Affiliations
- University of Warwick (GB)
- Columbia University (US)
- University College London (GB)
- University of Chicago (US)
- UNSW Sydney (AU)
- Erasmus University Rotterdam (NL)
- University of Michigan (US)
- Corresponding author
- Stewart, Neil
- ORCID
- Venue
- Judgment and Decision Making
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-09
- Volume
- 10
- Issue
- 5
- Pages
- 479-491
- ISSN
- 1930-2975 (ISSN-L)
- OpenAlex
- W1877906231
- Cited by
- 304
- References
- 34
- Citations by year
- 2026: 1 · 2025: 14 · 2024: 19 · 2023: 18 · 2022: 27 · 2021: 26 · 2020: 31 · 2019: 42 · 2018: 46 · 2017: 49 · 2016: 23 · 2015: 4
- FWCI
- 61.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Judgment and Decision Making (publishedVersion, cc-by)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- Judgment and Decision Making
- UK Data Archive
- Funders
- National Science Foundation (0951516, 1227462, ES/K002201/1)
- Leverhulme Trust (RP2012-V022, ES/K002201/1)
- Economic and Social Research Council (ES/K002201/1, RP2012-V-022, K002201, ES/K004948/1, ES/K002201/, ES/K002201/1 and ES/K004948/1)
- Australian Research Council (FT110100151, LP120100224)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Mathematics › Statistics and Probability › Census and Population Estimation
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- Amazon Mechanical Turk, digital labour markets, population size estimation, employee turnover, participant recruitment, crowdsourcing, capture-recapture analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. https://app.cochat.ai/papers/10.1017/s1930297500005611
- The paper
- Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making. https://doi.org/10.1017/s1930297500005611
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.