The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index, and 1 are not checked yet.
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Abstract
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this market have been assessed on dimensions that are universally relevant to understanding whether, why, and when they should be recruited as research participants. We discuss the characteristics of MTurk as a participant pool for psychology and other social sciences, highlighting the traits of the MTurk samples, why people become MTurk workers and research participants, and how data quality on MTurk compares to that from other pools and depends on controllable and uncontrollable factors.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Evidence shows that collecting crowdsourced data via platforms such as MTurk is highly reliable (Buhrmester, Kwang, & Gosling, 2011;Paolacci & Chandler, 2014;Paolacci, Chandler, & Ipeirotis, 2010).”
From Consumers Beware: Online Personalized Pricing in Action! How the Framing of a Mandated Discriminatory Pricing Disclosure Influences Intention to Purchase (2020) · cites this paper as Paolacci & Chandler, 2014
In the citing paper
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [80]
In the citing paper
“MTurk is a survey tool that has been used in several studies as an acceptable means to collect individual-level data from the general population of interest [57].”
From Use of AI-based tools for healthcare purposes: a survey study from consumers’ perspectives (2020) · cites this paper as [57]
In the citing paper
“the study above constitutes the first systematic examination of ProA's data quality.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Paolacci & Chandler, 2014
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citationConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 5×
“, 2013), and may search the Internet for the answers to factually verifiable questions (Goodman et al., 2013).”
BackgroundInfluentialAmerican journal of political science
(1983) · Electoral Studies · doi:10.1016/0261-3794(83)90076-8 · Cited by 1,985
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
(2010) · Nature Cell Biology · doi:10.1038/ncb0510-413b · Cited by 1,604
Research works printed but not linked: 5
- Aker A., 2012, Paper presented at the 8th International Conference on Language Resources and Evaluation
- Kazai G., 2010, Paper presented at CrowdConf 2010
- Klein R. A., Social Psychology
- Rogstadius J., 2011, Paper presented at the 5th International AAAI Conference on Weblogs and Social Media
- Suri S., 2011, Paper presented at the 3rd Association for the Advancement of Artificial Intelligence Human Computation Workshop
Cited by
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Do Investors Value Sustainability? A Natural Experiment Examining Ranking and Fund Flows
HARTZMARK, S. M., & SUSSMAN, A. B. (2019) · The Journal of Finance · doi:10.1111/jofi.12841 · Cited by 1,711
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338
Metadata
- Authors
- Paolacci, G., & Chandler, J.
- Affiliations
- Erasmus University Rotterdam (NL)
- University of Michigan (US)
- Venue
- Current Directions in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-06
- Volume
- 23
- Issue
- 3
- Pages
- 184-188
- ISSN
- 0963-7214 (ISSN-L)
- 1467-8721
- OpenAlex
- W2137872373
- Cited by
- 2,136
- References
- 41
- Citations by year
- 2026: 40 · 2025: 55 · 2024: 87 · 2023: 118 · 2022: 155 · 2021: 223 · 2020: 302 · 2019: 258 · 2018: 280 · 2017: 258 · 2016: 202 · 2015: 129 · 2014: 28
- FWCI
- 306.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Current Directions in Psychological Science (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Keywords
- Mechanical Turk, digital labour markets, participant recruitment, participant characteristics, data quality, crowdsourcing, Amazon Mechanical Turk, survey data, experimental data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Inside the Turk: Understanding Mechanical Turk as a Participant Pool. https://app.cochat.ai/papers/10.1177/0963721414531598
- The paper
- Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a Participant Pool. Current Directions in Psychological Science. https://doi.org/10.1177/0963721414531598
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.