The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Abstract
Amazon’s Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples; (b) participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively; (c) realistic compensation rates do not affect data quality; and (d) the data obtained are at least as reliable as those obtained via traditional methods. Overall, MTurk can be used to obtain high-quality data inexpensively and rapidly.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Evidence shows that collecting crowdsourced data via platforms such as MTurk is highly reliable (Buhrmester, Kwang, & Gosling, 2011;Paolacci & Chandler, 2014;Paolacci, Chandler, & Ipeirotis, 2010).”
From Consumers Beware: Online Personalized Pricing in Action! How the Framing of a Mandated Discriminatory Pricing Disclosure Influences Intention to Purchase (2020) · cites this paper as Buhrmester, Kwang, & Gosling, 2011
In the citing paper
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [20]
In the citing paper
“Diversity of participants recruited through platforms such as Prolific is higher than typical Internet samples or North American college-based samples, and the quality of data collected met or exceeded the psychometric standards considered acceptable in published research in the social sciences [98].”
From Disenfranchised Guilt-Pet Owners' Burden. (2022) · cites this paper as [98]
In the citing paper
“We chose Mechanical Turk because research suggests this platform can be used to obtain data from a diverse sample and the data is comparable to samples collected with traditional methods (Buhrmester et al., 2011; Casler et al., 2013).”
From A psychometric analysis of the reading the mind in the eyes test: toward a brief form for research and applied settings (2015) · cites this paper as Buhrmester et al., 2011
Citations
Cites
Society and the Adolescent Self-Image
Rosenberg, M. (1965) · doi:10.1515/9781400876136 · Cited by 30,126
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Self-citation(1999) · Cited by 7,995
Social dominance orientation: A personality variable predicting social and political attitudes.
Pratto, F., Sidanius, J., Stallworth, L. M., & Malle, B. F. (1994) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.67.4.741 · Cited by 4,954
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · cited 1×
“To examine how MTurk samples compare with the diversity of standard Internet samples, we compared the demographics of 3,006 MTurk participants with those in a large Internet sample (Gosling et al., 2004).”
Self-citationSears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119
Self-concept clarity: Measurement, personality correlates and cultural boundaries
Campbell, J. D., Trapnell, P. D., Heine, S. J., Katz, I. M., Lavallee, L. F., & Lehman, D. R. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.1.141 · Cited by 2,088
Simpson, J. A., Rholes, W. S., & Nelligan, J. S. (1992) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.62.3.434 · Cited by 1,321
Research works printed but not linked: 2
- Mason W.A., 2009, Association for Computing Machinery Explorations Newsletter, 11, 100
- 10.1037/0022-3514.51.3.515 · doi:10.1037/0022-3514.51.3.515 · Paper page
Cited by
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Introducing the Short Dark Triad (SD3): A Brief Measure of Dark Personality Traits
Jones, D. N., & Paulhus, D. L. (2014) · Assessment · doi:10.1177/1073191113514105 · Cited by 2,162
Metadata
- Authors
- Buhrmester, M., Kwang, T., & Gosling, S. D.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2011-01
- Volume
- 6
- Issue
- 1
- Pages
- 3-5
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2141708418
- Cited by
- 9,996
- References
- 17
- Citations by year
- 2026: 115 · 2025: 210 · 2024: 322 · 2023: 352 · 2022: 546 · 2021: 859 · 2020: 1068 · 2019: 1060 · 2018: 1101 · 2017: 1099 · 2016: 970 · 2015: 858 · 2014: 666 · 2013: 451 · 2012: 230
- FWCI
- 816.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, online participant recruitment, participant compensation, data quality, task length, online research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? https://app.cochat.ai/papers/10.1177/1745691610393980
- The paper
- Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science. https://doi.org/10.1177/1745691610393980
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.