The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk
Abstract
Researchers in a variety of disciplines use Amazon’s crowdsourcing platform called Mechanical Turk as a way to collect data from a respondent pool that is much more diverse than a typical student sample. The platform also provides cost efficiencies over other online panel services and data can be collected very quickly. However, some researchers have been slower to try the platform, perhaps because of a lack of awareness of its functions or concerns with validity. This article provides an overview of Mechanical Turk as an academic research platform and a critical examination of its strengths and weaknesses for research. Guidelines for collecting data that address issues of validity, reliability, and ethics are presented.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Respondents were recruited via Amazon Mechanical Turk (MTurk), which has been utilized for many social science studies for sample recruitment, and data quality confirmed in previous studies (e.g., Buhrmester et al., 2011; Sheehan, 2018).”
From Fostering Employee Trust via Effective Supervisory Communication during the COVID-19 Pandemic: Through the Lens of Motivating Language Theory (2022) · cites this paper as Sheehan, 2018
In the citing paper
“Another benefit of using MTurk is that the population is generally more diverse than the typical sample of college students used in social science research [60].”
From Internalized Racism and Mental Health: The Moderating Role of Collective Racial Self-Esteem (2024) · cites this paper as [60]
Citations
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,177
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Introducing the Short Dark Triad (SD3): A Brief Measure of Dark Personality Traits
Jones, D. N., & Paulhus, D. L. (2014) · Assessment · doi:10.1177/1073191113514105 · Cited by 2,162
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Research works printed but not linked: 2
- 10.1007/s10683-011-9273-9 · DOI
- Howe J., 2006, Wired Magazine, 14, 1
Cited by
Sorted by citations
Araujo, T. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.051 · Cited by 1,156
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
Ashfaq, M., Yun, J., Yu, S., & Loureiro, S. M. C. (2020) · Telematics and Informatics · doi:10.1016/j.tele.2020.101473 · Cited by 1,200
Americans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728
Russell, B. S., Hutchison, M., Tambling, R. R., Tomkunas, A. J., & Horton, A. L. (2020) · Child Psychiatry & Human Development · doi:10.1007/s10578-020-01037-x · Cited by 778
Metadata
- Authors
- Sheehan, K. B.
- Affiliations
- University of Oregon (US)
- Corresponding author
- Sheehan, Kim Bartel
- Venue
- Communication Monographs
- Publisher
- Informa UK Limited
- Published
- 2017-07-04
- Volume
- 85
- Issue
- 1
- Pages
- 140-156
- ISSN
- 0363-7751 (ISSN-L)
- 1479-5787
- OpenAlex
- W2724582867
- Cited by
- 408
- References
- 46
- Citations by year
- 2026: 13 · 2025: 33 · 2024: 39 · 2023: 43 · 2022: 78 · 2021: 73 · 2020: 64 · 2019: 49 · 2018: 12 · 2017: 2
- FWCI
- 41.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communication Monographs (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- Amazon Mechanical Turk, crowdsourcing, online data collection, validity, reliability statistics, research ethics, cost efficiency, online panels
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
