The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Open accessAbstract
A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“User studies for machine learning research often depend on online crowd platforms such as Amazon Mechanical Turk, which can lead to ethical issues.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [91]
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Motivation through the design of work: test of a theory
Hackman, J. R., & Oldham, G. R. (1976) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(76)90016-7 · Cited by 8,962
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Software Framework for Topic Modelling with Large Corpora
Rek, R. R. U., & Sojka, P. (2010) · doi:10.13140/2.1.2393.1847 · Cited by 3,647
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
An Introduction to Statistical Learning: with Applications in R
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021) · Springer Texts in Statistics · doi:10.1007/978-1-0716-1418-1 · Cited by 1,755
Layers of Silence, Arenas of Voice: The Ecology of Visible and Invisible Work
Star, S. L., & Strauss, A. (1999) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1008651105359 · Cited by 1,500
Understanding Self-Report Bias in Organizational Behavior Research
Donaldson, S. I., & Grant-Vallone, E. J. (2002) · Journal of Business and Psychology · doi:10.1023/a:1019637632584 · Cited by 1,068
Research works printed but not linked: 9
- 10.1016/j.chb.2015.08.031 · DOI · Paper page
- 10.5555/1866696.1866697 · doi:10.5555/1866696.1866697
- 10.5555/2908698.2908703 · doi:10.5555/2908698.2908703
- 10.5555/1613715.1613763 · doi:10.5555/1613715.1613763
- 10.5555/540298 · doi:10.5555/540298
- 10.5555/2517747 · doi:10.5555/2517747
- Philippe Marcadent, Janine Berg, Mariya Aleksynska, Valerio De Stefano, Martine Humblet, Christina Behrendt, Susan Hayter, Christopher Land-Kazlauskas, Angelika Muller, Niall O'Higgins, Friederike Eberlein, Simone Marino, Bao Chau Le, and Calum Carson. 2016. NonStandard Employment Around The World.
- Juan Ramos and others. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, 133--142.
- Amazon Mechanical Turk Participation Agreement (2014). Retrieved July 1, 2017 from https://www.mturk.com/mturk/conditionsofuse
Cited by
OECD Employment Outlook 2019: The Future of Work
OECD (2019) · OECD Employment Outlook · doi:10.1787/9ee00155-en · Cited by 821
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2022) · Information Systems Research · doi:10.1287/isre.2021.1079 · Cited by 478
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Rani, U., & Furrer, M. (2021) · Competition & Change · doi:10.1177/1024529420905187 · Cited by 306
Conceptualizing the Gig Economy and Its Regulatory Problems
Koutsimpogiorgos, N., van Slageren, J., Herrmann, A. M., & Frenken, K. (2020) · Policy & Internet · doi:10.1002/poi3.237 · Cited by 191
Metadata
- Authors
- Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P.
- Affiliations
- Carnegie Mellon University (US)
- University of Oxford (GB)
- McMaster University (CA)
- West Virginia University (US)
- University of Pennsylvania (US)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-14
- OpenAlex
- W2777647957
- Cited by
- 497
- References
- 64
- Citations by year
- 2026: 19 · 2025: 38 · 2024: 42 · 2023: 71 · 2022: 99 · 2021: 85 · 2020: 66 · 2019: 63 · 2018: 12
- FWCI
- 66.64
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Economic and Social Research Council (Grant ES/N017099/1)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Digital Economy and Work Transformation
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Labor market dynamics and wage inequality
- Keywords
- crowdwork, digital labour markets, worker income, wage distribution, Amazon Mechanical Turk, task pricing, unpaid work, task rejection, platform design, low-wage employment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. https://app.cochat.ai/papers/10.1145/3173574.3174023
- The paper
- Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018). A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3174023
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.