A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Abstract
A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“User studies for machine learning research often depend on online crowd platforms such as Amazon Mechanical Turk, which can lead to ethical issues.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [91]
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1×
“nt clustering and (ii) manual retrieval of topical phrases from the clustered documents. We use K-Means algorithm to cluster the transformed documents that are close to each other in the latent space [51]. K-Means clustering requires a user to specify a distance function and a number of clusters a priori. We use the c…”
Method(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Motivation through the design of work: test of a theory
Hackman, J. R., & Oldham, G. R. (1976) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(76)90016-7 · Cited by 8,962 · cited 1×
“Further, low wage rates and the ethical concerns of workers should be of importance to requesters given the association between poor working conditions, low quality, and high turnover [12,26,42].”
BackgroundScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 1×
“nt clustering and (ii) manual retrieval of topical phrases from the clustered documents. We use K-Means algorithm to cluster the transformed documents that are close to each other in the latent space [51]. K-Means clustering requires a user to specify a distance function and a number of clusters a priori. We use the c…”
MethodSoftware Framework for Topic Modelling with Large Corpora
Rek, R. R. U., & Sojka, P. (2010) · doi:10.13140/2.1.2393.1847 · Cited by 3,647
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
An Introduction to Statistical Learning: with Applications in R
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021) · Springer Texts in Statistics · doi:10.1007/978-1-0716-1418-1 · Cited by 1,755
Layers of Silence, Arenas of Voice: The Ecology of Visible and Invisible Work
Star, S. L., & Strauss, A. (1999) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1008651105359 · Cited by 1,500 · cited 1×
“identify potential strategies for workers to increase their earnings. Unpaid Work It is not always obvious what counts as work on crowdsourcing platforms. Working on AMT often involves invisible work [62]—time spent on work that is directly/indirectly related to completing HITs yet unpaid. There are several types of t…”
BackgroundUnderstanding Self-Report Bias in Organizational Behavior Research
Donaldson, S. I., & Grant-Vallone, E. J. (2002) · Journal of Business and Psychology · doi:10.1023/a:1019637632584 · Cited by 1,068 · cited 1×
“research into crowd work earnings has been limited by an absence of adequate quantitative data. Prior research based on self-reported income data (e.g., [4,34,49]) might be subject to systemic biases [22] and is often not sufficiently granular to facilitate a detailed investigation of earnings dispersion. Existing dat…”
BackgroundKittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · cited 2×
“Its remote and asynchronous work style, unbounded by time and location, is considered to extend the modern office work [42,44,45], enabling people with disabilities, at-home parents, and”
(2015) · Routledge eBooks · doi:10.4324/9781315865737 · Cited by 899
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879 · cited 3×
“, finding good pricing methods [34,48,59], suggesting effective task design for requesters [24,38]) or it focuses on characterizing the crowdsourcing market dynamics [21,35].”
BackgroundMethodHuman computation: a survey and taxonomy of a growing field
Quinn, A. J., & Bederson, B. B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979148 · Cited by 801
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798
Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of CrowdworkersPreprint
Berg, J. (2016) · RePEc: Research Papers in Economics · Cited by 769
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
Employment and adults with autism spectrum disorders: Challenges and strategies for success
Hendricks, D. (2010) · Journal of Vocational Rehabilitation · doi:10.3233/jvr-2010-0502 · Cited by 610 · cited 1×
“ment [4,65]. Hara and Bigham noted that some crowd work like image transcription can be done by autistic people—a population that has challenge in securing jobs compared to those without disabilities [10,30,33]—but this type of work is exactly what generates the lowest hourly wages as we showed in the topical analysis…”
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601
Self-citationThe labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · Proceedings of the 11th ACM conference on Electronic commerce · doi:10.1145/1807342.1807376 · Cited by 601 · cited 1×
“minimum wage [32,33,35,36,37,47] and many workers report not being paid for adequately completed tasks [36,49].”
BackgroundMore than fun and money. Worker motivation in crowdsourcing - a study on mechanical turk
Kaufmann, N., Schulze, T., & Veit, D. J. (2011) · MADOC (University of Mannheim) · Cited by 576
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Self-citationFinancial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600175 · Cited by 520
Latent Semantic Analysis: five methodological recommendations
Evangelopoulos, N., Zhang, X., & Prybutok, V. R. (2012) · European Journal of Information Systems · doi:10.1057/ejis.2010.61 · Cited by 306 · cited 3×
“4 is considered as a good choice [23], which we follow.”
BackgroundMethodInfluentialCreating Speech and Language Data With Amazon’s Mechanical Turk
Callison-Burch, C., & Dredze, M. H. (2010) · Cited by 294
Self-citationMartin, D., Hanrahan, B. V., O'Neill, J., & Gupta, N. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531663 · Cited by 278 · cited 8×
“This is problematic as income generation is the primary motivation of workers [4,13,44,47].”
BackgroundInfluentialThe Global Opportunity in Online Outsourcing
Kuek, S. C., Paradi-Guilford, C., Fayomi, T., Imaizumi, S., Ipeirotis, P., Pina, P., & Singh, M. (2015) · World Bank, Washington, DC eBooks · doi:10.1596/22284 · Cited by 277 · cited 5×
“r Keywords Crowdsourcing; Amazon Mechanical Turk; Hourly wage ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION Crowd work is growing [29,44]. A report by Harris and Krueger states that 600k workers participate in the online gig economy and the number i…”
BackgroundInfluentialWe Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers
Salehi, N., Irani, L. C., Bernstein, M. S., Alkhatib, A., Ogbe, E., Milland, K., & Clickhappier (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702508 · Cited by 274
Self-citationTemporal Event Sequence Simplification
Monroe, M., Lan, R., Lee, H., Plaisant, C., & Shneiderman, B. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.200 · Cited by 266 · cited 1×
“[50] that groups a series of temporally close time intervals into clusters using an interval threshold, D.”
MethodInterpretation and trust: designing model-driven visualizations for text analysis
Chuang, J., Ramage, D., Manning, C., & Heer, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207738 · Cited by 244
Studying the history of ideas using topic models
Hall, D., Jurafsky, D., & Manning, C. D. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613763 · Cited by 191 · cited 1×
“to a labeling process mediated by computation known as topic modeling. Topic modeling methods are often used to categorize a set of unlabeled documents into a finite number of coherent topic clusters [7,28,63], in turn helping researchers to navigate a large corpus (HIT documents in our case) [17,18,24,54]. However, u…”
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188 · cited 1×
“Further, low wage rates and the ethical concerns of workers should be of importance to requesters given the association between poor working conditions, low quality, and high turnover [12,26,42].”
BackgroundThe Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk
Difallah, D. E., Catasta, M., Demartini, G., Ipeirotis, P. G., & Cudré-Mauroux, P. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741685 · Cited by 187
Stories We Tell About Labor: Turkopticon and the Trouble with "Design"
Irani, L. C., & Silberman, M. S. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858592 · Cited by 174 · cited 2×
“rk style, unbounded by time and location, is considered to extend the modern office work [44,46,47], enabling people with disabilities, at-home parents, and temporarily out -of work engineers to work [1,4,39,46,65]. Yet, despite the potential for crowdsourcing platforms to extend the scope of the labor market, many ar…”
BackgroundTopic Model Diagnostics: Assessing Domain Relevance via Topical Alignment
Chuang, J., Gupta, S., Manning, C. D., & Heer, J. (2013) · International Conference on Machine Learning · Cited by 156
Examining Crowd Work and Gig Work Through The Historical Lens of Piecework
Alkhatib, A., Bernstein, M. S., & Levi, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025974 · Cited by 148 · cited 3×
“primary goal of workers is income generation [4,13,44,47], rather than having fun or making pocket money [2,4,41].”
BackgroundEthics and tactics of professional crowdwork
Silberman, M. S., Irani, L., & Ross, J. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869100 · Cited by 145 · cited 1×
“The use of asymmetric rating systems, for example, makes it difficult for workers to learn about unfair requesters [2,36,58], while platforms rarely offer protection against wage theft or provide mechanisms for workers to dispute task rejections and poor ratings [4,45].”
BackgroundResearch in the crowdsourcing age: a case study
Hitlin, P. (2016) · Cited by 141
An empirical study of required dimensionality for large-scale latent semantic indexing applications
Bradford, R. B. (2008) · Proceedings of the 17th ACM conference on Information and knowledge management · doi:10.1145/1458082.1458105 · Cited by 140 · cited 1×
“We follow Bradford recommendation and use a clusters size kcluster=200 [9].”
MethodBohannon, J. (2011) · Science · doi:10.1126/science.334.6054.307 · Cited by 130 · cited 1×
“, super turkers [8] who are already experienced may not be interested in using the plugin).”
BackgroundWhat's the right price? pricing tasks for finishing on time
Faridani, S., Hartmann, B., & Ipeirotis, P. G. (2011) · Cited by 126
Valuing Alternative Work Arrangements
Mas, A., Welch, K., & Charpentier, D. (2016) · AEA Randomized Controlled Trials · doi:10.1257/rct.1250 · Cited by 120
McInnis, B., Cosley, D., Nam, C., & Leshed, G. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858539 · Cited by 119
Zyskowski, K., Morris, M. R., Bigham, J. P., Gray, M. L., & Kane, S. K. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675158 · Cited by 116
Self-citationA taxonomy of microtasks on the web
Gadiraju, U., Kawase, R., & Dietze, S. (2014) · Proceedings of the 25th ACM conference on Hypertext and social media · doi:10.1145/2631775.2631819 · Cited by 116 · cited 1×
“[25] (with an additional category Research).”
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
Brewer, R., Morris, M. R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858198 · Cited by 111
Web workers unite! addressing challenges of online laborers
Bederson, B. B., & Quinn, A. J. (2011) · CHI '11 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1979742.1979606 · Cited by 106 · cited 1×
“hourly wage is not straightforward [3], we showed that different wage computation methods do not result in largely different hourly wage estimates.”
BackgroundDatabase of NIH grants using machine-learned categories and graphical clustering
Talley, E. M., Newman, D., Mimno, D., Herr, B. W., Wallach, H. M., Burns, G. A. P. C., Leenders, A. G. M., & McCallum, A. (2011) · Nature Methods · doi:10.1038/nmeth.1619 · Cited by 85 · cited 1×
“methods are often used to categorize a set of unlabeled documents into a finite number of coherent topic clusters [7,27,61], in turn helping researchers to navigate a large corpus (HIT documents in our case) [17,18,24,52].”
MethodGupta, N., Martin, D., Hanrahan, B. V., & O'Neill, J. (2014) · Proceedings of the 18th International Conference on Supporting Group Work · doi:10.1145/2660398.2660403 · Cited by 82 · cited 1×
“There are several types of this invisible work, including the time spent on the returned HITs, work done for the rejected HITs, and, again, time spent searching for HITs [26,49,52].”
BackgroundCrowdsourced Data Management: Industry and Academic Perspectives
Marcus, A., & Parameswaran, A. (2015) · Foundations and Trends in Databases · doi:10.1561/1900000044 · Cited by 76 · cited 1×
“that is widely used by industry [32,46] and the HCI community, as well as by other research areas such as NLP and computer vision [15,43].”
BackgroundTopicCheck: Interactive Alignment for Assessing Topic Model Stability
Chuang, J., Roberts, M. E., Stewart, B., Weiss, R., Tingley, D. H., Grimmer, J., & Heer, J. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1018 · Cited by 75 · cited 1×
“are often used to categorize a set of unlabeled documents into a finite number of coherent topic clusters [7,28,63], in turn helping researchers to navigate a large corpus (HIT documents in our case) [17,18,24,54]. However, unsupervised clustering of unlabeled documents is far from a solved problem [16,18]. Although t…”
BackgroundWhen Home Affects Pay: An Analysis of the Gender Pay Gap Among CrowdworkersPreprint
Adams‐Prassl, A., & Berg, J. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3048711 · Cited by 75 · cited 2×
“temporarily out-of-work engineers to work [1,4,37,44,63].”
BackgroundCrowdsourcing in Computer Vision
Kovashka, A., Russakovsky, O., Fei-Fei, L., & Grauman, K. (2016) · now publishers, Inc. eBooks · doi:10.1561/9781680832136 · Cited by 72
Crowdsourcing in Computer Vision
Kovashka, A., Russakovsky, O., Fei-Fei, L., & Grauman, K. (2016) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000071 · Cited by 65
Crowdsourced Data Management: Industry and Academic Perspectives
Marcus, A., & Parameswaran, A. (2015) · now publishers, Inc. eBooks · doi:10.1561/9781680830910 · Cited by 60
Jain, A., Sarma, A. D., Parameswaran, A., & Widom, J. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3067421.3067431 · Cited by 57
Crowd-Workers: Aggregating Information Across Turkers to Help Them Find Higher Paying Work
Callison-Burch, C. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13198 · Cited by 42 · cited 3×
“describe the tool that we used to collect task-level data on worker behavior and present basic statistics of the data. Crowd Worker Plugin The data was collected using the Crowd Workers Chrome plugin [14]. The plugin was used by workers in an opt-in basis. The plugin was designed to disclose the effective hourly wage…”
MethodInfluentialSelf-citationTurkBench: Rendering the Market for Turkers
Hanrahan, B. V., Willamowski, J. K., Swaminathan, S., & Martin, D. B. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702279 · Cited by 41
Ikeda, K., & Bernstein, M. S. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858327 · Cited by 38
Pricing tasks in online labor markets
Singer, Y., & Mittal, M. (2011) · Cited by 34
ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling
Poursabzi-Sangdeh, F., Boyd-Graber, J. L., Findlater, L., & Seppi, K. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1110 · Cited by 34 · cited 1×
“are often used to categorize a set of unlabeled documents into a finite number of coherent topic clusters [7,28,63], in turn helping researchers to navigate a large corpus (HIT documents in our case) [17,18,24,54]. However, unsupervised clustering of unlabeled documents is far from a solved problem [16,18]. Although t…”
BackgroundIrani, L. C. (2017) · Wiley eBooks · doi:10.1002/9781405165518.wbeos0994 · Cited by 32
Modernizing labor laws for twenty-first-century work: the "independent worker"
Harris, S. D., & Krueger, A. B. (2015) · Cited by 31
Introducing People with ASD to Crowd Work
Hara, K., & Bigham, J. P. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132544 · Cited by 30 · cited 1×
“Hara and Bigham noted that some crowd work like image transcription can be done by autistic people—a population that has challenge in securing jobs compared to those without disabilities [10,28,31]—but this type of work is exactly what generates the lowest hourly wages as we showed in the topical analysis.”
ResultSelf-citation
Research works printed but not linked: 9
- 10.1016/j.chb.2015.08.031 · DOI · Paper page
- 10.5555/1866696.1866697 · doi:10.5555/1866696.1866697
- 10.5555/2908698.2908703 · doi:10.5555/2908698.2908703
- 10.5555/1613715.1613763 · doi:10.5555/1613715.1613763
- 10.5555/540298 · doi:10.5555/540298
- 10.5555/2517747 · doi:10.5555/2517747
- Philippe Marcadent, Janine Berg, Mariya Aleksynska, Valerio De Stefano, Martine Humblet, Christina Behrendt, Susan Hayter, Christopher Land-Kazlauskas, Angelika Muller, Niall O'Higgins, Friederike Eberlein, Simone Marino, Bao Chau Le, and Calum Carson. 2016. NonStandard Employment Around The World.
- Juan Ramos and others. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, 133--142.
- Amazon Mechanical Turk Participation Agreement (2014). Retrieved July 1, 2017 from https://www.mturk.com/mturk/conditionsofuse
Cited by
OECD Employment Outlook 2019: The Future of Work
OECD (2019) · OECD Employment Outlook · doi:10.1787/9ee00155-en · Cited by 821
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2022) · Information Systems Research · doi:10.1287/isre.2021.1079 · Cited by 478
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
“Critics argue that these platforms are largely unregulated and that workers are poorly compensated [91, 224].”
BackgroundRani, U., & Furrer, M. (2021) · Competition & Change · doi:10.1177/1024529420905187 · Cited by 306
Conceptualizing the Gig Economy and Its Regulatory Problems
Koutsimpogiorgos, N., van Slageren, J., Herrmann, A. M., & Frenken, K. (2020) · Policy & Internet · doi:10.1002/poi3.237 · Cited by 191
Metadata
- Authors
- Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P.
- Affiliations
- Carnegie Mellon University (US)
- University of Oxford (GB)
- McMaster University (CA)
- West Virginia University (US)
- University of Pennsylvania (US)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-14
- OpenAlex
- W2777647957
- Cited by
- 497
- References
- 64
- Citations by year
- 2026: 19 · 2025: 38 · 2024: 42 · 2023: 71 · 2022: 99 · 2021: 85 · 2020: 66 · 2019: 63 · 2018: 12
- FWCI
- 66.64
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Economic and Social Research Council (Grant ES/N017099/1)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Digital Economy and Work Transformation
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Labor market dynamics and wage inequality
- Keywords
- crowdwork, digital labour markets, worker income, wage distribution, Amazon Mechanical Turk, task pricing, unpaid work, task rejection, platform design, low-wage employment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.