Financial incentives and the "performance of crowds"
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
The relationship between financial incentives and performance, long of interest to social scientists, has gained new relevance with the advent of web-based "crowd-sourcing" models of production. Here we investigate the effect of compensation on performance in the context of two experiments, conducted on Amazon's Mechanical Turk (AMT). We find that increased financial incentives increase the quantity, but not the quality, of work performed by participants, where the difference appears to be due to an "anchoring" effect: workers who were paid more also perceived the value of their work to be greater, and thus were no more motivated than workers paid less. In contrast with compensation levels, we find the details of the compensation scheme do matter--specifically, a "quota" system results in better work for less pay than an equivalent "piece rate" system. Although counterintuitive, these findings are consistent with previous laboratory studies, and may have real-world analogs as well.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Judgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
Multitask principal – agent analyses: Incentive contracts, asset ownership, and job design
Holmstrom, B., & Milgrom, P. (1991) · The Journal of Law, Economics, and Organization · doi:10.1093/jleo/7.special_issue.24 · Cited by 6,046
The Wealth of Networks: How Social Production Transforms Markets and Freedom
Benkler, Y. (2017) · Yale University Press eBooks · doi:10.12987/9780300127232 · Cited by 3,680
The Provision of Incentives in Firms
Prendergast, C. (1999) · Journal of Economic Literature · doi:10.1257/jel.37.1.7 · Cited by 3,338
Attention, similarity, and the identification-categorization relationship.
Nosofsky, R. M. (1986) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.115.1.39 · Cited by 2,766
Performance Pay and Productivity
Lazear, E. P. (2000) · American Economic Review · doi:10.1257/aer.90.5.1346 · Cited by 2,487
(2008) · Corporate Financial Strategy · doi:10.4324/9780080942841-11 · Cited by 2,390
One More Time: How Do You Motivate Employees?
Herzberg, F. (1976) · Job Satisfaction — A Reader · doi:10.1007/978-1-349-02701-9_2 · Cited by 2,180
Research works printed but not linked: 6
- Benkler Y., 2007, Yale University Press
- 10.2139/ssrn.1381502 · doi:10.2139/ssrn.1381502
- Howe J., 2008, Crown Business
- Ariely D., 2008, Review of Economic Studies Forthcoming
- Conyon M. J., 2000, The Prince and the Pauper?, The Economic Journal, 110, 467
- 10.4159/jla.v1i1.3 · DOI
Cited by
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
“In principle requesters can offer workers wages that are disproportionately low, even considering what the norms are on Mechanical Turk, with little concern since data quality seems to be not affected by payments (Mason & Watts, 2009).”
BackgroundCasler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816
Human computation: a survey and taxonomy of a growing field
Quinn, A. J., & Bederson, B. B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979148 · Cited by 801
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
“Our experience is that paying less slows down the later parts of the process, but it does not impact quality [19] — it would be viable for shortening paragraphs under a loose deadline.”
BackgroundEvaluation on crowdsourcing research: Current status and future direction
Zhao, Y., & Zhu, Q. (2014) · Information Systems Frontiers · doi:10.1007/s10796-012-9350-4 · Cited by 566
A Survey of Crowdsourcing Systems
Yuen, M.-C., King, I., & Leung, K.-S. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.203 · Cited by 392
Quality Control in Crowdsourcing Systems: Issues and Directions
Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013) · IEEE Internet Computing · doi:10.1109/mic.2013.20 · Cited by 385
Digital Footprints: Opportunities and Challenges for Online Social Research
Golder, S. A., & Macy, M. W. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043145 · Cited by 378
Analysis of a consumer survey on plug-in hybrid electric vehicles
Krupa, J. S., Rizzo, D. M., Eppstein, M. J., Brad Lanute, D., Gaalema, D. E., Lakkaraju, K., & Warrender, C. E. (2014) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2014.02.019 · Cited by 371
Collaborative learning in networks
Mason, W., & Watts, D. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1110069108 · Cited by 356
Self-citationCrowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301
TurKit: human computation algorithms on mechanical turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866040 · Cited by 307
An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets
Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., & Vukovic, M. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14105 · Cited by 299
Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems
Karger, D. R., Oh, S., & Shah, D. (2014) · Operations Research · doi:10.1287/opre.2013.1235 · Cited by 297
Chandler, J. J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 277
Litman, L., Robinson, J., & Rosenzweig, C. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275
Cooperation and assortativity with dynamic partner updating
Wang, J., Suri, S., & Watts, D. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1120867109 · Cited by 198
Self-citationLalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192
Reputation Transferability in Online Labor Markets
Kokkodis, M., & Ipeirotis, P. G. (2016) · Management Science · doi:10.1287/mnsc.2015.2217 · Cited by 166
“Mason and Watts (2009) studied the effect of compensation on performance in the context of two experiments conducted on Amazon Mechanical Turk and found that increased financial incentives increase the quantity but not the quality of work performed by participants.”
BackgroundIncentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
Brewer, R., Morris, M. R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858198 · Cited by 111
Mitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102
Understanding Human-Machine Networks: A Cross-Disciplinary Survey
Tsvetkova, M., Yasseri, T., Meyer, E. T., Pickering, J. B., Engen, V., Walland, P., Lüders, M., Følstad, A., & Bravos, G. (2018) · ACM Computing Surveys · doi:10.1145/3039868 · Cited by 67
“Experimental research has shown that higher piece rates increase the quantity but not the quality of contributions; further, quota rates result in higher quality work for a smaller budget than piece rates [Mason and Watts 2010].”
Enhancing reliability using peer consistency evaluation in human computation
Huang, S.-W., & Fu, W.-T. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441847 · Cited by 50
“These are also the three financial incentive levels used in [21].”
BackgroundOn the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4
Metadata
- Authors
- Mason, W., & Watts, D. J.
- Venue
- ACM SIGKDD Explorations Newsletter
- Published
- 2010-05-27
- Volume
- 11
- Issue
- 2
- Pages
- 100-108
- ISSN
- 1931-0145 (ISSN-L)
- 1931-0153
- OpenAlex
- W2735460460
- Cited by
- 463
- References
- 32
- FWCI
- 88.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.