Who are the crowdworkers?: shifting demographics in mechanical turk
Abstract
Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous studies of the demographics and usage behaviors of MTurk workers. We describe how the worker population has changed over time, shifting from a primarily moderate-income, U.S.-based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers. This change in population points to how workers may treat Turking as a full-time job, which they rely on to make ends meet.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Compared to the U.S. population, the study sample is younger and more educated, a common characteristic of samples recruited through online crowdsourcing platforms.
“Compared to the U.S. population, our sample is younger and more educated, a common characteristic of samples recruited through online crowdsourcing platforms [43, 45, 81, 88].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [88]
Cited by
Sorted by citations
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · see citing works
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · see citing works
MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391 · see citing works
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317 · see citing works
Rani, U., & Furrer, M. (2021) · Competition & Change · doi:10.1177/1024529420905187 · Cited by 306 · see citing works
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207 · see citing works
Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective
Ball, C., Huang, K.-T., & Francis, J. (2021) · Telematics and Informatics · doi:10.1016/j.tele.2021.101728 · Cited by 152 · see citing works
“Depending on the complexity of the task, Turkers earn anywhere from 1 cent to a couple dollars for each HIT completed (Ross et al., 2010).”
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125 · see citing works
Firkey, M. K., Sheinfil, A. Z., & Woolf-King, S. E. (2022) · Journal of American College Health · doi:10.1080/07448481.2020.1869750 · Cited by 122 · see citing works
Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125 · see citing works
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
Brewer, R., Morris, M. R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858198 · Cited by 111 · see citing works
Whose AI Dream? In search of the aspiration in data annotation.
Wang, D., Prabhat, S., & Sambasivan, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502121 · Cited by 70 · see citing works
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69 · see citing works
Utility of human-computer interactions: toward a science of preference measurement
Toomim, M., Kriplean, T., Pörtner, C., & Landay, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979277 · Cited by 64 · see citing works
Briones, E. M., & Benham, G. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0710-8 · Cited by 48 · see citing works
AppWatcher: unveiling the underground market of trading mobile app reviews
Xie, Z., & Zhu, S. (2015) · Proceedings of the 8th ACM Conference on Security & Privacy in Wireless and Mobile Networks · doi:10.1145/2766498.2766510 · Cited by 38 · see citing works
Who Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
Marshall, C. C., Goguladinne, P. S. R., Maheshwari, M., Sathe, A., & Shipman, F. M. (2023) · Proceedings of the 15th ACM Web Science Conference 2023 · doi:10.1145/3578503.3583622 · Cited by 20 · see citing works
Neshaei, S. P., Rietsche, R., Su, X., & Wambsganss, T. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645169 · Cited by 9 · see citing works
- Authors
- Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B.
- Affiliations
- University of California, Irvine (US)
- Venue
- CHI '10 Extended Abstracts on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2010-04-10
- Pages
- 2863-2872
- OpenAlex
- W2114269021
- Cited by
- 1,004
- References
- 0
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 26 · 2023: 30 · 2022: 46 · 2021: 67 · 2020: 64 · 2019: 67 · 2018: 79 · 2017: 95 · 2016: 90 · 2015: 114 · 2014: 105 · 2013: 85 · 2012: 62
- FWCI
- 59.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- Sound Ideas (University of Puget Sound) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Digital Economy and Work Transformation
- Health Sciences › Health Professions › General Health Professions › Employment and Welfare Studies
- Keywords
- Amazon Mechanical Turk, crowdsourcing, crowd workers, worker demographics, foreign workers, Indian employees, worker income, part-time work, usage behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
