Quantifying the Invisible Labor in Crowd Work
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was $3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to $2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making
Jarrahi, M. H. (2018) · Business Horizons · doi:10.1016/j.bushor.2018.03.007 · Cited by 2,772
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034
The sharing economy and digital platforms: A review and research agenda
Sutherland, W., & Jarrahi, M. H. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.07.004 · Cited by 725
Uberland: How Algorithms Are Rewriting the Rules of Work
Rosenblat, A. (2018) · doi:10.1525/9780520970632 · Cited by 694
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174023 · Cited by 497
Self-citationExamining Crowd Work and Gig Work Through The Historical Lens of Piecework
Alkhatib, A., Bernstein, M. S., & Levi, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025974 · Cited by 148 · cited 1×
“From the workers’ perspective, these markets provide a central place to find work and offer them the flexibility of working from wherever they desire [1, 87].”
BackgroundHuman-AI Collaboration with Bandit Feedback
Gao, R., Saar-Tsechansky, M., De-Arteaga, M., Han, L., Lee, M. K., & Lease, M. (2021) · arXiv · doi:10.48550/arxiv.2105.10614 · Cited by 23
Cited by
Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
Privacy, Surveillance, and Power in the Gig Economy
Sannon, S., Sun, B., & Cosley, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502083 · Cited by 61
“Our fndings show that many self-protective surveillance strategies impose costs around time, efort, wages, customer perception, and even safety, which adds to the invisible labor already rife on these platforms [64, 70].”
BackgroundThe Future of HCI-Policy Collaboration
Yang, Q., Wong, R. Y., Jackson, S., Junginger, S., Hagan, M. D., Gilbert, T., & Zimmerman, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642771 · Cited by 52
Spektor, F., Fox, S. E., Awumey, E., Begleiter, B., Kulkarni, C., Stringam, B., Riordan, C. A., Rho, H. J., Akridge, H., & Forlizzi, J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579466 · Cited by 29
Digital Labor, Platforms, and AI
Munn, L. (2024) · Introduction to Digital Humanism · doi:10.1007/978-3-031-45304-5_35 · Cited by 17
The Making of Performative Accuracy in AI Training: Precision Labor and Its Consequences
Zhang, B. Z., Yang, T., Miceli, M., Haimson, O. L., & Thomas, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713112 · Cited by 7
“This echoes an increasing body of literature on the hidden and often unrecognized and underpaid labor in data production [16, 29, 35, 54, 69, 77, 86].”
Labor, Capital, and Machine: Toward a Labor Process Theory for HCI
Qin, Y., & Cheon, E. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791147 · Cited by 2
Metadata
- Authors
- Toxtli, C., Suri, S., & Savage, S.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3203959757
- Cited by
- 92
- References
- 7
- FWCI
- 11.89
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.