The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Online Task Assignment in Crowdsourcing Markets
Open accessAbstract
We explore the problem of assigning heterogeneous tasks to workers with different, unknown skill sets in crowdsourcing markets such as Amazon Mechanical Turk. We first formalize the online task assignment problem, in which a requester has a fixed set of tasks and a budget that specifies how many times he would like each task completed. Workers arrive one at a time (with the same worker potentially arriving multiple times), and must be assigned to a task upon arrival. The goal is to allocate workers to tasks in a way that maximizes the total benefit that the requester obtains from the completed work. Inspired by recent research on the online adwords problem, we present a two-phase exploration-exploitation assignment algorithm and prove that it is competitive with respect to the optimal offline algorithm which has access to the unknown skill levels of each worker. We empirically evaluate this algorithm using data collected on Mechanical Turk and show that it performs better than random assignment or greedy algorithms. To our knowledge, this is the first work to extend the online primal-dual technique used in the online adwords problem to a scenario with unknown parameters, and the first to offer an empirical validation of an online primal-dual algorithm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 2×
“…to inexpensively obtain distributed labor and information, and have recently become popular among researchers who use sites like Mechanical Turk to conduct user studies (Kittur, Chi, and Suh 2008), run behavioral experiments (Mason and Suri 2012), and collect data (Horton and Chilton 2010).”
Background(2002) · doi:10.1137/s0097539701398375 · Cited by 2,275
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
An Introduction to Computational Learning Theory
Kearns, M. J., & Vazirani, U. (1994) · doi:10.7551/mitpress/3897.001.0001 · Cited by 1,697
The Hungarian method for the assignment problem
Kuhn, H. W. (2005) · Naval Research Logistics (NRL) · doi:10.1002/nav.20053 · Cited by 1,260
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · Proceedings of the 11th ACM conference on Electronic commerce · doi:10.1145/1807342.1807376 · Cited by 601
Iterative Learning for Reliable Crowdsourcing Systems
Karger, D. R., Oh, S., & Shah, D. (2011) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 468
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
Li, L., Chu, W., Langford, J., & Wang, X. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935878 · Cited by 443
Platemate: crowdsourcing nutritional analysis from food photographs
Noronha, J., Hysen, E., Zhang, H., & Gajos, K. Z. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047198 · Cited by 247
Epsilon–First Policies for Budget–Limited Multi-Armed Bandits
Tran-Thanh, L., Chapman, A. C., de Cote, E. M., Rogers, A., & Jennings, N. R. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7758 · Cited by 216
Near optimal online algorithms and fast approximation algorithms for resource allocation problems
Devanur, N. R., Jain, K., Sivan, B., & Wilkens, C. A. (2011) · Proceedings of the 12th ACM conference on Electronic commerce · doi:10.1145/1993574.1993581 · Cited by 160
Langford, J. C., Strehl, A. L., & Wortman, J. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390223 · Cited by 112
Haitner, I., Hoch, J. J., Reingold, O., & Segev, G. (2007) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2007.7 · Cited by 96
Approximation algorithms for budgeted learning problems
Guha, S., & Munagala, K. (2007) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1250790.1250807 · Cited by 86
ε-first policies for budget-limited multi-armed bandits
Han, T. A., Chapman, A. C., de Cote, E. M., Rogers, A., & Jennings, N. R. (2010) · Spiral (Imperial College London) · Cited by 83
A Primal-Dual Randomized Algorithm for Weighted Paging
Bansal, N., Buchbinder, N., & Naor, J. S. (2012) · Journal of the ACM · doi:10.1145/2339123.2339126 · Cited by 72
Law, E., & Zhang, H. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.8092 · Cited by 30
Cited by
Mobile Crowd Sensing and Computing: The Review of an Emerging Human-Powered Sensing Paradigm
Guo, B., Wang, Z., Yu, Z., Wang, Y., Yen, N. Y., Huang, R., & Zhou, X. (2015) · ACM Computing Surveys · doi:10.1145/2794400 · Cited by 719
Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2019) · ACM Computing Surveys · doi:10.1145/3148148 · Cited by 313
Crowdsourced Data Management: A Survey
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2535242 · Cited by 283
Incentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146
Self-citation
Metadata
- Authors
- Ho, C.-J., & Vaughan, J.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-09-20
- Volume
- 26
- Issue
- 1
- Pages
- 45-51
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2398690976
- Cited by
- 293
- References
- 24
- FWCI
- 14.53
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.