The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
A Survey of Crowdsourcing Systems
Abstract
Crowd sourcing is evolving as a distributed problem-solving and business production model in recent years. In crowd sourcing paradigm, tasks are distributed to networked people to complete such that a company's production cost can be greatly reduced. In 2003, Luis von Ahn and his colleagues pioneered the concept of "human computation", which utilizes human abilities to perform computation tasks that are difficult for computers to process. Later, the term "crowdsourcing" was coined by Jeff Howe in 2006. Since then, a lot of work in crowd sourcing has focused on different aspects of crowd sourcing, such as computational techniques and performance analysis. In this paper, we give a survey on the literature on crowd sourcing which are categorized according to their applications, algorithms, performances and datasets. This paper provides a structured view of the research on crowd sourcing to date.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204
Crowdsourcing as a Model for Problem Solving: An Introduction and Cases
Brabham, D. C. (2008) · Convergence: The International Journal of Research into New Media Technologies · doi:10.1177/1354856507084420 · Cited by 1,943
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
(2008) · Cited by 1,657
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973
von Ahn, L. (2006) · Computer · doi:10.1109/mc.2006.196 · Cited by 955
Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business
(2009) · Journal of Consumer Marketing · doi:10.1108/07363760910965918 · Cited by 882
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879
The Structure of Collaborative Tagging Systems
Golder, S. A., & Huberman, B. A. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0508082 · Cited by 828
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816
Human computation: a survey and taxonomy of a growing field
Quinn, A. J., & Bederson, B. B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979148 · Cited by 801
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 761
Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition
Leimeister, J. M., Huber, M., Bretschneider, U., & Krcmar, H. (2009) · Journal of Management Information Systems · doi:10.2753/mis0742-1222260108 · Cited by 745
Harnessing the wisdom of crowds in wikipedia: quality through coordination
Kittur, A., & Kraut, R. E. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460572 · Cited by 680
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · Proceedings of the 11th ACM conference on Electronic commerce · doi:10.1145/1807342.1807376 · Cited by 601
Collecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · Cited by 541
Peekaboom: a game for locating objects in images
von Ahn, L., Liu, R., & Blum, M. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124782 · Cited by 509
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463
CrowdForge: crowdsourcing complex work
Kittur, A., Smus, B., & Kraut, R. (2011) · CHI '11 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1979742.1979902 · Cited by 434
Are your participants gaming the system?: screening mechanical turk workers
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 418
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Nowak, S., & Rüger, S. (2010) · Proceedings of the international conference on Multimedia information retrieval · doi:10.1145/1743384.1743478 · Cited by 313
Verbosity: a game for collecting common-sense facts
von Ahn, L., Kedia, M., & Blum, M. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124784 · Cited by 309
TurKit: human computation algorithms on mechanical turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866040 · Cited by 307
Creating Speech and Language Data With Amazon’s Mechanical Turk
Callison-Burch, C., & Dredze, M. H. (2010) · Cited by 294
Crowdsourcing and all-pay auctions
DiPalantino, D., & Vojnovic, M. (2009) · Proceedings of the 10th ACM conference on Electronic commerce · doi:10.1145/1566374.1566392 · Cited by 292
Annotating named entities in Twitter data with crowdsourcing
Finin, T., Murnane, W., Karandikar, A., Keller, N., Martineau, J., & Dredze, M. H. (2010) · doi:10.13016/m21j97c2p · Cited by 276
Analysis of participation in an online photo-sharing community: a multidimensional perspective
Nov, O., Naaman, M., & Ye, C. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21278 · Cited by 263
Crowdsourcing, attention and productivity
Huberman, B. A., Romero, D. M., & Wu, F. (2009) · Journal of Information Science · doi:10.1177/0165551509346786 · Cited by 261
CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones
Yan, T., Kumar, V., & Ganesan, D. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814443 · Cited by 255
Data quality from crowdsourcing: a study of annotation selection criteria
Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009) · Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing - HLT '09 · doi:10.3115/1564131.1564137 · Cited by 251
Law, E., & von Ahn, L. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518881 · Cited by 248
Questions in, knowledge in?: a study of naver's question answering community
Nam, K. K., Ackerman, M. S., & Adamic, L. A. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518821 · Cited by 225
Crowdsourcing and knowledge sharing: strategic user behavior on taskcn
Yang, J., Adamic, L. A., & Ackerman, M. S. (2008) · Proceedings of the 9th ACM conference on Electronic commerce · doi:10.1145/1386790.1386829 · Cited by 190
Archak, N. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772694 · Cited by 187
Citizen Noise Pollution Monitoring
Maisonneuve, N., Stevens, M., Niessen, M. E., Hanappe, P., & Steels, L. (2009) · Cited by 161
Improving accessibility of the web with a computer game
von Ahn, L., Ginosar, S., Kedia, M., Liu, R., & Blum, M. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124785 · Cited by 156
Ethics and tactics of professional crowdwork
Silberman, M. ‘., Irani, L. C., & Ross, J. (2010) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1869086.1869100 · Cited by 144
A Web-Based Game for Collecting Music Metadata
Mandel, M., & Ellis, D. P. W. (2008) · Journal of New Music Research · doi:10.1080/09298210802479300 · Cited by 129
Crowdsourcing Document Relevance Assessment with Mechanical Turk
Grady, C. P. L., & Lease, M. (2010) · Cited by 123
A Survey of Human Computation Systems
Yuen, M.-C., Chen, L., & King, I. (2009) · doi:10.1109/cse.2009.395 · Cited by 99
Self-citationCrowdsourcing service-level network event monitoring
Choffnes, D., Bustamante, F. E., & Ge, Z. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851228 · Cited by 98
A Game-Based Approach For Collecting Semantic Annotations Of Music.
Turnbull, D., Liu, R., Barrington, L., & Lanckriet, G. R. G. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1416463 · Cited by 98
The effect of assessor error on IR system evaluation
Carterette, B., & Soboroff, I. M. (2010) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1835449.1835540 · Cited by 95
Casey, S. T., Kirman, B. J., & Rowland, D. (2007) · International Conference on Advances in Computer Entertainment Technology (ACE) · doi:10.1145/1255047.1255050 · Cited by 82
You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks
Harris, C. G. (2011) · Cited by 77
Crowdsourcing participation inequality: a SCOUT model for the enterprise domain
Stewart, O. T., Lubensky, D. M., & Huerta, J. M. (2010) · doi:10.1145/1837885.1837895 · Cited by 76
Personalization via friendsourcing
Bernstein, M. S., Tan, D. S., Smith, G., Czerwinski, M. P., & Horvitz, E. (2008) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1746259.1746260 · Cited by 74
Huang, E., Zhang, H., Parkes, D. C., Gajos, K. Z., & Chen, Y. (2010) · doi:10.1145/1837885.1837908 · Cited by 72
Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation
Akkaya, C., Conrad, A. P., Wiebe, J., & Mihalcea, R. F. (2010) · University of North Texas Digital Library (University of North Texas) · Cited by 68
How Crowdsourcable is Your Task
Eickhoff, C., & de Vries, A. P. (2011) · Data Archiving and Networked Services (DANS) · Cited by 67
How power users help and hinder open bug reporting
Ko, A. J., & Chilana, P. K. (2010) · International Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753576 · Cited by 67
Feedback Loops of Attention in Peer Production
Wu, F., Wilkinson, D. M., & Huberman, B. A. (2009) · doi:10.1109/cse.2009.430 · Cited by 57
Bernstein, M. S., Tan, D. S., Smith, G., Czerwinski, M. P., & Horvitz, E. (2009) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1622176.1622195 · Cited by 56
MobiMissions: the game of missions for mobile phones
Grant, L., Daanen, H., Benford, S., Hampshire, A., Drozd, A., & Greenhalgh, C. (2007) · ACM SIGGRAPH Educators Program · doi:10.1145/1282040.1282053 · Cited by 53
Improving Image Search with PHETCH
von Ahn, L., Ginosar, S., Kedia, M., & Blum, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367293 · Cited by 52
Annotating Large Email Datasets for Named Entity Recognition with
Lawson, N., Eustice, K., Perkowitz, M., & Yetisgen-Yildiz, M. (2010) · Cited by 51
Motivating contributors in social media networks
Singh, V. K., Jain, R., & Kankanhalli, M. (2009) · doi:10.1145/1631144.1631149 · Cited by 47
Estimating the Completion Time of Crowdsourced Tasks Using Survival Analysis Models
Wang, J., Faridani, S., & Ipeirotis, P. G. (2011) · Cited by 40
Crowdsourcing service-level network event monitoring
Choffnes, D., Bustamante, F. E., & Ge, Z. (2010) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1851275.1851228 · Cited by 38
The role of game theory in human computation systems
Jain, S., & Parkes, D. C. (2009) · doi:10.1145/1600150.1600171 · Cited by 36
Preliminary Experiments with Amazon's Mechanical Turk for Annotating Medical Named Entities
Yetisgen-Yildiz, M., Solti, I., Xia, F., & Halgrim, S. R. (2010) · Cited by 34
Opinion Mining of Spanish Customer Comments with Non-Expert Annotations on Mechanical Turk
Mellebeek, B., Benavent, F., Grivolla, J., Codina, J., Costa‐jussà, M. R., & Banchs, R. E. (2010) · Cited by 33
A scalable, collaborative similarity measure for social annotation systems
Markines, B., & Menczer, F. (2009) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/1557914.1557982 · Cited by 27
Corpus Creation for New Genres: A Crowdsourced Approach to PP Attachment
Jha, M., Andreas, J., Thadani, K., Rosenthal, S., & McKeown, K. R. (2010) · doi:10.7916/d8ff41pf · Cited by 26
Clustering dictionary definitions using Amazon Mechanical Turk
Parent, G., & Eskénazi, M. (2010) · Cited by 23
Mathematical Modeling of Social Games
Chan, K. T., King, I., & Yuen, M.-C. (2009) · doi:10.1109/cse.2009.166 · Cited by 22
Self-citationPredicting Cloze Task Quality for Vocabulary Training
Skory, A., & Eskénazi, M. (2010) · Cited by 19
MTurk Crowdsourcing: A Viable Method for Rapid Discovery of Arabic Nicknames?
Higgins, C., McGrath, E. J., & Moretto, L. (2010) · Cited by 13
Weng, L., & Menczer, F. (2010) · doi:10.1145/1837885.1837894 · Cited by 9
PackPlay: Mining Semantic Data in Collaborative Games
Green, N., Breimyer, P., Kumar, V., & Samatova, N. F. (2010) · Cited by 9
Cited by
Classification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Mobile Crowd Sensing and Computing: The Review of an Emerging Human-Powered Sensing Paradigm
Guo, B., Wang, Z., Yu, Z., Wang, Y., Yen, N. Y., Huang, R., & Zhou, X. (2015) · ACM Computing Surveys · doi:10.1145/2794400 · Cited by 719
Deep learning for site safety: Real-time detection of personal protective equipment
Nath, N. D., Behzadan, A. H., & Paal, S. G. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103085 · Cited by 540
A survey of the use of crowdsourcing in software engineering
Mao, K., Capra, L., Harman, M., & Jia, Y. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2016.09.015 · Cited by 385
Metadata
- Authors
- Yuen, M.-C., King, I., & Leung, K.-S.
- Venue
- 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing
- Publisher
- IEEE
- Published
- 2011-10
- Pages
- 766-773
- OpenAlex
- W2061863227
- Cited by
- 392
- References
- 73
- FWCI
- 21.18
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.