CrowdDB: answering queries with crowdsourcing
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204
Crowdsourcing systems on the World-Wide Web
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011) · Communications of the ACM · doi:10.1145/1924421.1924442 · Cited by 1,338
(2014) · Elsevier eBooks · doi:10.1016/b978-0-12-799957-9.00002-x · Cited by 1,314
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
The TSIMMIS project: Integration of heterogeneous information sources
Chawathe, S. S., García-Molina, H., Hammer, J., Ireland, K., Papakonstantinou, Y., Ullman, J. D., & Widom, J. (1994) · 情報処理学会研究報告データベースシステム(DBS) · Cited by 914
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879
Eddies: continuously adaptive query processing
Avnur, R., & Hellerstein, J. M. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335420 · Cited by 798
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
Optimizing Queries Across Diverse Data Sources
Haas, L. M., Kossmann, D., Wimmers, E. L., & Yang, J. (1997) · Very Large Data Bases · Cited by 456
Self-citation
Cited by
Crowdsourcing systems on the World-Wide Web
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011) · Communications of the ACM · doi:10.1145/1924421.1924442 · Cited by 1,338
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968
The cultural work of microwork
Irani, L. (2015) · New Media & Society · doi:10.1177/1461444813511926 · Cited by 568
Evaluation on crowdsourcing research: Current status and future direction
Zhao, Y., & Zhu, Q. (2014) · Information Systems Frontiers · doi:10.1007/s10796-012-9350-4 · Cited by 566
CrowdER: crowdsourcing entity resolution
Wang, J., Kraska, T., Franklin, M. J., & Feng, J. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350263 · Cited by 507
Self-citationTruth inference in crowdsourcing: is the problem solved?
Zheng, Y., Li, G., Li, Y., Shan, C., & Cheng, R. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3055540.3055547 · Cited by 421
“It can be seen from Figure 3(e) that workers’ RMSE vary in [20, 45], and the average RMSE is 28.”
BackgroundMethodInfluentialMedusa: a programming framework for crowd-sensing applications
Ra, M.-R., Liu, B., La Porta, T. F., & Govindan, R. (2012) · Proceedings of the 10th international conference on Mobile systems, applications, and services · doi:10.1145/2307636.2307668 · Cited by 330
Open challenges for data stream mining research
Krempl, G., Žliobaite, I., Brzeziński, D., Hüllermeier, E., Last, M., Lemaire, V., Noack, T., Shaker, A., Sievi, S., Spiliopoulou, M., et al. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2674026.2674028 · Cited by 295
Collaboratively crowdsourcing workflows with turkomatic
Kulkarni, A., Can, M., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145354 · Cited by 286
Crowdsourced Data Management: A Survey
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2535242 · Cited by 283
Self-citationMarcus, A., Wu, E., Karger, D., Madden, S., & Miller, R. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/2047485.2047487 · Cited by 278
Magellan: toward building entity matching management systems
Konda, P., Das, S., Suganthan G. C., P., Doan, A., Ardalan, A., Ballard, J. R., Li, H., Panahi, F., Zhang, H., Naughton, J., et al. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994535 · Cited by 262
CrowdScreen: algorithms for filtering data with humans
Parameswaran, A. G., Garcia-Molina, H., Park, H., Polyzotis, N., Ramesh, A., & Widom, J. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213878 · Cited by 207
“Recently, there has been a lot of interest in the database research community in using crowdsourcing within a database [2, 3, 4, 12, 17].”
BackgroundSo who won?: dynamic max discovery with the crowd
Guo, S., Parameswaran, A., & Garcia-Molina, H. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213880 · Cited by 178
“More generally, in the last several years, there has been a significant amount of work regarding crowdsourcing systems, both inside [14, 16] and outside [22, 23] the database community.”
BackgroundSTFU NOOB! Predicting Crowdsourced Decisions on Toxic Behavior in Online Games
Blackburn, J., & Kwak, H. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5905 · Cited by 169
“While crowdsourcing, coined by Jeff Howe [17], has had huge impacts on a variety of areas, such as answering queries [12], assessing visual design [15], translating texts [43], conducting user studies [20], and collecting disaster responses [14], inherent noise in outcomes by laborers has received…”
BackgroundHybrid entity clustering using crowds and data
Lee, J., Cho, H., Park, J.-W., Cha, Y.-R., Hwang, S.-W., Nie, Z., & Wen, J.-R. (2013) · The VLDB Journal · doi:10.1007/s00778-013-0328-8 · Cited by 15
“To address this problem, we introduce crowdsourcing [18, 19,32,38,43], which is effective for disambiguating different relationships by harnessing human computations.”
BackgroundMethodInfluentialNouri, Z., Prakash, N., Gadiraju, U., & Wachsmuth, H. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584039 · Cited by 9
“Thus, workers may interpret the instructions they get differently [12].”
Logical and Physical Optimizations for SQL Query Execution over Large Language Models
Satriani, D., Veltri, E., Santoro, D., Rosato, S., Varriale, S., & Papotti, P. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725411 · Cited by 8
Combining Human and Machine Confidence in Truthfulness Assessment
Qu, Y., Roitero, K., Barbera, D. L., Spina, D., Mizzaro, S., & Demartini, G. (2023) · Journal of Data and Information Quality · doi:10.1145/3546916 · Cited by 6
Metadata
- Authors
- Franklin, M. J., Kossmann, D., Kraska, T., Ramesh, S., & Xin, R.
- Venue
- Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
- Publisher
- ACM
- Published
- 2011-06-12
- Pages
- 61-72
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2168144930
- Cited by
- 623
- References
- 20
- FWCI
- 63.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.