The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
CrowdDB: answering queries with crowdsourcing
Open accessAbstract
Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204
Crowdsourcing systems on the World-Wide Web
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011) · Communications of the ACM · doi:10.1145/1924421.1924442 · Cited by 1,338
(2014) · Elsevier eBooks · doi:10.1016/b978-0-12-799957-9.00002-x · Cited by 1,313
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
The TSIMMIS project: Integration of heterogeneous information sources
Chawathe, S. S., García-Molina, H., Hammer, J., Ireland, K., Papakonstantinou, Y., Ullman, J. D., & Widom, J. (1994) · 情報処理学会研究報告データベースシステム(DBS) · Cited by 914
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879
Eddies: continuously adaptive query processing
Avnur, R., & Hellerstein, J. M. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335420 · Cited by 798
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
Optimizing Queries Across Diverse Data Sources
Haas, L. M., Kossmann, D., Wimmers, E. L., & Yang, J. (1997) · Very Large Data Bases · Cited by 456
Self-citationEfficient mid-query re-optimization of sub-optimal query execution plans
Kabra, N., & DeWitt, D. J. (1998) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/276304.276315 · Cited by 331
CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones
Yan, T., Kumar, V., & Ganesan, D. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814443 · Cited by 255
TurKit: tools for iterative tasks on mechanical Turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600159 · Cited by 237
Cost-based query scrambling for initial delays
Urhan, T., Franklin, M. J. M., & Amsaleg, L. (1998) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/276304.276317 · Cited by 211
Self-citationCrowdsourced Databases: Query Processing with People
Marcus, A., Wu, E., Karger, D. R., Madden, S. R., & Miller, R. C. (2011) · Cited by 191
Optimization of dynamic query evaluation plans
Cole, R. L., & Graefe, G. (1994) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/191839.191872 · Cited by 182
Answering Queries using Humans, Algorithms and Databases
Parameswaran, A., & Polyzotis, N. (2011) · Cited by 108
Scrambling query plans to cope with unexpected delays
Amsaleg, L., Tomasic, A., Franklin, M. J. M., & Urhan, T. (2002) · International Conference on Parallel and Distributed Information Systems · doi:10.1109/pdis.1996.568681 · Cited by 102
Self-citationHuang, E., Zhang, H., Parkes, D. C., Gajos, K. Z., & Chen, Y. (2010) · doi:10.1145/1837885.1837908 · Cited by 72
Designing adaptive feedback for improving data entry accuracy
Chen, K., Hellerstein, J. M., & Parikh, T. S. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866029.1866068 · Cited by 22
Cited by
Crowdsourcing systems on the World-Wide Web
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011) · Communications of the ACM · doi:10.1145/1924421.1924442 · Cited by 1,338
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968
The cultural work of microwork
Irani, L. (2015) · New Media & Society · doi:10.1177/1461444813511926 · Cited by 568
Evaluation on crowdsourcing research: Current status and future direction
Zhao, Y., & Zhu, Q. (2014) · Information Systems Frontiers · doi:10.1007/s10796-012-9350-4 · Cited by 566
Metadata
- Authors
- Franklin, M. J., Kossmann, D., Kraska, T., Ramesh, S., & Xin, R.
- Venue
- Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
- Publisher
- ACM
- Published
- 2011-06-12
- Pages
- 61-72
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2168144930
- Cited by
- 623
- References
- 20
- FWCI
- 63.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.