Crowdsourcing as a Tool for Research: Implications of Uncertainty
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Numerous crowdsourcing platforms are now available to support research as well as commercial goals. However, crowdsourcing is not yet widely adopted by researchers for generating, processing or analyzing research data. This study develops a deeper understanding of the circumstances under which crowdsourcing is a useful, feasible or desirable tool for research, as well as the factors that may influence researchers' decisions around adopting crowdsourcing technology. We conducted semi-structured interviews with 18 researchers in diverse disciplines, spanning the humanities and sciences, to illuminate how research norms and practitioners' dispositions were related to uncertainties around research processes, data, knowledge, delegation and quality. The paper concludes with a discussion of the design implications for future crowdsourcing systems to support research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gieryn, T. F. (1983) · American Sociological Review · doi:10.2307/2095325 · Cited by 4,838 · cited 1×
“For example, Gieryn’s foundational study illustrated how pioneering 19th-century British scientists deployed claims of professional status and objectivity to demarcate their work [35] from amateur contributions.”
BackgroundEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Organization design: an information processing view
Galbraith, J. R. (1974) · Interfaces · doi:10.1287/inte.4.3.28 · Cited by 2,451
Collins, H., & Evans, R. (2007) · doi:10.7208/chicago/9780226113623.001.0001 · Cited by 1,503
Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., et al. (2008) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2008.13689.x · Cited by 1,488
Scientific Collaboration on the Internet
(2008) · doi:10.7551/mitpress/9780262151207.001.0001 · Cited by 286
Data sharing, small science and institutional repositories
Cragin, M. H., Palmer, C. L., Carlson, J. R., & Witt, M. (2010) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2010.0165 · Cited by 286
Goals and Tasks: Two Typologies of Citizen Science Projects
Wiggins, A., & Crowston, K. (2012) · 2012 45th Hawaii International Conference on System Sciences · doi:10.1109/hicss.2012.295 · Cited by 93 · cited 2×
“Across different types of citizen science projects, researchers’ roles vary [76, 89, 64]: they act as contractors (interested parties ask researchers to conduct a specific scientific inquiry), leaders (researchers design projects for which volunteers contribute data), collaborators (researchers design projects for whi…”
BackgroundSelf-citation
Cited by
The Accuracy of Citizen Science Data: A Quantitative Review
Aceves‐Bueno, E., Adeleye, A. S., Feraud, M., Huang, Y., Tao, M., Yang, Y., & Anderson, S. E. (2017) · The Bulletin of the Ecological Society of America · doi:10.1002/bes2.1336 · Cited by 338
Everyday Uncertainty: How Blind People Use GenAI Tools for Information Access
Tang, X., Abdolrahmani, A., Gergle, D., & Piper, A. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713433 · Cited by 30
Simons, R. N., Gurari, D., & Fleischmann, K. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415176 · Cited by 27
Drawing Conversations Mediated by AI
Yurman, P., & Reddy, A. V. (2022) · Creativity and Cognition · doi:10.1145/3527927.3531448 · Cited by 23
“In HCI, uncertainty has been attended to through tools for modelling it or reducing its effects [17,43] or through methods for better understanding how to design for uncertainty [36].”
MethodLLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions
Liao, Z., Antoniak, M., Cheong, I., Cheng, E. Y.-Y., Lee, A.-H., Lo, K., Chang, J. C., & Zhang, A. X. (2024) · arXiv · doi:10.48550/arxiv.2411.05025 · Cited by 7
Metadata
- Authors
- Law, E., Gajos, K. Z., Wiggins, A., Gray, M. L., & Williams, A.
- Venue
- Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
- Publisher
- ACM
- Published
- 2017-02-25
- Pages
- 1544-1561
- OpenAlex
- W2589038172
- Cited by
- 69
- References
- 9
- FWCI
- 10.89
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.