Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Distributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams -- familiarity with other members -- stands in contrast to crowd work's flexible, on-demand nature. We enable effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints. Huddler utilizes a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown. We first present a field experiment that demonstrates the value of familiarity for crowd teams: familiar crowd teams doubled the performance of ad-hoc (unfamiliar) teams on a collaborative task. We then report a two-week field deployment wherein Huddler enabled crowd workers to convene highly familiar teams in 18 minutes on average. This research advances the goal of supporting long-term, team-based collaborations without sacrificing the flexibility of crowd work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Psychological Safety and Learning Behavior in Work Teams
Edmondson, A. (1999) · Administrative Science Quarterly · doi:10.2307/2666999 · Cited by 12,026 · cited 3×
“To learn more, we included two related questions drawn from measures of psychological safety [14] in our survey.”
BackgroundInfluentialWhat Makes Teams Work: Group Effectiveness Research from the Shop Floor to the Executive Suite
Cohen, S. G., & Bailey, D. E. (1997) · Journal of Management · doi:10.1177/014920639702300303 · Cited by 3,526
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Olson, G. M., & Olson, J. S. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1523_4 · Cited by 1,983
Determinants of Coordination Modes within Organizations
Ven, A. H. V. D., Delbecq, A. L., & Koenig, R. (1976) · American Sociological Review · doi:10.2307/2094477 · Cited by 1,979
Hinds, P. J., & Mortensen, M. (2005) · Organization Science · doi:10.1287/orsc.1050.0122 · Cited by 1,072
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
Self-citationVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601
Parallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367
Research works printed but not linked: 44
- 10.1111/j.1083-6101.2010.01529.x · DOI · Paper page
- Linda Argote Paul Ingram John M. Levine and Richard L. Moreland. "Knowledge transfer in organizations: Learning from the experience of others." Organizational behavior and human decision processes 82 no. 1 (2000): 1--8. · DOI · Paper page
- Linda Argote. "Input uncertainty and organizational coordination in hospital emergency units." Administrative science quarterly (1982): 420--434.
- Jeffrey P. Bigham, Michael S. Bernstein, and Eytan Adar. "Human-Computer Interaction and Collective Intelligence." Handbook of Collective Intelligence (2015): 57.
- 10.1287/orsc.3.1.20 · DOI · Paper page
- Tom Finholt Lee Sproull and Sara Kiesler. "Communication and performance in ad hoc task groups." Intellectual teamwork: Social and technological foundations of cooperative work (1990): 291--325. · doi:10.5555/117848.117859
- 10.1080/13691180110097030 · DOI · Paper page
- Scott B. Droege, and Jenny M. Hoobler. "Employee turnover and tacit knowledge diffusion: A network perspective." Journal of Managerial Issues(2003): 5064.
- Amy C. Edmondson Teaming: How organizations learn, innovate, and compete in the knowledge economy. John Wiley & Sons, 2012.
- 10.2753/MIS0742-1222240104 · DOI · Paper page
- 10.1287/mnsc.31.9.1055 · doi:10.1287/mnsc.31.9.1055
- 10.1145/2818048.2819942 · DOI · Paper page
- Deborah H. Gruenfeld Elizabeth A. Mannix Katherine Y. Williams and Margaret A. Neale. "Group composition and decision making: How member familiarity and information distribution affect process and performance." Organizational behavior and human decision processes 67 no. 1 (1996): 1--15. · DOI · Paper page
- 10.1108/09696470910939215 · DOI
- 10.14778/2824032.2824062 · DOI
- J. Richard. Hackman, Leading teams: Setting the stage for great performances. Harvard Business Press, 2002.
- 10.1111/j.1744-6570.2003.tb00753.x · DOI · Paper page
- Mark Hefke, and Ljiljana Stojanovic. "An ontologybased approach for competence bundling and composition of ad-hoc teams in an organization." Proc. I-KNOW'04 (2004): 126--134.
- 10.5555/2007322.2007325 · doi:10.5555/2007322.2007325
- 10.1287/mnsc.1080.0921 · DOI · Paper page
- 10.1145/1240624.1240698 · DOI · Paper page
- 10.1145/2047196.2047202 · DOI · Paper page
- 10.1145/1869086.1869096 · DOI
- David A. Kravitz and Barbara Martin. "Ringelmann rediscovered: The original article." (1986): 936. · doi:10.1037//0022-3514.50.5.936 · Paper page
- Karim Lakhani, David A. Garvin, and Eric Lonstein. "Topcoder (a): Developing software through crowdsourcing." Harvard Business School General Management Unit Case 610-032 (2010).
- 10.1145/2702123.2702565 · DOI · Paper page
- 10.1177/0146167295214009 · DOI · Paper page
- 10.1145/1600150.1600159 · DOI · Paper page
- 10.1145/2851581.2892421 · DOI
- Ioanna Lykourentzou, Angeliki Antoniou, Yannick Naudet, and Steven P. Dow. "Personality Matters: Balancing for Personality Types Leads to Better Outcomes for Crowd Teams." (2016).
- 10.1371/journal.pone.0153048 · DOI
- 10.1145/2531602.2531663 · DOI · Paper page
- 10.1207/S15327957PSPR0401_8 · DOI · Paper page
- Mitzi M. Montoya-Weiss, Anne P. Massey, and Michael Song. "Getting it together: Temporal coordination and conflict management in global virtual teams." Academy of management Journal 44, no. 6 (2001): 1251--1262. · DOI · Paper page
- M. Mortensen 2014 "Constructing the Team: The Antecedents and Effects of Membership Model Divergence." Organization Science 25: 909--931. 10.1287/orsc.2013.0881 · DOI
- 10.1089/109493104323024410 · doi:10.1089/109493104323024410
- 10.1287/mnsc.1050.0366 · DOI · Paper page
- 10.1145/2642918.2647409 · DOI · Paper page
- 10.5465/19416520.2013.766405 · DOI
- 10.1145/1978942.1979102 · DOI · Paper page
- Anita Williams Woolley Christopher F. Chabris Alex Pentland Nada Hashmi and Thomas W. Malone. "Evidence for a collective intelligence factor in the performance of human groups." science 330 no. 6004 (2010): 686--688. · DOI · Paper page
- 10.1126/science.1205869 · DOI · Paper page
- 10.1145/2702123.2702508 · DOI · Paper page
- 10.1145/2531602.2531718 · DOI
Cited by
Larson, L., & DeChurch, L. A. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2019.101377 · Cited by 358
Gig work and gig workers: An integrative review and agenda for future research
Wu, D., & Huang, J. L. (2024) · Journal of Organizational Behavior · doi:10.1002/job.2775 · Cited by 168
Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations
Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025811 · Cited by 123
Self-citationTrue or False: Studying the Work Practices of Professional Fact-Checkers
Micallef, N., Armacost, V., Memon, N., & Patil, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512974 · Cited by 70
“For instance, mechanisms would need to be implemented to generate fagging of misinformation by a diverse set of individuals and to ensure the development of rapport between the crowd and fact-checkers [104, 105, 125].”
Background
Metadata
- Authors
- Salehi, N., McCabe, A., Valentine, M., & Bernstein, M.
- Venue
- Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
- Publisher
- ACM
- Published
- 2017-02-25
- Pages
- 1700-1713
- OpenAlex
- W2544952174
- Cited by
- 63
- References
- 9
- FWCI
- 8.86
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- Office of Naval Research (N00014-16-1-2894)
- National Science Foundation (IIS-1351131)
- Stanford Graduate Fellowship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.