Shepherding the crowd yields better work

Dow, Steven, Kulkarni, Anand, Klemmer, Scott, Hartmann, Björn·2012·Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
DOI 10.1145/2145204.2145355

Cited by (335)

Items 101–200 of 335

  1. How One Microtask Affects Another
    Edward Newell, Derek A. Ruths · 2016 · CHI Conference on Human Factors in Computing Systems
  2. Engagement effects of player rating system-based matchmaking for level ordering in human computation games
    Anurag Sarkar, Michael Williams, Sebastian Deterding, Seth Cooper · 2017 · International Conference on the Foundations of Digital Games (FDG)
  3. Crowd-AI Camera Sensing in the Real World
    Anhong Guo, Anuraag Jain, Shomiron Ghose, Gierad Laput, Chris Harrison, Jeffrey P. Bigham · 2018 · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies
  4. CrowdIA
    Tianyi Li, Kurt Luther, Chris L. North · 2018 · Proceedings of the ACM on Human-Computer Interaction
  5. Crowd vs. crowd
    Cheong Ha Park, KyoungHee Son, Joon Hyub Lee, Seok-Hyung Bae · 2013 · ACM Conference on Computer Supported Cooperative Work (CSCW)
  6. The Practice of Crowdsourcing
    Omar Alonso · 2019 · Synthesis lectures on information concepts, retrieval, and services
  7. Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection
    Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, Stefan Dietze · 2018 · Computer Supported Cooperative Work (CSCW)
  8. Collective firm-internal online idea development
    Qian Chen, Mats G. Magnusson, Jennie Björk · 2019 · European Journal of Innovation Management
  9. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships
    Ryo Suzuki, Niloufar Salehi, Michelle S. Lam, Juan C. Marroquin, Michael S. Bernstein · 2016 · arXiv (Cornell University)
  10. Tropel: Crowdsourcing Detectors with Minimal Training
    Geneviève Patterson, Grant Van Horn, Serge Belongie, Pietro Perona, James H. Hays · 2015 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  11. Considering human aspects on strategies for designing and managing distributed human computation
    Lesandro Ponciano, Francisco Brasileiro, Nazareno Andrade, Livia M. R. Sampaio · 2014 · Journal of Internet Services and Applications
  12. Understanding Worker Moods and Reactions to Rejection in Crowdsourcing
    Ujwal Gadiraju, Gianluca Demartini · 2019 · ACM Conference on Hypertext and Social Media (HT)
  13. Ga11y: An Automated GIF Annotation System for Visually Impaired Users
    Mingrui Ray Zhang, Mingyuan Zhong, Jacob Otto Wobbrock · 2022 · CHI Conference on Human Factors in Computing Systems
  14. Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool
    Sang Won Lee, Rebecca Krosnick, Sun Young Park, Brandon Keelean, Sach Vaidya, Stephanie D. O’Keefe, Walter S. Lasecki · 2018 · Proceedings of the ACM on Human-Computer Interaction
  15. Collaborative Problem Solving in an Open-Ended Scientific Discovery Game
    Aaron Bauer, Zoran Popović · 2017 · Proceedings of the ACM on Human-Computer Interaction
  16. What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
    Nikita Nangia, Saku Sugawara, Harsh K. Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman · 2021
  17. A Survey on Volunteer Management Systems
    Johannes Schönböck, Markus Raab, Josef Altmann, Elisabeth Kapsammer, Angelika Kusel, Birgit Pröll, Werner Retschitzegger, Wieland Schwinger · 2016 · Hawaii International Conference on System Sciences (HICSS)
  18. Learning From the Crowd
    Lena Mamykina, Thomas N. Smyth, Jill P. Dimond, Krzysztof Z. Gajos · 2016 · CHI Conference on Human Factors in Computing Systems
  19. Gut Instinct
    Vineet Pandey, Amnon Amir, Justine W. Debelius, Embriette R. Hyde, Tomasz Kościółek, Rob D. Knight, Scott Klemmer · 2017 · CHI Conference on Human Factors in Computing Systems
  20. Crowd Memory: Learning in the Collective
    Walter S. Lasecki, White, Samuel C., Kyle I. Murray, Jeffrey P. Bigham · 2012 · arXiv (Cornell University)
  21. Beyond AMT: An Analysis of Crowd Work Platforms
    Donna Vakharia, Matthew Lease · 2013 · arXiv (Cornell University)
  22. Trustworthiness in enterprise crowdsourcing
    Anurag Dwarakanath, N. C. Shrikanth, Kumar Abhinav, Alex Kass · 2016 · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)
  23. A system for scalable and reliable technical-skill testing in online labor markets
    Maria Christoforaki, Panagiotis G. Ipeirotis · 2015 · Computer Networks
  24. Effects of rationale awareness in online ideation crowdsourcing tasks
    Lu Xiao · 2014 · Journal of the Association for Information Science and Technology
  25. The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help
    Danula Hettiachchi, Mike Schaekermann, Tristan McKinney, Matthew Lease · 2021 · Proceedings of the ACM on Human-Computer Interaction
  26. Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation
    Wang, Yunlong, Venkatesh, Priyadarshini, Lim, Brian Y · 2022 · CHI Conference on Human Factors in Computing Systems
  27. Crowdsourcing in HCI Research
    Serge Egelman, Ed H., Steven P. Dow · 2014 · Springer eBooks
  28. Reputation Agent: Prompting Fair Reviews in Gig Markets
    Carlos Toxtli, Angela Richmond-Fuller, Saiph Savage · 2020 · International World Wide Web Conference (WWW)
  29. Architecting Real-Time Crowd-Powered Systems
    Walter S. Lasecki, Christopher Michael Homan, Jeffrey P. Bigham · 2014 · Human Computation
  30. Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters
    Alexandra Papoutsaki, Hua Guo, Danaë Metaxa, Connor Gramazio, Jeff Rasley, Wenting Xie, Guan Wang, Jeff J. S. Huang · 2015 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  31. Capturing Winning Ideas in Online Design Communities
    Faez Ahmed, Mark Fuge · 2017 · ACM Conference on Computer Supported Cooperative Work (CSCW)
  32. Kurator
    David C. Merritt, Jasmine Jones, Mark Steven Ackerman, Walter S. Lasecki · 2017 · ACM Conference on Computer Supported Cooperative Work (CSCW)
  33. MechanicalHeart
    William Callaghan, Joslin Goh, Michael M. Mohareb, Andrew S. P. Lim, Edith Law · 2018 · Proceedings of the ACM on Human-Computer Interaction
  34. Building a WordNet for Sinhala
    Indeewari Wijesiri, Malaka Gallage, Buddhika Gunathilaka, Madhuranga Lakjeewa, Daya C. Wimalasuriya, Gihan Dias, Rohini Paranavithana, Nisansa de Silva · 2014 · Global WordNet Conference (GWC)
  35. IdeaGens
    Joel Chan, Steven Dang, Steven P. Dow · 2016 · ACM Conference on Computer Supported Cooperative Work (CSCW)
  36. Social Incentives in Paid Collaborative Crowdsourcing
    Oluwaseyi Feyisetan, Elena Paslaru Bontas Simperl · 2017 · ACM Transactions on Intelligent Systems and Technology
  37. Creating Crowdsourced Research Talks at Scale
    Rajan Vaish, Shirish Goyal, Amin Saberi, Sharad Goel · 2018 · International World Wide Web Conference (WWW)
  38. Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy
    Tahir Abbas, Vassilis-Javed Khan, Ujwal Gadiraju, Panos Markopoulos · 2020 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  39. Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History
    Nai-Ching Wang, David L. Hicks, Kurt Luther · 2018 · Proceedings of the ACM on Human-Computer Interaction
  40. Critter
    Aditya Bharadwaj, Pao Siangliulue, Adam Marcus, Kurt Luther · 2019 · CHI Conference on Human Factors in Computing Systems
  41. Provisioning Quality-Aware Social Compute Units in the Cloud
    Muhammad Zuhri Catur Candra, Hong‐Linh Truong, Schahram Dustdar · 2013 · Lecture notes in computer science
  42. An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks
    Masaki Kobayashi, Hiromi Morita, Masaki Matsubara, Nobuyuki Shimizu, Atsuyuki Morishima · 2018 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  43. Crowdclass: Designing Classification-Based Citizen Science Learning Modules
    Doris Jung‐Lin Lee, Joanne C.M. Lo, Moonhyok Kim, Eric Paulos · 2016 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  44. Beyond Micro-Tasks
    Roman Lukyanenko, Jeffrey Parsons · 2018 · Journal of Database Management
  45. Beyond efficiency: Trust, AI, and surprise in knowledge work environments
    Allen Samuel Brown, Christopher R. Dishop, Andrew V. Kuznetsov, Ping-Ya Chao, Anita Williams Woolley · 2025 · Computers in Human Behavior
  46. Crowdsourcing Accessibility: Human-Powered Access Technologies
    Erin L. Brady, Jeffrey P. Bigham · 2015 · now publishers, Inc. eBooks
  47. Mining Crowdsourcing Problems from Discussion Forums of Workers
    Zahra Nouri, Henning Wachsmuth, Gregor Engels · 2020 · International Conference on Computational Linguistics
  48. Task Assignment Strategies for Crowd Worker Ability Improvement
    Masaki Matsubara, Ria Mae Borromeo, Sihem Amer-Yahia, Atsuyuki Morishima · 2021 · Proceedings of the ACM on Human-Computer Interaction
  49. Wish: Amplifying Creative Ability with Expert Crowds
    Anand P. Kulkarni, Prayag Narula, David Rolnitzky, Nathan Kontny · 2014 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  50. Online Crowdsourcing Campaigns: Bottom-Up versus Top-Down Process Model
    Jie Ren, Pinar Ozturk, William Yeoh · 2017 · Journal of Computer Information Systems
  51. Crowd Work and Collective Learning
    Jeffrey V. Nickerson · 2013 · SSRN Electronic Journal
  52. What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task
    Huaming Rao, Shih-Wen Huang, Wai‐Tat Fu · 2013 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  53. A Study of Urban Heat
    Stacey Kuznetsov, Martin Tomitsch · 2018 · CHI Conference on Human Factors in Computing Systems
  54. Human-centered computing and the future of work: lessons from Mechanical Turk and Turkopticon, 2008-2015
    Michael ‘Six’ Silberman · 2015 · eScholarship (California Digital Library)
  55. Diamonds from the rough: Improving drawing painting, and singing via crowdsourcing
    Yotam I. Gingold, Etienne Vouga, Eitan Grinspun, Haym Hirsh · 2012
  56. How We Write with Crowds
    Molly Q Feldman, Brian James McInnis · 2021 · Proceedings of the ACM on Human-Computer Interaction
  57. Galileo: Citizen-led Experimentation Using a Social Computing System
    Vineet Pandey, Tushar Koul, Chen Yang, Daniel T. McDonald, Mad Price Ball, Bastian Greshake Tzovaras, Rob D. Knight, Scott Klemmer · 2021 · CHI Conference on Human Factors in Computing Systems
  58. Interactive Consensus Agreement Games for Labeling Images
    Paul Upchurch, Daniel Sedra, Andrew Mullen, Haym Hirsh, Kavita Bala · 2016 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  59. Increasing Quality and Involvement in Online Peer Feedback Exchange
    Sneha R. Krishna Kumaran, Deana McDonagh, Brian P. Bailey · 2017 · Proceedings of the ACM on Human-Computer Interaction
  60. Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution
    David Q. Sun, Hadas Kotek, Christopher Klein, Mayank Gupta, William Ho Cheung Li, J. D. Williams · 2020 · International Conference on Computational Linguistics
  61. Plan Early, Revise More
    Sneha R. Krishna Kumaran, Yue Yin, Brian P. Bailey · 2021 · Proceedings of the ACM on Human-Computer Interaction
  62. WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
    Wesley Hanwen Deng, Wang Claire, Howard Ziyu Han, Jason Hong, Kenneth Holstein, Motahhare Eslami · 2025 · Proceedings of the ACM on Human-Computer Interaction
  63. CoSINT: Designing a Collaborative Capture the Flag Competition to Investigate Misinformation
    Sukrit Venkatagiri, Anirban Mukhopadhyay, David L. Hicks, Aaron Franklin Brantly, Kurt Luther · 2023 · Designing Interactive Systems Conference
  64. LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
    Chu Li, Zhihan Zhang, Michael Saugstad, Esteban Safranchik, Chaitanyashareef Kulkarni, Xiao-Yu Huang, Shwetak N. Patel, Vikram Iyer, Tim Althoff, Jon E. Froehlich · 2024 · International Conference on Human Factors in Computing Systems
  65. DesignQuizzer: A Community-Powered Conversational Agent for Learning Visual Design
    Zhenhui Peng, Qiaoyi Chen, Zhiyu Shen, Xiaojuan Ma, Antti Oulasvirta · 2024 · Proceedings of the ACM on Human-Computer Interaction
  66. Crowdsourcing planar facility location allocation problems
    Mohammad Allahbakhsh, Saeed Arbabı, Mohammadreza Galavii, Florian Daniel, Boualem Benatallah · 2018 · Computing
  67. The Practice of Crowdsourcing
    Omar Alonso · 2019 · Synthesis lectures on information concepts, retrieval, and services
  68. Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training
    Amy Rechkemmer, Ming Yin · 2020 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  69. The Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk Evaluation
    Xiaoni Duan, Chien-Ju Ho, Ming Yin · 2022 · International World Wide Web Conference (WWW)
  70. An expert‐in‐the‐loop method for domain‐specific document categorization based on small training data
    Kanyao Han, Rezvaneh Rezapour, Katia Nakamura, Dikshya Devkota, Daniel C. Miller, Jana Diesner · 2022 · Journal of the Association for Information Science and Technology
  71. Agile Modeling: From Concept to Classifier in Minutes
    Otilia Stretcu, Edward B. Vendrow, Kenji Hata, Krishnamurthy Viswanathan, Vittorio Ferrari, Sasan Tavakkol, Wenlei Zhou, Aditya Avinash, Enming Luo, Neil Alldrin et al. · 2023 · IEEE/CVF International Conference on Computer Vision (ICCV)
  72. Turker Tales
    Anna Kasunic, Chun-Wei Chiang, Geoff F. Kaufman, Saiph Savage · 2019 · Designing Interactive Systems Conference
  73. Improving Accuracy in Mobile Human Contributions
    Niels van Berkel, Matthias Budde, Senuri Wijenayake, Jorge Gonçalves · 2018 · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)
  74. Scaling Up to Tackle Low Levels of Urban Food Waste Recycling
    Katie Seaborn, Johanna Mähönen, Yvonne Rogers · 2020 · Designing Interactive Systems Conference
  75. To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing
    Markus Krause, René F. Kizilcec · 2015 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  76. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing
    Quan Ze Chen, Jonathan Bragg, Lydia B. Chilton, Daniel S. Weld · 2018 · arXiv (Cornell University)
  77. CoTacs: A Haptic Toolkit to Explore Effective On-Body Haptic Feedback by Ideating, Designing, Evaluating and Refining Haptic Designs Using Group Collaboration
    Moritz Alexander Messerschmidt, Juan Pablo Forero Cortés, Suranga Chandima Nanayakkara · 2024 · International Journal of Human-Computer Interaction
  78. Reimagining Meaningful Data Work through Citizen Science
    Ashley Boone, Annabel Rothschild, Xander Koo, Grace Pfohl, Alyssa Sheehan, Betsy James DiSalvo, Christopher A. Le Dantec, Carl Francis DiSalvo · 2024 · Proceedings of the ACM on Human-Computer Interaction
  79. WearWrite
    Michael Nebeling, Anhong Guo, Alexandra To, Steven P. Dow, Jaime B. Teevan, Jeffrey P. Bigham · 2015 · ACM Symposium on User Interface Software and Technology (UIST)
  80. PledgeWork
    Keiko Katsuragawa, Qi Shu, Edward Lank · 2019 · CHI Conference on Human Factors in Computing Systems
  81. Machine-Crowd-Expert Model for Increasing User Engagement and Annotation Quality
    Ana Elisa Méndez Méndez, Mark Cartwright, Juan Pablo Bello · 2019 · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA)
  82. Dropping the Baton?
    Tianyi Li, Chandler J. Manns, Chris L. North, Kurt Luther · 2019 · Proceedings of the ACM on Human-Computer Interaction
  83. In Their Shoes
    Yihong Wang, Konstantinos Papangelis, Ioanna Lykourentzou, Hai‐Ning Liang, Irwyn Sadien, Evangelia Demerouti, Vassilis-Javed Khan · 2020 · Proceedings of the ACM on Human-Computer Interaction
  84. CrowdLayout
    Divit P. Singh, Lee Lisle, T. M. Murali, Kurt Luther · 2018 · CHI Conference on Human Factors in Computing Systems
  85. Improvising with an Audience-Controlled Robot Performer
    Claire Mikalauskas, Tiffany Wun, Kevin Ta, Joshua Horacsek, Lora Oehlberg · 2018 · Designing Interactive Systems Conference
  86. Can Crowds Customize Instructional Materials with Minimal Expert Guidance?
    Kexin Bella Yang, Tomohiro Nagashima, Junhui Yao, Joseph Jay Williams, Kenneth Holstein, Vincent Aleven · 2021 · Proceedings of the ACM on Human-Computer Interaction
  87. VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository
    Hu, Kevin, Neil Gaikwad, Michiel A. Bakker, Madelon Hulsebos, Emanuel Zgraggen, César A. Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, Çağatay Demiralp · 2019 · arXiv (Cornell University)
  88. Quality Assured: Rethinking Annotation Strategies in Imaging AI
    Tim Rädsch, Annika Reinke, Vivienn Weru, Minu D. Tizabi, Nicholas E. Heller, Fabian Isensee, Annette Kopp‐Schneider, Lena Maier‐Hein · 2024 · Lecture notes in computer science
  89. Creating and Sharing Records of Multiplayer Online Game Play: Practices and Attitudes
    Frank M. Shipman III, Catherine Marshall · 2014 · Proceedings of the International AAAI Conference on Web and Social Media
  90. Not Everyone Writes Good Examples but Good Examples Can Come from Anywhere
    Shayan Doroudi, Ece Semiha Kamar, Emma Brunskill · 2019 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  91. A Knowledge Perspective on Quality in Complex Citizen Science
    Montserrat Prats López, Maura Soekijad, Hans Berends, Marleen Huysman · 2020 · Citizen Science Theory and Practice
  92. Am I Ready to Get Feedback? A Taxonomy of Factors Creators Consider Before Seeking Feedback on In-Progress Creative Work
    Sneha R. Krishna Kumaran, Wenxuan Wendy Shi, Brian P. Bailey · 2021 · Creativity and Cognition
  93. Correctness assessment of a crowdcoding project in a computer programming introductory course
    Sebastián Ferrán, Alejandra Beghelli, Gonzalo Huerta‐Cánepa, Federico Jensen · 2017 · Computer Applications in Engineering Education
  94. Startup dilemmas - Strategic problems of early-stage platforms on the internet
    Joni O. Salminen · 2014 · UTUPub (University of Turku)
  95. Your Text Is Hard to Read: Facilitating Readability Awareness to Support Writing Proficiency in Text Production
    Jakob Karolus, Sebastian S. Feger, Albrecht Schmidt, Paweł W. Woźniak · 2023 · Designing Interactive Systems Conference
  96. OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence Investigations
    Anirban Mukhopadhyay, Sukrit Venkatagiri, Kurt Luther · 2024 · Proceedings of the ACM on Human-Computer Interaction
  97. STEP: A Scalable Testing and Evaluation Platform
    Maria Christoforaki, Panagiotis G. Ipeirotis · 2014 · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
  98. Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks
    June Ahn, Ha Nguyen, Fabio Campos, William Young · 2021 · International Conference on Learning Analytics and Knowledge (LAK)
  99. Crowd Research: Open and Scalable University Laboratories
    Rajan Vaish, Snehalkumar S. Gaikwad, Geza Kovacs, Andreas Veit, Ranjay Krishna, Imanol Arrieta-Ibarra, Camelia Simoiu, Michael J. Wilber, Serge Belongie, Sharad Goel et al. · 2018 · Understanding innovation
  100. Collective intelligence and knowledge exploration: an introduction
    Salma Sassi, Mirjana Drenovak Ivanović, Richard Chbeir, Rajendra Prasath, Yannis Manolopoulos · 2022 · International Journal of Data Science and Analytics