Shepherding the crowd yields better work
Abstract
Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work has demonstrated that feedback improves the quality of crowd work.
“Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
From Peer and self assessment in massive online classes (2013) · cites this paper as Dow et al. 2012
Self-assessment provides learning gains not seen with external evaluation.
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
From Peer and self assessment in massive online classes (2013) · cites this paper as Dow et al. 2012
Cites
Situated Learning: Legitimate Peripheral Participation.
Bloch, M. E. F., Lave, J., & Wenger, É. (1994) · Man · doi:10.2307/2804509 · Cited by 39,323 · see citing works
Lave, J., & Wenger, E. (1991) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815355 · Cited by 33,219 · see citing works
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631 · see citing works
Formative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · see citing works
The Cambridge Handbook of Expertise and Expert Performance
Ericsson, K. A., Charness, N. H., Feltovich, P. J., Hoffman, R. R., Michelene, T. H., Hunt, E., Prietula, M. J., Amirault, R. J., Branson, R. K., Buchanan, B. G., et al. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816796 · Cited by 3,160 · see citing works
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941 · see citing works
Sustainable Assessment: Rethinking assessment for the learning society
Boud, D. (2000) · Studies in Continuing Education · doi:10.1080/713695728 · Cited by 1,545 · see citing works
Becoming a self-regulated learner: Which are the key subprocesses?
Zimmerman, B. J. (1986) · Contemporary Educational Psychology · doi:10.1016/0361-476x(86)90027-5 · Cited by 1,247 · see citing works
The Checklist Manifesto: How to Get Things Right
Gawande, A. A. (2011) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(15)30310-0 · Cited by 1,155 · see citing works
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F. J., & Wang, J. (2010) · doi:10.1145/1837885.1837906 · Cited by 991 · see citing works
Enhancing Learning Through Self-assessment
Boud, D. J. (2013) · Routledge eBooks · doi:10.4324/9781315041520 · Cited by 914 · see citing works
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 890 · see citing works
Crowdsourcing graphical perception
Heer, J., & Bostock, M. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 819 · see citing works
Human Development And Education
Havighurst, R. J. (1953) · Andalas University Repository (Andalas University) · Cited by 776 · see citing works
A revised class play method of peer assessment.
Masten, A. S., Morison, P., & Pellegrini, D. S. (1985) · Developmental Psychology · doi:10.1037/0012-1649.21.3.523 · Cited by 761 · see citing works
Soylent: a word processor with a crowd inside
(2010) · doi:10.1145/1866029.1866078 · Cited by 686 · see citing works
Self-citationEnhancing learning through self-assessment
Vella, F. (1996) · Biochemical Education · doi:10.1016/0307-4412(96)82523-4 · Cited by 671 · see citing works
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · see citing works
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A. E., Miller, R. C., Tatarowicz, A., White, B., White, S., & YEH, T. P. (2010) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536 · see citing works
Assessing Self- and Peer-assessment: The students' views
Hanrahan, S. J., & Isaacs, G. (2001) · Higher Education Research & Development · doi:10.1080/07294360123776 · Cited by 516 · see citing works
Lampe, C. A. C., & Resnick, P. (2004) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/985692.985761 · Cited by 405 · see citing works
Breaking monotony with meaning: Motivation in crowdsourcing markets
Chandler, D. R., & Kapelner, A. D. (2013) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2013.03.003 · Cited by 400 · see citing works
Hinds, P. J. (1999) · Journal of Experimental Psychology: Applied · doi:10.1037/1076-898x.5.2.205 · Cited by 397 · cited 1× · see citing works
“However, experts often underestimate the difficulty novices face in solving tasks [13] or use language or concepts that are beyond the grasp of novices [8].”
BackgroundThere’s No Place Like Home: An Evaluation of Early Supported Discharge for Stroke
Mayo, N. E., Wood-Dauphinee, S., Côté, R., Gayton, D., Carlton, J., Buttery, J., & Tamblyn, R. (2000) · Stroke · doi:10.1161/01.str.31.5.1016 · Cited by 395 · see citing works
Using Assessment for Learning and Learning from Assessment
Taras, M. (2002) · Assessment & Evaluation in Higher Education · doi:10.1080/0260293022000020273 · Cited by 314 · see citing works
To Feedback or Not to Feedback in Student Self-assessment
Taras, M. (2003) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930301678 · Cited by 271 · see citing works
Viégas, F. B., Wattenberg, M., & McKeon, M. M. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-73257-0_49 · Cited by 195 · see citing works
Exploring iterative and parallel human computation processes
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010) · doi:10.1145/1837885.1837907 · Cited by 167 · see citing works
Budget-optimal crowdsourcing using low-rank matrix approximations
Karger, D. R., Oh, S., & Shah, D. (2011) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2011.6120180 · Cited by 150 · see citing works
A Study in Self‐assessment: tutor and students’ perceptions of performance criteria
Orsmond, P., Merry, S. P., & Reiling, K. (1997) · Assessment & Evaluation in Higher Education · doi:10.1080/0260293970220401 · Cited by 145 · see citing works
The Checklist Manifesto: How to Get Things Right
Rosen, D. (2010) · JAMA · doi:10.1001/jama.2010.144 · Cited by 144 · see citing works
The interactive examination: assessing students' self‐assessment ability
Mattheos, N., Nattestad, A., Falk‐Nilsson, E., & Attström, R. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01788.x · Cited by 118 · see citing works
The Social Psychological Effects of Feedback on the Production of Internet Information Pools
Cheshire, C., & Antin, J. (2008) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2008.00416.x · Cited by 101 · see citing works
Sellers' problems in human computation markets
Silberman, M. ‘., Ross, J., Irani, L. C., & Tomlinson, B. (2010) · doi:10.1145/1837885.1837891 · Cited by 74 · see citing works
The impact of social information on visual judgments
Hullman, J. R., Adar, E., & Shah, P. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979157 · Cited by 72 · see citing works
Annett, J. (1969) · Penguin Books · Cited by 55 · see citing works
The Causes of Peer Effects in Production: Evidence from a Series of Field ExperimentsPreprint
Horton, J. J., & Zeckhauser, R. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1652993 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · see citing works
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390 · see citing works
Self-citationQuality Control in Crowdsourcing Systems: Issues and Directions
Allahbakhsh, M., Benatallah, B., Ignjatović, A., Motahari‐Nezhad, H. R., Bertino, E., & Dustdar, S. (2013) · IEEE Internet Computing · doi:10.1109/mic.2013.20 · Cited by 384 · see citing works
Quality Control in Crowdsourcing
Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2018) · ACM Computing Surveys · doi:10.1145/3148148 · Cited by 316 · see citing works
Crowd science user contribution patterns and their implications
Sauermann, H., & Franzoni, C. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1408907112 · Cited by 303 · see citing works
Understanding Malicious Behavior in Crowdsourcing Platforms
Gadiraju, U., Kawase, R., Dietze, S., & Demartini, G. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702443 · Cited by 241 · see citing works
Sharing Knowledge and Expertise: The CSCW View of Knowledge Management
Ackerman, M. S., Dachtera, J., Pipek, V., & Wulf, V. (2013) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-013-9192-8 · Cited by 229 · see citing works
Expert crowdsourcing with flash teams
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T. S., Valentine, M., & Bernstein, M. S. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647409 · Cited by 225 · see citing works
Chang, J. C., Amershi, S., & Kamar, E. S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3026044 · Cited by 217 · see citing works
Two's company, three's a crowd: a case study of crowdsourcing software development
Stol, K.-J., & Fitzgerald, B. F. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568249 · Cited by 204 · see citing works
Learning from crowdsourced labeled data: a survey
Zhang, J., Wu, X., & Sheng, V. S. (2016) · Artificial Intelligence Review · doi:10.1007/s10462-016-9491-9 · Cited by 203 · see citing works
What can crowdsourcing do for decision support?
Chiu, C., Liang, T., & Turban, E. (2014) · Decision Support Systems · doi:10.1016/j.dss.2014.05.010 · Cited by 190 · see citing works
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · see citing works
Haptic experience design: What hapticians do and where they need help
Schneider, O., MacLean, K. E., Swindells, C., & Booth, K. S. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.04.004 · Cited by 161 · see citing works
Xu, A., Huang, S., & Bailey, B. P. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531604 · Cited by 157 · see citing works
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A., & Arawjo, I. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676450 · Cited by 151 · see citing works
Self-citationOlson, J. S., & Kellogg, W. A. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8 · Cited by 148 · see citing works
Task assignment optimization in knowledge-intensive crowdsourcing
Roy, S. B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2015) · The VLDB Journal · doi:10.1007/s00778-015-0385-2 · Cited by 134 · see citing works
McInnis, B. J., Cosley, D., Nam, C., & Leshed, G. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858539 · Cited by 119 · see citing works
Peer and Self Assessment in Massive Online Classes
Kulkarni, C. E., Koh, P. W., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. (2014) · Understanding innovation · doi:10.1007/978-3-319-06823-7_9 · Cited by 104 · see citing works
Self-citationLykourentzou, I., Antoniou, A., Naudet, Y., & Dow, S. P. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819979 · Cited by 100 · see citing works
Self-citationYuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819953 · Cited by 98 · see citing works
Self-citationCOCO Attributes: Attributes for People, Animals, and Objects
Patterson, G., & Hays, J. H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_6 · Cited by 97 · see citing works
Hu, K., Gaikwad, S. S., Hulsebos, M., Bakker, M. A., Zgraggen, E., Hidalgo, C. A., Kraska, T., Li, G., Satyanarayan, A., & Demiralp, Ç. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300892 · Cited by 91 · see citing works
Components and Functions of Crowdsourcing Systems – A Systematic Literature Review
Hetmank, L. (2013) · Journal of the Association for Information Systems · doi:10.13140/2.1.3836.4166 · Cited by 94 · see citing works
Using Metrics of Curation to Evaluate Information-Based Ideation
Kerne, A., Webb, A. M., Smith, S. M., Linder, R., Lupfer, N., Qu, Y., Moeller, J., & Damaraju, S. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2591677 · Cited by 93 · see citing works
Structuring Interactions for Large-Scale Synchronous Peer Learning
Coetzee, D., Lim, S., Fox, A., Hartmann, B., & Hearst, M. A. (2015) · Conference on Computer Supported Cooperative Work · doi:10.1145/2675133.2675251 · Cited by 90 · see citing works
Self-citationToward a Learning Science for Complex Crowdsourcing Tasks
Doroudi, S., Kamar, E. S., Brunskill, E., & Horvitz, E. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858268 · Cited by 91 · see citing works
Rzeszotarski, J. M., & Kittur, A. (2012) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2380116.2380125 · Cited by 90 · see citing works
Improving Crowd Innovation with Expert Facilitation
Chan, J., Dang, S., & Dow, S. P. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2820023 · Cited by 86 · see citing works
Self-citationA Checklist to Combat Cognitive Biases in Crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86 · see citing works
Toward Efficient Team Formation for Crowdsourcing in Noncooperative Social Networks
Wang, W., Jiang, J., An, B., Jiang, Y., & Chen, B. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2602498 · Cited by 86 · see citing works
MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy
Drapeau, R., Chilton, L. B., Bragg, J., & Weld, D. S. (2016) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v4i1.13270 · Cited by 83 · see citing works
MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture
Kulkarni, A. P., Gutheim, P., Narula, P., Rolnitzky, D., Parikh, T. S., & Hartmann, B. (2012) · IEEE Internet Computing · doi:10.1109/mic.2012.72 · Cited by 81 · see citing works
Self-citationA Survey on Task Assignment in Crowdsourcing
Hettiachchi, D., Kostakos, V., & Gonçalves, J. (2022) · ACM Computing Surveys · doi:10.1145/3494522 · Cited by 78 · see citing works
Combining crowdsourcing and learning to improve engagement and performance
Dontcheva, M., Morris, R. R., Brandt, J., & Gerber, E. M. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557217 · Cited by 78 · see citing works
No Workflow Can Ever Be Enough
Retelny, D., Bernstein, M. S., & Valentine, M. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134724 · Cited by 76 · see citing works
Work design in a digitized gig economy
Schroeder, A. N., Bricka, T. M., & Whitaker, J. H. (2019) · Human Resource Management Review · doi:10.1016/j.hrmr.2019.100692 · Cited by 79 · see citing works
Phuttharak, J., & Loke, S. W. (2018) · IEEE Access · doi:10.1109/access.2018.2885353 · Cited by 74 · see citing works
Rating Working Conditions on Digital Labor Platforms
Harmon, E., & Silberman, M. ‘. (2018) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-018-9313-5 · Cited by 73 · see citing works
Nebeling, M., To, A., Guo, A., de Freitas, A. A., Teevan, J. B., Dow, S. P., & Bigham, J. P. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858169 · Cited by 71 · see citing works
Self-citationAccounting for Market Frictions and Power Asymmetries in Online Labor Markets
Kingsley, S., Gray, M. L., & Suri, S. (2015) · Policy & Internet · doi:10.1002/poi3.111 · Cited by 70 · see citing works
Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., & Bernstein, M. S. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858121 · Cited by 69 · see citing works
Khovanskaya, V. D., Dombrowski, L. S., Rzeszotarski, J. M., & Sengers, P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359310 · Cited by 68 · see citing works
A pilot study of using crowds in the classroom
Dow, S. P., Gerber, E. M., & Wong, A. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470686 · Cited by 66 · see citing works
Self-citationWhiting, M. E., Gamage, D., Gaikwad, S. S., Gilbee, A., Goyal, S., Ballav, A., Majeti, D., Chhibber, N., Richmond-Fuller, A., Vargus, F., et al. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998234 · Cited by 66 · see citing works
Designing moocs to facilitate participant self-monitoring for self-directed learning
Zhu, M., & Bonk, C. J. (2019) · Online Learning · doi:10.24059/olj.v23i4.2037 · Cited by 64 · see citing works
Experiences surveying the crowd
Marshall, C. C., & Shipman, F. M., III (2013) · ACM Web Science Conference (WebSci) · doi:10.1145/2464464.2464485 · Cited by 63 · see citing works
Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025883 · Cited by 60 · see citing works
Self-citationCrowdsourced Data Management: Industry and Academic Perspectives
Marcus, A., & Parameswaran, A. (2015) · now publishers, Inc. eBooks · doi:10.1561/9781680830910 · Cited by 58 · see citing works
Huang, S., & Fu, W. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470743 · Cited by 58 · see citing works
Experiments on Motivational Feedback for Crowdsourced Workers
Lee, T. Y., Dugan, C., Geyer, W., Ratchford, T., Rasmussen, J. C., Shami, N. S., & Lupushor, S. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v7i1.14428 · Cited by 57 · see citing works
Interactive Guidance Techniques for Improving Creative Feedback
Ngoon, T. J., Fraser, C. A., Weingarten, A. S., Dontcheva, M., & Klemmer, S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173629 · Cited by 55 · see citing works
Self-citationUsing Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks
Gadiraju, U., Fetahu, B., Kawase, R., Siehndel, P., & Dietze, S. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3119930 · Cited by 54 · see citing works
Stol, K.-J., Çağlayan, B., & Fitzgerald, B. F. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2774297 · Cited by 53 · see citing works
Zhu, H., Dow, S. P., Kraut, R. E., & Kittur, A. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531718 · Cited by 52 · see citing works
Self-citationDaniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02546 · Cited by 52 · see citing works
Cai, C. J., Iqbal, S. T., & Teevan, J. B. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858237 · Cited by 51 · see citing works
Greenberg, M. D., Easterday, M. W., & Gerber, E. M. (2015) · Creativity and Cognition · doi:10.1145/2757226.2757249 · Cited by 51 · see citing works
Nguyen, T. T. D. T., Garncarz, T., Ng, F., Dabbish, L., & Dow, S. P. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998319 · Cited by 51 · see citing works
Self-citationEnhancing reliability using peer consistency evaluation in human computation
Huang, S.-W., & Fu, W.-T. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441847 · Cited by 50 · see citing works
A survey of mobile crowdsensing and crowdsourcing strategies for smart mobile device users
Ray, A., Chowdhury, C., Bhattacharya, S., & Roy, S. (2022) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-022-00110-9 · Cited by 48 · see citing works
Providing real-time assistance in disaster relief by leveraging crowdsourcing power
Yang, D., Zhang, D., Frank, K., Robertson, P., Jennings, E., Roddy, M., & Lichtenstern, M. (2014) · Personal and Ubiquitous Computing · doi:10.1007/s00779-014-0758-3 · Cited by 48 · see citing works
Vaish, R., Gaikwad, S. S., Kovacs, G., Veit, A., Krishna, R., Arrieta-Ibarra, I., Simoiu, C., Wilber, M. J., Belongie, S., Goel, S., et al. (2017) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3126594.3126648 · Cited by 48 · see citing works
Cross, A., Bayyapunedi, M., Ravindran, D., Cutrell, E. B., & Thies, W. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531670 · Cited by 47 · see citing works
Modeling, Enacting, and Integrating Custom Crowdsourcing Processes
Tranquillini, S., Daniel, F., Kucherbaev, P., & Casati, F. (2015) · ACM Transactions on the Web · doi:10.1145/2746353 · Cited by 47 · see citing works
Lafreniere, B. J., Grossman, T., Anderson, F., Matejka, J., Kerrick, H., Nagy, D., Vasey, L., Atherton, E., Beirne, N., Coelho, M., et al. (2016) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2984511.2984553 · Cited by 47 · see citing works
Attention Please: Your Attention Check Questions in Survey Studies Can Be Automatically Answered
Pei, W., Mayer, A., Tu, K., & Yue, C. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380195 · Cited by 48 · see citing works
Hata, K., Krishna, R., Fei-Fei, L., & Bernstein, M. S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998248 · Cited by 45 · see citing works
Kokkalis, N., Köhn, T., Huebner, J., Lee, M., Schulze, F., & Klemmer, S. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2513560 · Cited by 44 · see citing works
Self-citationCrowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting
Agapie, E., Teevan, J. B., & Monroy‐Hernández, A. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13235 · Cited by 44 · see citing works
Mahyar, N., James, M. R., Ng, M., Wu, R. A., & Dow, S. P. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173769 · Cited by 43 · see citing works
Self-citationDifferences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data
Findlater, L., Zhang, J., Froehlich, J. E., & Moffatt, K. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025820 · Cited by 42 · see citing works
Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing
Chen, Q., Bragg, J., Chilton, L. B., & Weld, D. S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300761 · Cited by 41 · see citing works
Collective Creativity: Where we are and where we might goPreprint
Yu, L., Nickerson, J. V., & Sakamoto, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1204.3890 · Cited by 41 · see citing works
Ma, R., Gui, X., & Kou, Y. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581106 · Cited by 46 · see citing works
Sprinks, J., Wardlaw, J., Houghton, R. J., Bamford, S. P., & Morley, J. G. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.03.003 · Cited by 40 · see citing works
Gleason, C., Ahmetovic, D., Savage, S., Toxtli, C., Posthuma, C., Asakawa, C., Kitani, K., & Bigham, J. P. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3191741 · Cited by 39 · see citing works
Community heuristics for user interface evaluation of crowdsourcing platforms
Campo, S. À., Khan, V.-J., Papangelis, K., & Markopoulos, P. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.02.028 · Cited by 39 · see citing works
Reynante, B. M., Dow, S. P., & Mahyar, N. (2021) · Digital Government Research and Practice · doi:10.1145/3487607 · Cited by 42 · see citing works
Self-citationKokkalis, N., Köhn, T., Pfeiffer, C., Chornyi, D., Bernstein, M. S., & Klemmer, S. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441922 · Cited by 38 · see citing works
Self-citationLampinen, A., & Brown, B. A. T. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025515 · Cited by 38 · see citing works
Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd
Gadiraju, U., Möller, S., Nöllenburg, M., Saupe, D., Egger-Lampl, S., Archambault, D., & Fisher, B. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66435-4_2 · Cited by 38 · see citing works
Lampinen, A., McGregor, M., Comber, R., & Brown, B. A. T. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274369 · Cited by 38 · see citing works
What? How? Where? A Survey of Crowdsourcing
Yin, X. X., Liu, W., Wang, Y., Yang, C., & Lü, L. (2013) · Lecture notes in electrical engineering · doi:10.1007/978-94-007-7618-0_22 · Cited by 37 · see citing works
Online Sequencing of Non-Decomposable Macrotasks in Expert Crowdsourcing
Schmitz, H., & Lykourentzou, I. (2018) · ACM Transactions on Social Computing · doi:10.1145/3140459 · Cited by 37 · see citing works
Ren, J., Nickerson, J., Mason, W. A., Sakamoto, Y., & Graber, B. (2014) · Decision Support Systems · doi:10.1016/j.dss.2014.05.009 · Cited by 43 · see citing works
Ikeda, K., & Bernstein, M. S. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858327 · Cited by 34 · see citing works
Communicating Context to the Crowd for Complex Writing Tasks
Salehi, N., Teevan, J. B., Iqbal, S. T., & Kamar, E. S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998332 · Cited by 33 · see citing works
Comparing infrared and webcam eye tracking in the Visual World Paradigm
Vos, M., Minor, S., & Ramchand, G. C. (2022) · Glossa Psycholinguistics · doi:10.5070/g6011131 · Cited by 33 · see citing works
Distributed Apprenticeship in Online Communities
Hui, J. S., Easterday, M. W., & Gerber, E. M. (2018) · Human-Computer Interaction · doi:10.1080/07370024.2018.1469409 · Cited by 33 · see citing works
On-Demand Digital Economy: Can Experience Ensure Work and Income Security for Microtask Workers?
Rani, U., & Furrer, M. (2019) · Jahrbücher für Nationalökonomie und Statistik · doi:10.1515/jbnst-2018-0019 · Cited by 33 · see citing works
Crowd-based design activities: helping students connect with users online
Hui, J., Gerber, E., & Dow, S. W. (2014) · Conference on Designing Interactive Systems · doi:10.1145/2598510.2598538 · Cited by 40 · see citing works
Self-citationThe Effects of Sequence and Delay on Crowd Work
Lasecki, W. S., Rzeszotarski, J. M., Marcus, A., & Bigham, J. P. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702594 · Cited by 32 · see citing works
Factors Influencing the Decision to Crowdsource
Thuan, N. H., Antunes, P. A., & Johnstone, D. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41347-6_9 · Cited by 32 · see citing works
Ikeda, K., & Hoashi, K. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025917 · Cited by 32 · see citing works
Deka, B., Huang, Z., Franzen, C., Nichols, J. W., Li, Y., & Kumar, R. (2017) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3126594.3126647 · Cited by 32 · see citing works
Crowds, not Drones: Modeling Human Factors in Interactive Crowdsourcing
Roy, S. B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2013) · Cited by 31 · see citing works
Bragg, J., Mausam, M., & Weld, D. S. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242598 · Cited by 31 · see citing works
Morris, M. R., Bigham, J. P., Brewer, R. N., Bragg, J., Kulkarni, A. P., Li, J., & Savage, S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025687 · Cited by 31 · see citing works
Self-citation
- Authors
- Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B.
- Affiliations
- Carnegie Mellon University (US)
- University of California, Berkeley (US)
- Stanford University (US)
- Venue
- Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
- Publisher
- ACM
- Published
- 2012-02-11
- Pages
- 1013-1022
- OpenAlex
- W2168765606
- Cited by
- 334
- References
- 38
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 14 · 2023: 14 · 2022: 18 · 2021: 24 · 2020: 17 · 2019: 40 · 2018: 43 · 2017: 33 · 2016: 31 · 2015: 23 · 2014: 27 · 2013: 28 · 2012: 9
- FWCI
- 19.20
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Computer Supported Cooperative Work (CSCW) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- crowdsourcing, self-assessment, expert feedback, work quality, consumer reviews, worker learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
