On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Abstract
Crowdsourcing is being increasingly adopted as a platform to run studies with human subjects. Running a crowdsourcing experiment involves several choices and strategies to successfully port an experimental design into an otherwise uncontrolled research environment, e.g., sampling crowd workers, mapping experimental conditions to micro-tasks, or ensure quality contributions. While several guidelines inform researchers in these choices, guidance of how and what to report from crowdsourcing experiments has been largely overlooked. If under-reported, implementation choices constitute variability sources that can affect the experiment's reproducibility and prevent a fair assessment of research outcomes. In this paper, we examine the current state of reporting of crowdsourcing experiments and offer guidance to address associated reporting issues. We start by identifying sensible implementation choices, relying on existing literature and interviews with experts, to then extensively analyze the reporting of 171 crowdsourcing experiments. Informed by this process, we propose a checklist for reporting crowdsourcing experiments.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
(2002) · Cited by 13,211
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · cited 5×
“The first example was the PRISMA checklist [71] that assesses the methodological rigor of systematic reviews.”
BackgroundMethodInfluentialSaturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033 · cited 1×
“From this list, we took a random sample of 15 papers published in the last eight years, which we drafted incrementally until we reached saturation [66].”
MethodThe American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 3×
“Research has shown how crowdsourcing could be leveraged in a wide range of tasks and domains (e.g., from labeling data for ML [73, 74] to serving as a platform for experimental research [11, 47, 53]).”
BackgroundConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 7×
“Existing guidelines and best practices focus on how to run experiments in crowdsourcing environments successfully, proposing strategies to overcome common pitfalls found in crowdsourcing platforms [9, 38, 46, 64].”
BackgroundMethodInfluentialModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“…evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 29, 49, 55].”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 3×
“Examples of this can be found in evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 2…”
BackgroundCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 5×
“This simple technique can help to deal with malicious workers and avoid wasting collected contributions [38].”
BackgroundInfluentialCheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887 · cited 1×
“, from labeling data for ML [73, 74] to serving as a platform for experimental research [11, 47, 53]).”
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Trials · doi:10.1186/1745-6215-11-32 · Cited by 1,871
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“A typical strategy is to use majority voting; however, more sophisticated and effective alternatives have been developed to derive the right answer even when the majority may be wrong (e.g., [12, 15, 78]).”
BackgroundEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · cited 2×
“Crowdsourcing platforms are being widely adopted as an environment to run experiments with human subjects [11, 38, 47, 53].”
BackgroundThe Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,530
Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
Whitehill, J., Wu, T., Bergsma, J., Movellan, J. R., & Ruvolo, P. (2009) · neural information processing systems · Cited by 1,364
Identifying and Avoiding Bias in Research
Pannucci, C. J., & Wilkins, E. G. (2010) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e3181de24bc · Cited by 1,079
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · cited 2×
“Proper compensation can become a more frequent practice due to current features (like qualifications or badges) in crowdsourcing environments, advances in techniques to safeguard quality, plus underpayment issues being a reiterating topic addressed by recent literature [5, 33, 39, 79].”
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · cited 3×
“Existing guidelines and best practices focus on how to run experiments in crowdsourcing environments successfully, proposing strategies to overcome common pitfalls found in crowdsourcing platforms [9, 38, 46, 64].”
BackgroundBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 1×
“Examples of this can be found in evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 2…”
BackgroundUnderstanding the Hawthorne effect
Sedgwick, P., & Greenwood, N. (2015) · BMJ · doi:10.1136/bmj.h4672 · Cited by 824
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767 · cited 1×
“Surprisingly, guidelines for reporting crowdsourcing experiments have received little attention even though underreporting is a serious concern in science, and crowdsourcing is no exception [7].”
BackgroundRand, D. G. (2012) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 683
Utility data annotation with Amazon Mechanical Turk
Sorokin, A., & Forsyth, D. (2008) · 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4562953 · Cited by 616 · cited 1×
“Research has shown how crowdsourcing could be leveraged in a wide range of tasks and domains (e.g., from labeling data for ML [73, 74] to serving as a platform for experimental research [11, 47, 53]).”
BackgroundFinancial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600175 · Cited by 520
Demographics and Dynamics of Mechanical Turk Workers
Difallah, D., Filatova, E., & Ipeirotis, P. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159661 · Cited by 501 · cited 5×
“6% did not report over what timeframe the experiment run, which is known to determine the characteristics of the active pool of workers [19].”
BackgroundInfluentialA Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174023 · Cited by 497 · cited 1×
“Proper compensation can become a more frequent practice due to current features (like qualifications or badges) in crowdsourcing environments, advances in techniques to safeguard quality, plus under-payment issues being a reiterating topic addressed by recent literature [5, 33, 39, 79].”
BackgroundStatistics As Principled Argument
Abelson, R. P. (2012) · Psychology Press eBooks · doi:10.4324/9781410601155 · Cited by 465
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
Reproducibility vs. Replicability: A Brief History of a Confused Terminology
Plesser, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 424 · cited 1×
“Efforts in this regard have been branded under the definitions of repeatability , replicability and reproducibility , to denote attempts at obtaining similar results, by the same or different teams and experimental conditions, under acceptable margins of error [56].”
ResultShepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334 · cited 3×
“Other strategies involve imposing quality by design, such as adding feedback loops (where workers self-assess or receive external feedback) [22], or making workers collaborate [10, 16, 67].”
BackgroundDaniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2019) · ACM Computing Surveys · doi:10.1145/3148148 · Cited by 313
Self-citationAn Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets
Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., & Vukovic, M. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14105 · Cited by 299
Martin, D., Hanrahan, B. V., O'Neill, J., & Gupta, N. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531663 · Cited by 278 · cited 1×
“The community is increasingly voicing the need for ● ethical approvals for experiments run in crowdsourcing environments [31, 45].”
BackgroundOlson, J. S., & Kellogg, W. A. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8 · Cited by 273 · cited 1×
“These are well-understood aspects covered in experimental research guidelines and textbooks on research methods (e.g., [30, 51]).”
BackgroundMcCambridge, J., de Bruin, M., & Witton, J. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039116 · Cited by 271
An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets.
Rogstadius, J., Kostakos, V., Kittur, A., et al. (2011) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.13140/rg.2.2.19170.94401 · Cited by 268 · cited 4×
“Existing guidelines and best practices focus on how to run experiments in crowdsourcing environments successfully, proposing strategies to overcome common pitfalls found in crowdsourcing platforms [9, 38, 46, 64].”
BackgroundInfluentialPineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232 · cited 2×
“…evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 29, 49, 55].”
BackgroundMaking Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research
Vaughan, J. W. (2018) · Journal of Machine Learning Research · Cited by 227
Working The Crowd: Employment And Labor Law In The Crowdsourcing Industry
Felstiner, A. (2011) · bepress Legal Repository · doi:10.15779/z38z92x · Cited by 188
Incentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146 · cited 6×
“shown to impact the number of contributions produced by workers [47], the quality of their work [18], and being a relevant aspect from an ethical perspective.”
BackgroundInfluentialA taxonomy of microtasks on the web
Gadiraju, U., Kawase, R., & Dietze, S. (2014) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/2631775.2631819 · Cited by 143 · cited 1×
“The nature and goal of the experiment shape the kind of tasks that are sent down to workers in crowdsourcing platforms, these aspects have been considered by previous research to propose a categorization of micro-tasks [26]; we included in the taxonomy the ● task type attribute to capture this information.”
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129 · cited 1×
“For example, crowdsourcing is helping researchers evaluate the impact of different interface designs on user performance, comprehension and understanding [14, 60, 75], assess the difference in performance between users with different expertise, background and even mood levels [36, 81, 82].”
BackgroundRehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning
Barbosa, N. M., & Chen, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300773 · Cited by 118 · cited 3×
“as underpayments in crowdsourcing (see [5] for a brief overview).”
BackgroundTransparency of CHI Research Artifacts: Results of a Self-Reported Survey
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376448 · Cited by 118 · cited 1×
“Reproducibility of experiments is essential in science [77].”
BackgroundFair Work: Crowd Work Minimum Wage with One Line of Code
Whiting, M. E., Hugh, G., & Bernstein, M. S. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5283 · Cited by 115 · cited 3×
“One aspect of this regards to accurate and fair task pricing [79], which previous work has shown to impact the number of contributions produced by workers [47], the quality of their work [18], and being a relevant aspect from an ethical perspective.”
BackgroundMicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy
Drapeau, R., Chilton, L. B., Bragg, J., & Weld, D. S. (2016) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v4i1.13270 · Cited by 106 · cited 1×
“Other strategies involve imposing quality by design, such as adding feedback loops (where workers self-assess or receive external feedback) [22], or making workers collaborate [10, 16, 67].”
BackgroundCrowdsourcing: A taxonomy and systematic mapping study
Hosseini, M., Shahri, A., Phalp, K. T., Taylor, J., & Ali, R. (2015) · Computer Science Review · doi:10.1016/j.cosrev.2015.05.001 · Cited by 94 · cited 1×
“For this, we took a small seed of well-known guidelines for experimental design and crowdsourcing [27, 30, 35, 46, 57] and expanded it through snowballing and keywords search using Google Scholar (“crowdsourcing + guidelines”, “crowdsourcing + best practices”, “crowdsourcing + recommendations”, and “crowdsourcing + re…”
MethodExperiences surveying the crowd: reflections on methods, participation, and reliability
Marshall, C. C., & Shipman, F. M., III (2013) · ACM Web Science Conference (WebSci) · doi:10.1145/2464464.2464485 · Cited by 89 · cited 1×
“example, removing contributions that do not fall above a minimum time or agreement threshold [32, 44].”
BackgroundEmbracing Error to Enable Rapid Crowdsourcing
Krishna, R., Hata, K., Chen, S. P., Kravitz, J., Shamma, D. A., Fei-Fei, L., & Bernstein, M. S. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858115 · Cited by 87 · cited 1×
“allotted workers to spend on the task naturally limits the associated cost of the experiment, besides studies in cost-effective crowdsourcing shed light on the impact of time on worker behavior and performance [41, 42].”
BackgroundClarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing
Gadiraju, U., Yang, J., & Bozzon, A. (2017) · Proceedings of the 28th ACM Conference on Hypertext and Social Media · doi:10.1145/3078714.3078715 · Cited by 87 · cited 1×
“Poorly written task instructions could misguide workers and result in low-quality work [17, 28, 39, 80].”
BackgroundCrowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work?
Wilson, S., Schaub, F., Ramanath, R., Sadeh, N., Liu, F., Smith, N. A., & Liu, F. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883035 · Cited by 80 · cited 2×
“In contrast, enhanced interfaces may facilitate the job of crowd workers and aid these to improved performance [59, 60, 65, 72], as well as adequately limiting the time to judge can accelerate task completion without compromising the quality of the results [42].”
BackgroundResolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work
Schaekermann, M., Goh, J., Larson, K. S., & Law, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274423 · Cited by 77
An exploration of crowdsourcing citation screening for systematic reviews
Mortensen, M. L., Adam, G. P., Trikalinos, T. A., Kraska, T., & Wallace, B. C. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1252 · Cited by 76 · cited 2×
“For example, requesters may not need to interact with workers to study task design in the context of classification tasks, relying entirely on leveraging clear guidelines to articulate what they expect from workers [50, 60].”
BackgroundEffective Crowd Annotation for Relation Extraction
Liu, A., Soderland, S., Bragg, J., Lin, C. H., Ling, X., & Weld, D. S. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1104 · Cited by 68 · cited 2×
“● Training protocols can be used to prepare workers for the job, which can have a positive impact on their resulting performance [17].”
BackgroundAll Those Wasted Hours: On Task Abandonment in Crowdsourcing
Han, L., Roitero, K., Gadiraju, U., Sarasua, C., Checco, A., Maddalena, E., & Demartini, G. (2019) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3289600.3291035 · Cited by 68 · cited 3×
“Besides, it can also reduce task intake and negatively affect how long the experiment takes to finish [21].”
BackgroundSelf-citationGergle, D., & Tan, D. S. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8_9 · Cited by 64 · cited 2×
“The soundness of the data analysis process and how to articulate the findings aid to derive the right conclusions from experiments [30].”
BackgroundMethodScaling-Up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement
Difallah, D., Catasta, M., Demartini, G., & Cudré-Mauroux, P. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13154 · Cited by 64 · cited 1×
“Different payment strategies could also lead to effects on how workers perform [13].”
BackgroundSelf-citationConfusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk
Wu, M., & Quinn, A. J. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13317 · Cited by 64 · cited 4×
“, instructions [80], compensation [18], task interface [65], and time [42] among others), external factors such as the workers’ environment [24], platforms [58], and population [19] — all of which serve as the foundation for running user studies.”
BackgroundInfluentialModus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
Gadiraju, U., Checco, A., Gupta, N., & Demartini, G. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3130914 · Cited by 55
Self-citationData Fusion: Resolving Conflicts from Multiple Sources
Dong, X. L., Berti‐Équille, L., & Srivastava, D. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-36257-6_13 · Cited by 53
Cognitively inspired task design to improve user performance on crowdsourcing platforms
Sampath, H. A., Rajeshuni, R., & Indurkhya, B. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557155 · Cited by 53 · cited 4×
“, instructions [80], compensation [18], task interface [65], and time [42] among others), external factors such as the workers’ environment [24], platforms [58], and population [19] — all of which serve as the foundation for running user studies.”
BackgroundInfluentialHansen, D. L., Schone, P., Corey, D. L., Reid, M., & Gehring, J. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441848 · Cited by 51 · cited 1×
“example, removing contributions that do not fall above a minimum time or agreement threshold [32, 44].”
BackgroundGetting More for Less: Optimized Crowdsourcing with Dynamic Tasks and Goals
Kobren, A., Tan, C. H., Ipeirotis, P. G., & Gabrilovich, E. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741681 · Cited by 48 · cited 1×
“Therefore, the taxonomy captures what ● dropouts prevention mechanisms were used to deal with the prevalence of task abandonment in crowdsourcing [21, 40].”
BackgroundImproving learning through achievement priming in crowdsourced information finding microtasks
Gadiraju, U., & Dietze, S. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3027402 · Cited by 45 · cited 1×
“Tasks can also include elements that target intrinsic motivational factors, for example, to aid how workers engage and commit [25] (although one could argue that intrinsic factors are effects we may want to reduce in an experimental setting unless it is the subject of study).”
BackgroundTurkServer: Enabling Synchronous and Longitudinal Online Experiments
Mao, A., Parkes, D., Chen, Y., Procaccia, A. D., Gajos, K. Z., & Zhang, H. (2012) · Cited by 41
Internet-based crowdsourcing and research ethics: the case for IRB review
Graber, M. A., & Graber, A. D. (2012) · Journal of Medical Ethics · doi:10.1136/medethics-2012-100798 · Cited by 37
Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge
Maddalena, E., Basaldella, M., Nart, D. D., Degl’Innocenti, D., Mizzaro, S., & Demartini, G. (2016) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v4i1.13284 · Cited by 36 · cited 4×
“, instructions [80], compensation [18], task interface [65], and time [42] among others), external factors such as the workers’ environment [24], platforms [58], and population [19] — all of which serve as the foundation for running user studies.”
InfluentialSelf-citationHuman Computation Must Be Reproducible
Paritosh, P. K. (2012) · Cited by 33
Porter, N. D., Verdery, A. M., & Gaddis, S. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233154 · Cited by 30 · cited 6×
“Initially, crowdsourcing platforms such as Amazon Mechanical Turk were deemed a “marked for lemons” [37], an environment restraining committed workers from earning at least a legal minimum wage due to the prevalence of less-committed or malicious workers.”
BackgroundMethodInfluentialNarratives in Crowdsourced Evaluation of Visualizations: A Double-Edged Sword?
Dimara, E., Bezerianos, A., & Dragicevic, P. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025870 · Cited by 25 · cited 1×
“For example, crowdsourcing is helping researchers evaluate the impact of different interface designs on user performance, comprehension and understanding [14, 60, 75], assess the difference in performance between users with different expertise, background and even mood levels [36, 81, 82].”
BackgroundUnderstanding the Impact of Text Highlighting in Crowdsourcing Tasks
Ramírez, J., Báez, M., Casati, F., & Benatallah, B. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5268 · Cited by 25
Self-citationSupporting the Modern Polyglot: A Comparison of Multilingual Search Interfaces
Steichen, B., & Freund, L. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702541 · Cited by 20 · cited 1×
“For example, crowdsourcing is helping researchers evaluate the impact of different interface designs on user performance, comprehension and understanding [14, 60, 75], assess the difference in performance between users with different expertise, background and even mood levels [36, 81, 82].”
BackgroundRevealing the Role of User Moods in Struggling Search Tasks
Xu, L., Zhou, X., & Gadiraju, U. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331353 · Cited by 19
Novices Who Focused or Experts Who Didn't?
Wu, Y., & Bailey, B. P. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858330 · Cited by 17 · cited 1×
“For example, crowdsourcing is helping researchers evaluate the impact of different interface designs on user performance, comprehension and understanding [14, 60, 75], assess the difference in performance between users with different expertise, background and even mood levels [36, 81, 82].”
BackgroundPlatform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks
Qarout, R., Checco, A., Demartini, G., & Bontcheva, K. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5264 · Cited by 15 · cited 6×
“Thus, failing to account for (and therefore report on) this aspect may add confounding factors to the experiment that hurt its reproducibility [58].”
BackgroundInfluentialSelf-citationCicero: Multi-Turn, Contextual Argumentation for Accurate CrowdsourcingPreprint
Chen, Q. Z., Bragg, J., Chilton, L. B., & Weld, D. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10733 · Cited by 14
Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
Ramírez, J., Báez, M., Casati, F., & Benatallah, B. (2019) · BMC Research Notes · doi:10.1186/s13104-019-4858-z · Cited by 11
Self-citationRamírez, J., Báez, M., Casati, F., Cernuzzi, L., & Benatallah, B. (2020) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3406865.3418318 · Cited by 8 · cited 3×
“However, these works are so far focused on specific use cases and platforms (data collection for social sciences in Amazon Mechanical Turk [57]), or are still work-in-progress [61].”
BackgroundMethodSelf-citationUnderstanding the Impact of Text Highlighting in Crowdsourcing Tasks
Ramírez, J., Báez, M., Casati, F., et al. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.48550/arxiv.1909.02780 · Cited by 7 · cited 4×
“For example, requesters may not need to interact with workers to study task design in the context of classification tasks, relying entirely on leveraging clear guidelines to articulate what they expect from workers [50, 60].”
BackgroundInfluentialSelf-citationCrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designsPreprint
Ramírez, J., Degiacomi, S., Zanella, D., Báez, M., Casati, F., & Benatallah, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02800 · Cited by 4 · cited 1×
“Depending on the experimental design and how it was handled, this situation may be desirable or introduce unintended biases affecting the integrity of the experimental conditions and, ultimately, the experiment results [53, 62].”
BackgroundSelf-citationRevealing the Role of User Moods in Struggling Search TasksPreprint
Xu, L., Zhou, X., & Gadiraju, U. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07717 · Cited by 3
Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd.
Gadiraju, U., Möller, S., Nöllenburg, M., Saupe, D., Egger-Lampl, S., Archambault, D., & Fisher, B. (2015) · Cited by 2
Research works printed but not linked: 8
- Kitchenham Barbara and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. 2 (01 2007).
- 10.3758/s13428-013-0365--7 · doi:10.3758/s13428-013-0365--7
- 10.1007/978--3--642--36257--6 · doi:10.1007/978--3--642--36257--6
- 10.1007/978--3--319--66435--4_2 · doi:10.1007/978--3--319--66435--4_2
- 10.1007/978--1--4939-0378--8_9 · doi:10.1007/978--1--4939-0378--8_9
- 10.1136/medethics-2012--100798 · doi:10.1136/medethics-2012--100798
- 10.1007/s10683-011--9273--9 · doi:10.1007/s10683-011--9273--9
- 10.3758/s13428-011-0124--6 · doi:10.3758/s13428-011-0124--6
Cited by
A Checklist to Combat Cognitive Biases in Crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“This includes datasheets [23, 35], data statements [9] and checklists [58].”
BackgroundMethodInfluentialDíaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
“…complements and extends dataset development and documentation frameworks that have previously been developed in service oftransparency, accountability, and reproducibility [6, 9, 17, 26, 27, 40, 42], but focuses specificallyon uniqueconsiderations relating tocrowdsourceddata-set annotation.”
Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
Correia, A., Grover, A., Schneider, D., Pimentel, A. P., Chaves, R., de Almeida, M. A., & Fonseca, B. (2023) · Applied Sciences · doi:10.3390/app13042198 · Cited by 34
Franzén, D., von Voigt, S. N., Sörries, P., Tschorsch, F., & Müller-Birn, C. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560693 · Cited by 37
Metadata
- Authors
- Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G.
- Affiliations
- University of Trento (IT)
- Université Claude Bernard Lyon 1, Villeurbanne, France
- ServiceNow (United States) (US)
- Universidad Católica Nuestra Señora de la Asunción (PY)
- UNSW Sydney (AU)
- The University of Queensland (AU)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-34
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3184384643
- Cited by
- 4
- References
- 83
- Citations by year
- 2023: 2 · 2022: 1 · 2021: 1
- FWCI
- 0.85
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University)
- Funders
- Russian Science Foundation (Project No. 19-18-00282)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reporting guidelines, reporting checklist, reproducibility, microtasks, quality control, human subjects research, crowdsourcing platforms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.