On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Abstract
Crowdsourcing is being increasingly adopted as a platform to run studies with human subjects. Running a crowdsourcing experiment involves several choices and strategies to successfully port an experimental design into an otherwise uncontrolled research environment, e.g., sampling crowd workers, mapping experimental conditions to micro-tasks, or ensure quality contributions. While several guidelines inform researchers in these choices, guidance of how and what to report from crowdsourcing experiments has been largely overlooked. If under-reported, implementation choices constitute variability sources that can affect the experiment's reproducibility and prevent a fair assessment of research outcomes. In this paper, we examine the current state of reporting of crowdsourcing experiments and offer guidance to address associated reporting issues. We start by identifying sensible implementation choices, relying on existing literature and interviews with experts, to then extensively analyze the reporting of 171 crowdsourcing experiments. Informed by this process, we propose a checklist for reporting crowdsourcing experiments.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
(2002) · Cited by 13,211
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · cited 5×
“The first example was the PRISMA checklist [71] that assesses the methodological rigor of systematic reviews.”
BackgroundMethodInfluentialSaturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033 · cited 1×
“From this list, we took a random sample of 15 papers published in the last eight years, which we drafted incrementally until we reached saturation [66].”
MethodThe American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 3×
“Research has shown how crowdsourcing could be leveraged in a wide range of tasks and domains (e.g., from labeling data for ML [73, 74] to serving as a platform for experimental research [11, 47, 53]).”
BackgroundConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 7×
“Existing guidelines and best practices focus on how to run experiments in crowdsourcing environments successfully, proposing strategies to overcome common pitfalls found in crowdsourcing platforms [9, 38, 46, 64].”
BackgroundMethodInfluentialModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“…evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 29, 49, 55].”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 3×
“Examples of this can be found in evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 2…”
BackgroundCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 5×
“This simple technique can help to deal with malicious workers and avoid wasting collected contributions [38].”
BackgroundInfluential
Research works printed but not linked: 8
- Kitchenham Barbara and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. 2 (01 2007).
- 10.3758/s13428-013-0365--7 · doi:10.3758/s13428-013-0365--7
- 10.1007/978--3--642--36257--6 · doi:10.1007/978--3--642--36257--6
- 10.1007/978--3--319--66435--4_2 · doi:10.1007/978--3--319--66435--4_2
- 10.1007/978--1--4939-0378--8_9 · doi:10.1007/978--1--4939-0378--8_9
- 10.1136/medethics-2012--100798 · doi:10.1136/medethics-2012--100798
- 10.1007/s10683-011--9273--9 · doi:10.1007/s10683-011--9273--9
- 10.3758/s13428-011-0124--6 · doi:10.3758/s13428-011-0124--6
Cited by
A Checklist to Combat Cognitive Biases in Crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“This includes datasheets [23, 35], data statements [9] and checklists [58].”
BackgroundMethodInfluentialDíaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
“…complements and extends dataset development and documentation frameworks that have previously been developed in service oftransparency, accountability, and reproducibility [6, 9, 17, 26, 27, 40, 42], but focuses specificallyon uniqueconsiderations relating tocrowdsourceddata-set annotation.”
Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
Correia, A., Grover, A., Schneider, D., Pimentel, A. P., Chaves, R., de Almeida, M. A., & Fonseca, B. (2023) · Applied Sciences · doi:10.3390/app13042198 · Cited by 34
Franzén, D., von Voigt, S. N., Sörries, P., Tschorsch, F., & Müller-Birn, C. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560693 · Cited by 37
Metadata
- Authors
- Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G.
- Affiliations
- University of Trento (IT)
- Université Claude Bernard Lyon 1, Villeurbanne, France
- ServiceNow (United States) (US)
- Universidad Católica Nuestra Señora de la Asunción (PY)
- UNSW Sydney (AU)
- The University of Queensland (AU)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-34
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3184384643
- Cited by
- 4
- References
- 83
- Citations by year
- 2023: 2 · 2022: 1 · 2021: 1
- FWCI
- 0.85
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University)
- Funders
- Russian Science Foundation (Project No. 19-18-00282)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reporting guidelines, reporting checklist, reproducibility, microtasks, quality control, human subjects research, crowdsourcing platforms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices. https://app.cochat.ai/papers/10.1145/3479531
- The paper
- Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021). On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3479531
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.