CoChat

On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices

Open access
Ramírez, Jorge, Sayin, Burcu, Baez, Marcos, Casati, Fabio, Cernuzzi, Luca, Benatallah, Boualem, Demartini, Gianluca · 2021 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.

Abstract

Crowdsourcing is being increasingly adopted as a platform to run studies with human subjects. Running a crowdsourcing experiment involves several choices and strategies to successfully port an experimental design into an otherwise uncontrolled research environment, e.g., sampling crowd workers, mapping experimental conditions to micro-tasks, or ensure quality contributions. While several guidelines inform researchers in these choices, guidance of how and what to report from crowdsourcing experiments has been largely overlooked. If under-reported, implementation choices constitute variability sources that can affect the experiment's reproducibility and prevent a fair assessment of research outcomes. In this paper, we examine the current state of reporting of crowdsourcing experiments and offer guidance to address associated reporting issues. We start by identifying sensible implementation choices, relying on existing literature and interviews with experts, to then extensively analyze the reporting of 171 crowdsourcing experiments. Informed by this process, we propose a checklist for reporting crowdsourcing experiments.

Overview

Integrity signals

Registry

Crossref · 10.1145/3479531

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 85 references

Research works printed but not linked: 8

  1. Kitchenham Barbara and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. 2 (01 2007).
  2. 10.3758/s13428-013-0365--7 · doi:10.3758/s13428-013-0365--7
  3. 10.1007/978--3--642--36257--6 · doi:10.1007/978--3--642--36257--6
  4. 10.1007/978--3--319--66435--4_2 · doi:10.1007/978--3--319--66435--4_2
  5. 10.1007/978--1--4939-0378--8_9 · doi:10.1007/978--1--4939-0378--8_9
  6. 10.1136/medethics-2012--100798 · doi:10.1136/medethics-2012--100798
  7. 10.1007/s10683-011--9273--9 · doi:10.1007/s10683-011--9273--9
  8. 10.3758/s13428-011-0124--6 · doi:10.3758/s13428-011-0124--6

Cited by

Metadata

Authors
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G.
Affiliations
  • University of Trento (IT)
  • Université Claude Bernard Lyon 1, Villeurbanne, France
  • ServiceNow (United States) (US)
  • Universidad Católica Nuestra Señora de la Asunción (PY)
  • UNSW Sydney (AU)
  • The University of Queensland (AU)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2021-10-13
Volume
5
Issue
CSCW2
Pages
1-34
ISSN
2573-0142 (ISSN-L)
OpenAlex
W3184384643
Cited by
4
References
83
Citations by year
2023: 2 · 2022: 1 · 2021: 1
FWCI
0.85
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
  • arXiv (Cornell University)
Funders
Russian Science Foundation (Project No. 19-18-00282)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
Keywords
reporting guidelines, reporting checklist, reproducibility, microtasks, quality control, human subjects research, crowdsourcing platforms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.