The DOI matches its Crossref record.
Quality through flow and immersion: gamifying crowdsourced relevance assessments
Abstract
Crowdsourcing is a market of steadily-growing importance upon which both academia and industry increasingly rely. However, this market appears to be inherently infested with a significant share of malicious workers who try to maximise their profits through cheating or sloppiness. This serves to undermine the very merits crowdsourcing has come to represent. Based on previous experience as well as psychological insights, we propose the use of a game in order to attract and retain a larger share of reliable workers to frequently-requested crowdsourcing tasks such as relevance assessments and clustering. In a large-scale comparative study conducted using recent TREC data, we investigate the performance of traditional HIT designs and a game-based alternative that is able to achieve high quality at significantly lower pay rates, facing fewer malicious submissions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification
Hamari, J., Koivisto, J., & Sarsa, H. (2014) · 2014 47th Hawaii International Conference on System Sciences · doi:10.1109/hicss.2014.377 · Cited by 5,201
Gamification in theory and action: a survey
Seaborn, K., & Fels, D. I. (2015) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2014.09.006 · Cited by 2,515
The rise of motivational information systems: A review of gamification research
Koivisto, J., & Hamari, J. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.10.013 · Cited by 1,750
Gamification of task performance with leaderboards: A goal setting experiment
Landers, R. N., Bauer, K. N., & Callan, R. C. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.008 · Cited by 474
“Working out for likes”: An empirical study on social influence in exercise gamification
Hamari, J., & Koivisto, J. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.04.018 · Cited by 473
Metadata
- Authors
- Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P.
- Venue
- Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2012-08-12
- Pages
- 871-880
- OpenAlex
- W2097246133
- Cited by
- 190
- References
- 0
- FWCI
- 13.05
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Quality through flow and immersion: gamifying crowdsourced relevance assessments. https://app.cochat.ai/papers/10.1145/2348283.2348400
- The paper
- Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012). Quality through flow and immersion: gamifying crowdsourced relevance assessments. Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. https://doi.org/10.1145/2348283.2348400
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.