Quizz: targeted crowdsourcing with a billion (potential) users
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Toward a Theory of Intrinsically Motivating Instruction*
Malone, T. W. (1981) · Cognitive Science · doi:10.1207/s15516709cog0504_2 · Cited by 2,197 · cited 2×
“Based on theories of intrinsic motivation [25], we imple-”
BackgroundPay Enough or Don't Pay at All*
Gneezy, U., & Rustichini, A. (2000) · Quarterly Journal of Economics · doi:10.1162/003355300554917 · Cited by 2,175 · cited 1×
“Furthermore, studies indicate that the use of monetary rewards can be highly detrimental for users who are already intrinsically motivated [4]: the introduction of monetary compensation reveals to the users exactly how much their work is valued by the task requester, and low payments make things worse [16].”
BackgroundMaximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 2×
“[8] Dawid, A. P., and Skene, A. M. Maximum likelihood estimation of observer error-rates using the EM algorithm.”
Background
Cited by
The rise of motivational information systems: A review of gamification research
Koivisto, J., & Hamari, J. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.10.013 · Cited by 1,750
Why do people use gamification services?
Hamari, J., & Koivisto, J. (2015) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2015.04.006 · Cited by 642
“Working out for likes”: An empirical study on social influence in exercise gamification
Hamari, J., & Koivisto, J. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.04.018 · Cited by 473
“…& Sung, 2013; Lounis, Pramatari, & Theotokis, 2014), marketing and advertising (Cechanowicz, Gutwin, Brownell, & Goodfellow, 2013; Terlutter & Capella, 2013), and activities such as crowdsourcing (Eickhoff, Harris, de Vries, & Srinivasan, 2012; Ipeirotis & Gabrilovich, 2014), to name a few.”
Gamified crowdsourcing: Conceptualization, literature review, and future agenda
Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.04.005 · Cited by 297
Empirical Support for a Causal Relationship Between Gamification and Learning Outcomes
Denny, P., McDonald, F., Empson, R., Kelly, P., & Petersen, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173885 · Cited by 135
Metadata
- Authors
- Ipeirotis, P. G., & Gabrilovich, E.
- Venue
- Proceedings of the 23rd international conference on World wide web
- Publisher
- ACM
- Published
- 2014-04-07
- Pages
- 143-154
- OpenAlex
- W2064677871
- Cited by
- 152
- References
- 4
- FWCI
- 12.73
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Quizz: targeted crowdsourcing with a billion (potential) users. https://app.cochat.ai/papers/10.1145/2566486.2567988
- The paper
- Ipeirotis, P. G., & Gabrilovich, E. (2014). Quizz: targeted crowdsourcing with a billion (potential) users. Proceedings of the 23rd international conference on World wide web. https://doi.org/10.1145/2566486.2567988
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.