Efficient Elicitation Approaches to Estimate Collective Crowd Answers
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
When crowdsourcing the creation of machine learning datasets, statistical distributions that capture diverse answers can represent ambiguous data better than a single best answer. Unfortunately, collecting distributions is expensive because a large number of responses need to be collected to form a stable distribution. Despite this, the efficient collection of answer distributions-that is, ways to use less human effort to collect estimates of the eventual distribution that would be formed by a large group of responses-is an under-studied topic. In this paper, we demonstrate that this type of estimation is possible and characterize different elicitation approaches to guide the development of future systems. We investigate eight elicitation approaches along two dimensions: annotation granularity and estimation perspective. Annotation granularity is varied by annotating i) a single "best" label, ii) all relevant labels, iii) a ranking of all relevant labels, or iv) real-valued weights for all relevant labels. Estimation perspective is varied by prompting workers to either respond with their own answer or an estimate of the answer(s) that they expect other workers would provide. Our study collected ordinal annotations on the emotional valence of facial images from 1,960 crowd workers and found that, surprisingly, the most fine-grained elicitation methods were not the most accurate, despite workers spending more time to provide answers. Instead, the most efficient approach was to ask workers to choose all relevant classes that others would have selected. This resulted in a 21.4% reduction in the human time required to reach the same performance as the baseline (i.e., selecting a single answer with their own perspective). By analyzing cases in which finer-grained annotations degraded performance, we contribute to a better understanding of the trade-offs between answer elicitation approaches. Our work makes it more tractable to use answer distributions in large-scale tasks such as ML training, and aims to spark future work on techniques that can efficiently estimate answer distributions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
On the universality and cultural specificity of emotion recognition: a meta-analysis
Elfenbein, H. A., & Ambady, N. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.2.203 · Cited by 2,049 · cited 2×
“Previous work in emotion perception found that interpreting emotion is a universal human ability [21], therefore, for different annotator-image race pairs, we expect estimation-based approaches to still work.”
BackgroundCheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Jonas, E., Schulz-Hardt, S., Frey, D., & Thelen, N. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.80.4.557 · Cited by 725 · cited 2×
“However, this can affect the results due to confirmation bias from the sequential interpretation [45] (also explained in Section 6.”
BackgroundA Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · cited 5×
“Bayesian Truth Serum [65, 66], which inspired the estimation of other people’s answers, was also more reliable with a higher number of people.”
BackgroundMethodResultInfluentialComparing Rating Scales of Different Lengths: Equivalence of Scores from 5-Point and 7-Point Scales
Colman, A. M., Norris, C. E., & Preston, C. C. (1997) · Psychological Reports · doi:10.2466/pr0.1997.80.2.355 · Cited by 318
A solution to the single-question crowd wisdom problem
Prelec, D., Seung, H. S., & McCoy, J. (2017) · Nature · doi:10.1038/nature21054 · Cited by 314
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · cited 1×
“For instance, in previous work, with a lower number of tokens, people were more accurate at understanding the probability [48].”
ResultProviding Timely Examples Improves the Quantity and Quality of Generated Ideas
Siangliulue, P., Chan, J., Gajos, K. Z., & Dow, S. P. (2015) · Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition · doi:10.1145/2757226.2757230 · Cited by 99 · cited 1×
“[69] exposed crowds to a more diverse set of examples and showed that the timely exposure of examples can help crowds generate more ideas [70].”
BackgroundLinguistically debatable or just plain wrong?
Plank, B., Hovy, D., & Søgaard, A. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2083 · Cited by 86 · cited 2×
“[64] found that even expert annotators disagreed in part-of-speech tagging and claimed that diverse answers can be due to the ambiguity of the data.”
Background
Cited by
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
Self-citationYes: Affirmative Consent as a Theoretical Framework for Understanding and Imagining Social Platforms
Im, J., Dimond, J., Berton, M., Lee, U., Mustelier, K., Ackerman, M. S., & Gilbert, E. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445778 · Cited by 119
Ayobi, A., Marshall, P., & Cox, A. L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376809 · Cited by 110
The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality
Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., & Bernstein, M. S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445423 · Cited by 101
Crowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84
Metadata
- Authors
- Chung, J. J. Y., Song, J. Y., Kutty, S., Hong, S. (., Kim, J., & Lasecki, W. S.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2019-11-07
- Volume
- 3
- Issue
- CSCW
- Pages
- 1-25
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2987755095
- Cited by
- 46
- References
- 15
- FWCI
- 7.35
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.