Learning from crowdsourced labeled data: a survey
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Aggregation methods have been widely studied in crowdsourcing and ensemble learning.
“Aggregation methods have been widely studied in fields such as crowdsourcing and ensemble learning, where strategies are developed to combine predictions (also referred to as worker predictions) to enhance decision quality during training and inference.”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zhang et al., 2016
Most existing aggregation methods focus on weighting worker contributions without accounting for contextual information, though other methods introduce context-aware mechanisms.
“Most existing aggregation methods focus on weighting worker contributions without accounting for contextual information (Zhang et al., 2016;Zheng et al., 2017;Zhang, 2022)), though other methods introduce context-aware mechanisms (Jin et al., 2020;Zhang, 2022).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zhang et al., 2016
Cites
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781 · see citing works
Losses over Labels: Weakly Supervised Learning via Direct Loss Construction
Sam, D., & Kolter, J. Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26159 · Cited by 6 · see citing works
SkillAggregation: Reference-free LLM-Dependent Aggregation
Sun, G., Kagrecha, A., Manakul, P., Woodland, P., & Gales, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.756 · Cited by 3 · see citing works
- Authors
- Zhang, J., Wu, X., & Sheng, V. S.
- Affiliations
- Nanjing University of Science and Technology (CN)
- Hefei University of Technology (CN)
- Nanjing University of Information Science and Technology (CN)
- Corresponding author
- Zhang, Jing
- Venue
- Artificial Intelligence Review
- Published
- 2016-07-02
- Volume
- 46
- Issue
- 4
- Pages
- 543-576
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W2475282416
- Cited by
- 203
- References
- 3
- Citations by year
- 2026: 12 · 2025: 30 · 2024: 16 · 2023: 16 · 2022: 33 · 2021: 33 · 2020: 19 · 2019: 19 · 2018: 20 · 2017: 5
- FWCI
- 17.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Artificial Intelligence Review (publishedVersion)
- Funders
- Division of Information and Intelligent Systems (1115417)
- Changjiang Scholar Program of Chinese Ministry of Education (IRT13059)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- crowdsourcing, label quality, supervised learning, open-source library
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
