Community-based bayesian aggregation models for crowdsourcing
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
This paper addresses the problem of extracting accurate labels from crowdsourced datasets, a key challenge in crowdsourcing. Prior work has focused on modeling the reliability of individual workers, for instance, by way of confusion matrices, and using these latent traits to estimate the true labels more accurately. However, this strategy becomes ineffective when there are too few labels per worker to reliably estimate their quality. To mitigate this issue, we propose a novel community-based Bayesian label aggregation model, CommunityBCC, which assumes that crowd workers conform to a few different types, where each type represents a group of workers with similar confusion matrices. We assume that each worker belongs to a certain community, where the worker's confusion matrix is similar to (a perturbation of) the community's confusion matrix. Our model can then learn a set of key latent features: (i) the confusion matrix of each community, (ii) the community membership of each user, and (iii) the aggregated label of each item. We compare the performance of our model against established aggregation methods on a number of large-scale, real-world crowdsourcing datasets. Our experimental results show that our CommunityBCC model consistently outperforms state-of-the-art label aggregation methods, requiring, on average, 50% less data to pass the 90% accuracy mark.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692
(2005) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5555/1046920.1088695 · Cited by 650
Cited by
AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528120 · Cited by 606
“However, this problem can also be related to the community of the crowd itself [37].”
BackgroundTruth inference in crowdsourcing: is the problem solved?
Zheng, Y., Li, G., Li, Y., Shan, C., & Cheng, R. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3055540.3055547 · Cited by 421
Spatial crowdsourcing: a survey
Tong, Y., Zhou, Z., Zeng, Y., Chen, L., & Shahabi, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00568-7 · Cited by 305
Crowdsourced Data Management: A Survey
Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2535242 · Cited by 283
FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation
Ma, F., Li, Y., Li, Q., Qiu, M., Gao, J., Zhi, S., Su, L., Zhao, B., Ji, H., & Han, J. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783314 · Cited by 186
Trustworthy human-AI partnerships
Ramchurn, S. D., Stein, S., & Jennings, N. R. (2021) · iScience · doi:10.1016/j.isci.2021.102891 · Cited by 73
Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective
Zhang, J. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105434 · Cited by 66
“Generative RY [12], GLAD [21], [14], [22], [15], IV-BP/MF [23], CBias [24] DS [9], Multi-D [25], BCC [26], ZenCrowd [27], LC-ME [28], MinimaxEntropy [29], cBCC [30], SpectralDS [31], EBCC [32], [33], BayesDGC [34] MLNB [35], P-DS [36], ND-DS [36], MCMLD [37], MCMLD-OC [38] RY N [12]”
A framework for evaluating automatic indexing or classification in the context of retrieval
Golub, K., Soergel, D., Buchanan, G., Tudhope, D., Lykke, M., & Hiom, D. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23600 · Cited by 49
Towards Unbiased and Accurate Deferral to Multiple Experts
Keswani, V., Lease, M., & Kenthapadi, K. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462516 · Cited by 45
“Since crowdworkers employed for such annotation tasks come from diverse backgrounds, prior work in crowdsourcing has looked at the related issue of efficient distribution of input amongst the available workers [56, 75, 60, 73, 37, 58, 44, 64, 72].”
BackgroundEvaluating the Quality of Learning Resources: A Learnersourcing Approach
Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2021.3058644 · Cited by 40
“With the widespread adoption of crowdsourcing in the problems that require human intelligence, optimal integration of crowdsourced decisions in the absence of ground truth has become a challenging task [37].”
BackgroundInfluentialA technical survey on statistical modelling and design methods for crowdsourcing quality control
Jin, Y., Carman, M., Zhu, Y., & Xiang, Y. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103351 · Cited by 28
RobustFed: A Truth Inference Approach for Robust Federated Learning
Tahmasebian, F., Lou, J., & Xiong, L. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557439 · Cited by 26
“There exist various approaches to infer the truth of tasks [9, 12, 16, 19, 22, 31, 36], including direct computing [16], optimization [16, 22], probabilistic graphical model (PGM) [9, 19, 31], and neural network based approaches [35].”
BackgroundInfluentialA Large-Scale Study of Relevance Assessments with Large Language Models Using UMBRELA
Upadhyay, S., Pradeep, R., Thakur, N., Campos, D., Craswell, N., Soboroff, I., & Lin, J. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744605 · Cited by 11
Metadata
- Authors
- Venanzi, M., Guiver, J., Kazai, G., Kohli, P., & Shokouhi, M.
- Venue
- Proceedings of the 23rd international conference on World wide web
- Publisher
- ACM
- Published
- 2014-04-07
- Pages
- 155-164
- OpenAlex
- W2141649520
- Cited by
- 242
- References
- 2
- FWCI
- 15.92
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.