CoChat
Sign Up
Cited by 10Mobile Crowdsensing and CrowdsourcingMachine Learning and Data ClassificationMachine Learning and Algorithms

Deep Learning from Crowds

Open access
Rodrigues, Filipe, Pereira, Francisco·2018·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v32i1.11506
Open this paper in CoChat — free account

Abstract

Over the last few years, deep learning has revolutionized the field of machine learning by dramatically improving the state-of-the-art in various domains. However, as the size of supervised artificial neural networks grows, typically so does the need for larger labeled datasets. Recently, crowdsourcing has established itself as an efficient and cost-effective solution for labeling large sets of data in a scalable manner, but it often requires aggregating labels from multiple noisy contributors with different levels of expertise. In this paper, we address the problem of learning deep neural networks from crowds. We begin by describing an EM algorithm for jointly learning the parameters of the network and the reliabilities of the annotators. Then, a novel general-purpose crowd layer is proposed, which allows us to train deep neural networks end-to-end, directly from the noisy labels of multiple annotators, using only backpropagation. We empirically show that the proposed approach is able to internally capture the reliability and biases of different annotators and achieve new state-of-the-art results for various crowdsourced datasets across different settings, namely classification, regression and sequence labeling.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v32i1.11506

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Crowdlayer predicts a context-dependent distribution P(c|X) using a neural context encoder.

    “Crowdlayer predicts a context-dependent distribution P (c n |X n ) using a neural context encoder.”

    From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Rodrigues and Pereira, 2018

    The Crowd layer section (p. 3) says a neural network’s softmax output gives the probability that an input belongs to class c. It does not describe c as a context variable or identify a neural context encoder; the crowd layer instead maps those class probabilities to annotator-specific label outputs. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • SkillAggregation extends the Crowdlayer aggregation method to exploit judge estimates during inference.

    “It extends the Crowdlayer aggregation method, developed for image classification, to exploit the judge estimates during inference.”

    From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Rodrigues and Pereira, 2018

  • The Crowdlayer method performs inference by directly selecting the class with the highest probability from the bottleneck output s_n without utilizing worker predictions.

    “The Crowdlayer method performs inference by directly selecting the class with the highest probability from the bottleneck output s n without utilizing worker predictions.”

    From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Rodrigues and Pereira, 2018

  • SkillAggregation is based on a reformulation of Crowdlayer.

    “We propose SkillAggregation, an aggregation method based on a reformulation of Crowdlayer (Rodrigues and Pereira, 2018).”

    From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Rodrigues and Pereira, 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. [Albarqouni et al. 2016] Albarqouni, S.; Baur, C.; Achilles, F.; Belagiannis, V.; Demirci, S.; and Navab, N. 2016. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE transactions on medical imaging 35(5):1313-1321.
  2. [Buhrmester, Kwang, and Gosling 2011] Buhrmester, M.; Kwang, T.; and Gosling, S. D. 2011. Amazon's mechanical turk a new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6(1):3-5.
  3. [Dawid and Skene 1979] Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer errorrates using the EM algorithm. J. of the Royal Statistical Society. Series C 28(1):20-28.
  4. [Donmez and Carbonell 2008] Donmez, P., and Carbonell, J. 2008. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In Proc. of the 17th ACM Conf. on Information and Knowledge Management , 619-628.
  5. [Greenspan, van Ginneken, and Summers 2016] Greenspan, H.; van Ginneken, B.; and Summers, R. M. 2016. Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging 35(5):1153-1159.
  6. [Guan et al. 2017] Guan, M. Y.; Gulshan, V.; Dai, A. M.; and Hinton, G. E. 2017. Who said what: Modeling individual labelers improves classification. arXiv preprint arXiv:1703.08774 . · arXiv
  7. [Ioffe and Szegedy 2015] Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 . · arXiv
  8. [Ipeirotis, Provost, and Wang 2010] Ipeirotis, P. G.; Provost, F.; and Wang, J. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation , 64-67. ACM.
  9. [LeCun, Bengio, and Hinton 2015] LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. Nature 521(7553):436-444.
  10. [Parent and Eskenazi 2011] Parent, G., and Eskenazi, M. 2011. Speaking to the crowd: Looking at past achieve- ments in using crowdsourcing for speech and predicting future challenges. In INTERSPEECH , 3037-3040. Citeseer.
  11. [Pennington, Socher, and Manning 2014] Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP , volume 14, 1532-1543.
  12. [Rodrigues et al. 2017] Rodrigues, F.; Lourenc ¸o, M.; Ribeiro, B.; and Pereira, F. 2017. Learning supervised topic models for classification and regression from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence .
  13. [Rodrigues, Pereira, and Ribeiro 2014] Rodrigues, F.; Pereira, F.; and Ribeiro, B. 2014. Gaussian process classification and active learning with multiple annotators. In Proc. of the 31st Int. Conf. on Machine Learning , 433-441.
  14. [Russell et al. 2008] Russell, B.; Torralba, A.; Murphy, K.; and Freeman, W. 2008. Labelme: a database and web-based tool for image annotation. Int. J. of Computer Vision 77(13):157-173.
  15. [Sang and Meulder 2003] Sang, E., and Meulder, F. D. 2003. Introduction to the conll-2003 shared task: languageindependent named entity recognition. In Proceedings of the 7th Conference on Natural Language Learning at HLTNAACL , volume 4, 142-147.
  16. [Sheng, Provost, and Ipeirotis 2008] Sheng, V.; Provost, F.; and Ipeirotis, P. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proc. of the Int. Conf. on Knowledge Discovery and Data Mining , 614-622.
  17. [Simonyan and Zisserman 2014] Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 . · arXiv
  18. [Smyth et al. 1995] Smyth, P.; Fayyad, U.; Burl, M.; Perona, P.; and Baldi, P. 1995. Inferring ground truth from subjective labelling of venus images. In Advances in Neural Information Processing Systems , 1085-1092.
  19. [Snow et al. 2008] Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. 2008. Cheap and fast - but is it good?: Evaluating non-expert annotations for natural language tasks. In Proc. of the Conf. on Empirical Methods in Natural Language Processing , 254-263.
  20. [Su, Deng, and Fei-Fei 2012] Su, H.; Deng, J.; and Fei-Fei, L. 2012. Crowdsourcing annotations for visual object detection. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence , volume 1.
  21. [Welinder et al. 2010] Welinder, P.; Branson, S.; Perona, P.; and Belongie, S. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems , 2424-2432.
  22. [Whitehill et al. 2009] Whitehill, J.; Wu, T.-f.; Bergsma, J.; Movellan, J. R.; and Ruvolo, P. L. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems , 2035-2043.
  23. [Yan et al. 2014] Yan, Y.; Rosales, R.; Fung, G.; Subramanian, R.; and Dy, J. 2014. Learning from multiple annotators with varying expertise. Mach. Learn. 95(3):291-327.

Cited by

Sorted by citations

Authors
Rodrigues, F., & Pereira, F.
Affiliations
Technical University of Denmark (DK)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2018-04-25
Volume
32
Issue
1
Pages
1611-1618
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v32i1.11506
OpenAlex
W2750671411
Cited by
10
References
1
Citations by year
2026: 2 · 2021: 2 · 2019: 3 · 2018: 3
FWCI
2.25
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • European Commission (H2020-MSCA-IF-2016, 2007–2013, FP7 2007-2013, HORIZON2020, H2020, 609405, 745673, FP7/2007, H2020‐MSCA‐IF, H2020-MSCA)
  • Horizon 2020 Framework Programme (FP7/2007-2013, MSCA-IF-2016, H2020-MSCA, 609405, H2020-MSCA-IF-2016)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
crowdsourcing, label noise, deep neural network, annotator reliability, expectation-maximization algorithm, end-to-end learning, label aggregation, annotator bias, sequence labeling, regression analysis, classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review