CoChat
Sign Up
Cited by 926Emotion and Mood RecognitionMachine Learning and Data ClassificationFace recognition and analysis

Training deep networks for facial expression recognition with crowd-sourced label distribution

Open access
Barsoum, Emad, Zhang, Cha, Ferrer, Cristian Canton, Zhang, Zhengyou·2016·Proceedings of the 18th ACM International Conference on Multimodal Interaction
DOI 10.1145/2993148.2993165
Open this paper in CoChat — free accountOpen PDF

Abstract

Crowd sourcing has become a widely adopted scheme to collect ground truth labels. However, it is a well-known problem that these labels can be very noisy. In this paper, we demonstrate how to learn a deep convolutional neural network (DCNN) from noisy labels, using facial expression recognition as an example. More specifically, we have 10 taggers to label each input image, and compare four different approaches to utilizing the multiple labels: majority voting, multi-label learning, probabilistic label drawing, and cross-entropy loss. We show that the traditional majority voting scheme does not perform as well as the last two approaches that fully leverage the label distribution. An enhanced FER+ data set with multiple labels for each face image will also be shared with the research community.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/2993148.2993165

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. V. Ambati. Active Learning and Crowdsourcing for Machine Translation in Low Resource Scenarios. PhD thesis, Pittsburgh, PA, USA, 2012. AAI3528171.
  2. A. Batliner, S. Steidl, C. Hacker, and E. Nöth. Private emotions versus social interaction: a data-driven approach towards analysing emotion in speech. User Modeling and User-Adapted Interaction, 18(1):175-206, 2007.
  3. A. Bosch, A. Zisserman, and X. Munoz. Representing shape with a spatial pyramid kernel. In Proceedings of the 6th ACM international conference on Image and video retrieval, pages 401-408. ACM, 2007.
  4. A. Burmania, S. Parthasarathy, and C. Busso. Increasing the reliability of crowdsourcing evaluations using online quality assessment. IEEE Transactions on Affective Computing, PP(99):1-1, 2015.
  5. H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma. CREMA-D: crowd-sourced emotional multimodal actors dataset. IEEE Trans. Affective Computing, 5(4):377-390, 2014.
  6. L. Devillers, L. Vidrascu, and L. Lamel. Challenges in real-life emotion annotation and machine learning based detection. Neural Networks, 18(4):407-422, 2005.
  7. C. Eickhoff and A. P. de Vries. Increasing cheat robustness of crowdsourcing tasks. Inf. Retr., 16(2):121-137, 2013.
  8. P. Ekman and W. V. Friesen. Constants across cultures in the face and emotion. Journal of personality and social psychology, 17(2):124, 1971.
  9. I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, et al. Challenges in representation learning: A report on three machine learning contests. In Neural information processing, pages 117-124. Springer, 2013.
  10. S. E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, Ç. Gülçehre, R. Memisevic, P. Vincent, A. Courville, Y. Bengio, R. C. Ferrari, et al. Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 543-550. ACM, 2013.
  11. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In M. Czerwinski, A. M. Lund, and D. S. Tan, editors, Proceedings of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, 2008, Florence, Italy, April 5-10, 2008, pages 453-456. ACM, 2008.
  12. M. Liu, S. Li, S. Shan, R. Wang, and X. Chen. Deeply learning deformable facial action parts model for dynamic expression analysis. In Computer Vision-ACCV 2014, pages 143-157. Springer, 2014.
  13. P. Liu, S. Han, Z. Meng, and Y. Tong. Facial expression recognition via a boosted deep belief network. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1805-1812. IEEE, 2014.
  14. E. Mower, A. Metallinou, C.-C. Lee, A. Kazemzadeh, C. Busso, S. Lee, and S. Narayanan. Interpreting ambiguous emotional expressions. In Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on, pages 1-8. IEEE, 2009.
  15. V. Ojansivu and J. Heikkilä. Blur insensitive texture classification using local phase quantization. In Image and signal processing, pages 236-243. Springer, 2008.
  16. R. Rosenthal. Conducting judgment studies: Some methodological issues. The new handbook of methods in nonverbal behavior research, pages 199-234, 2005.
  17. N. Sadoughi, Y. Liu, and C. Busso. Speech-driven animation constrained by appropriate discourse functions. In A. A. Salah, J. F. Cohn, B. W. Schuller, O. Aran, L. Morency, and P. R. Cohen, editors, Proceedings of the 16th International Conference on Multimodal Interaction, ICMI 2014, Istanbul, Turkey, November 12-16, 2014, pages 148-155. ACM, 2014.
  18. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. · arXiv
  19. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast-but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, pages 254-263, Stroudsburg, PA, USA, 2008. Association for Computational Linguistics.
  20. T. Sobol-Shikler and P. Robinson. Classification of complex information: Inference of co-occurring affective states from their expressions in speech. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(7):1284-1297, 2010.
  21. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  22. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1-9, 2015.
  23. Y. Tang. Deep learning using linear support vector machines. arXiv preprint arXiv:1306.0239, 2013. · arXiv
  24. K. Trohidis, G. Tsoumakas, G. Kalliris, and I. P. Vlahavas. Multi-label classification of music into emotions. In ISMIR, volume 8, pages 325-330, 2008.
  25. L. Xie, J. Wang, Z. Wei, M. Wang, and Q. Tian. Disturblabel: Regularizing cnn on the loss layer. arXiv preprint arXiv:1605.00055, 2016. · arXiv
  26. Z. Yu and C. Zhang. Image based static facial expression recognition with multiple deep network learning. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI '15, pages 435-442, New York, NY, USA, 2015. ACM.
  27. Z. Zhang. Feature-based facial expression recognition: Sensitivity analysis and experiments with a multi-layer perceptron. International Journal of Pattern Recognition and Artificial Intelligence, 13(6):893-911, 1999.
  28. G. Zhao and M. Pietikainen. Dynamic texture recognition using local binary patterns with an application to facial expressions. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 29(6):915-928, 2007.
  29. Y. Zhou, H. Xue, and X. Geng. Emotion distribution recognition from facial expressions. In Proceedings of the 23rd ACM International Conference on Multimedia, MM '15, pages 1247-1250, New York, NY, USA, 2015. ACM.

Cited by

Sorted by citations

Authors
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
Proceedings of the 18th ACM International Conference on Multimodal Interaction
Publisher
ACM
Published
2016-10-31
Pages
279-283
DOI
10.1145/2993148.2993165
OpenAlex
W2481681431
Cited by
926
References
3
Citations by year
2026: 85 · 2025: 155 · 2024: 159 · 2023: 156 · 2022: 101 · 2021: 105 · 2020: 71 · 2019: 55 · 2018: 28 · 2017: 10 · 2016: 1
FWCI
36.24
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
facial expression recognition, label noise, label distribution, deep convolutional neural networks, majority voting, multi-label classification, cross-entropy loss

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review