The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild
Abstract
Research in face perception and emotion theory requires very large annotated databases of images of facial expressions of emotion. Annotations should include Action Units (AUs) and their intensities as well as emotion category. This goal cannot be readily achieved manually. Herein, we present a novel computer vision algorithm to annotate a large database of one million images of facial expressions of emotion in the wild (i.e., face images downloaded from the Internet). First, we show that this newly proposed algorithm can recognize AUs and their intensities reliably across databases. To our knowledge, this is the first published algorithm to achieve highly-accurate results in the recognition of AUs and their intensities across multiple databases. Our algorithm also runs in real-time (>30 images/second), allowing it to work with large numbers of images and video sequences. Second, we use WordNet to download 1,000,000 images of facial expressions with associated emotion keywords from the Internet. These images are then automatically annotated with AUs, AU intensities and emotion categories by our algorithm. The result is a highly useful database that can be readily queried using semantic descriptions for applications in computer vision, affective computing, social and cognitive psychology and neuroscience, e.g., "show me all the images with happy faces" or "all images with AU 1 at intensity c".
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 7×
“WordNet includes synonyms (i.e., words that have the same or nearly the same meaning), hyponyms (i.e., subordinate nouns or nouns of more specific meaning, which defines a hierarchy of relationships), troponymys (i.e., verbs of more specific meaning, which defines a hierarchy of verbs), and entailments (i.e., deductio…”
MethodInfluentialLucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
LabelMe: A Database and Web-Based Tool for Image Annotation
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0090-8 · Cited by 4,210
Daugman, J. G. (1985) · Journal of the Optical Society of America A · doi:10.1364/josaa.2.001160 · Cited by 3,219
Face recognition by elastic bunch graph matching
Wiskott, L., Fellous, J.-M., Kuiger, N., & von der Malsburg, C. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598235 · Cited by 2,883
One millisecond face alignment with an ensemble of regression trees
Kazemi, V., & Sullivan, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.241 · Cited by 2,831
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,941
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & De la Torre, F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.75 · Cited by 1,896
Compound facial expressions of emotion
Du, S., Tao, Y., & Martinez, A. M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1322355111 · Cited by 806
Self-citationDISFA: A Spontaneous Facial Action Intensity Database
Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.4 · Cited by 793
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE MultiMedia · doi:10.1109/mmul.2012.26 · Cited by 663
Painful data: The UNBC-McMaster shoulder pain expression archive database
Lucey, P., Cohn, J. F., Prkachin, K. M., Solomon, P. E., & Matthews, I. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771462 · Cited by 599
Subclass discriminant analysis
Zhu, M., & Martinez, A. M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.172 · Cited by 378
Self-citationSelective Transfer Machine for Personalized Facial Action Unit Detection
Chu, W.-S., De la Torre, F., & Cohn, J. F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.451 · Cited by 307
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305
Automatically Detecting Pain in Video Through Facial Action Units
Lucey, P., Cohn, J. F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., & Prkachin, K. M. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2082525 · Cited by 267
A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models
Koelstra, S., Pantic, M., & Patras, I. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.50 · Cited by 242
McDuff, D., el Kaliouby, R., Senechal, T., Amr, M., Cohn, J. F., & Picard, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.130 · Cited by 234
Joint patch and multi-label learning for facial action unit detection
Zhao, K., Chu, W.-S., De la Torre, F., Cohn, J. F., & Zhang, H. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298833 · Cited by 196
Martinez, A. M., & Du, S. (2017) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-57021-1_6 · Cited by 173
Self-citationWhere are linear feature extraction methods applicable?
Martı́nez, A. M., & Zhu, M. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.250 · Cited by 128
Capturing Global Semantic Relationships for Facial Action Unit Recognition
Wang, Z., Li, Y., Wang, S., & Ji, Q. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.410 · Cited by 126
The not face: A grammaticalization of facial expressions of emotion
Benitez-Quiroz, C. F., Wilbur, R. Β., & Martínez, A. M. (2016) · Cognition · doi:10.1016/j.cognition.2016.02.004 · Cited by 125
Self-citationThe utility of surface reflectance for the recognition of upright and inverted faces
Russell, R. P., Biederman, I., Nederhouser, M., & Sinha, P. (2006) · Vision Research · doi:10.1016/j.visres.2006.11.002 · Cited by 104
Kernel Optimization in Discriminant Analysis
You, D., Hamsici, O. C., & Martı́nez, A. M. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.173 · Cited by 95
Action unit detection with segment-based SVMs
Simon, T., Nguyen, M. H., la Torre, F. D., & Cohn, J. F. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539998 · Cited by 86
Emotion perception in emotionless face images suggests a norm-based representation
Neth, D., & Martínez, A. M. (2009) · Journal of Vision · doi:10.1167/9.1.5 · Cited by 79
A Neural Basis of Facial Action Recognition in Humans
Srinivasan, R., Golomb, J. D., & Martínez, A. M. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.1704-15.2016 · Cited by 75
Self-citationPain Intensity Evaluation through Facial Action Units
Zafar, Z., & Khan, N. A. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.803 · Cited by 44
A lp-norm MTMKL framework for simultaneous detection of multiple facial action units
Zhang, X., Mahoor, M. H., Mavadati, S. M., & Cohn, J. F. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835735 · Cited by 26
Cited by
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019) · Psychological Science in the Public Interest · doi:10.1177/1529100619832930 · Cited by 1,748
Self-citationDeep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild
Li, S., Deng, W., & Du, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.277 · Cited by 1,470
Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
Wang, K., Peng, X., Yang, J., Meng, D., & Qiao, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2956143 · Cited by 940
Metadata
- Authors
- Benitez-Quiroz, C. F., Srinivasan, R., & Martinez, A. M.
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 5562-5570
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2436394355
- Cited by
- 607
- References
- 32
- FWCI
- 60.38
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.