CoChat
Sign Up
Cited by 1,778Face recognition and analysisEmotion and Mood RecognitionFace and Expression Recognition

The DOI matches its Crossref record; 107 of the 111 references it cites were found in a registry or an index; none of the 124 sources its claims cite is checked for support yet.

Automatic analysis of facial expressions: the state of the art

Open access
Pantie, M., Rothkrantz, L.J.M.·2000·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/34.895976
Open this paper in CoChat — free accountOpen PDF

Abstract

Humans detect and interpret faces and facial expressions in a scene with little or no effort. Still, development of an automated system that accomplishes this task is rather difficult. There are several related problems: detection of an image segment as a face, extraction of the facial expression information, and classification of the expression (e.g., in emotion categories). A system that performs these operations accurately and in real time would form a big step in achieving a human-like interaction between man and machine. The paper surveys the past work in solving these problems. The capability of the human visual system with respect to these problems is discussed, too. It is meant to serve as an ultimate goal and a guide for determining recommendations for development of an automatic facial expression analyzer.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews the state of the art (through 2000) in automatic facial expression analysis, decomposing the problem into face detection, facial expression data extraction, and facial expression classification. It compares surveyed systems against an "ideal analyzer" modeled on the human visual system and concludes that none of the three subproblems has been solved in the general case.

Key findings

  • The paper identifies three related subproblems — face detection, facial expression information extraction, and facial expression classification — and argues an ideal system must automate all three, ideally in real time for HCI use.
  • Early systems (1991–1995) achieved correct recognition rates of 70–92 percent when recognizing 3–8 emotion categories; none performed quantified expression classification in terms of facial actions, and only Moses et al. [57] performed in real time.
  • Among the surveyed recent systems, reported recognition rates include 92 percent (Yoneyama et al., four expressions), 89 percent for familiar subjects and 73 percent for unknown persons (Hong et al.), 84.5 percent (Huang and Huang), 92 percent generalization / 75 percent for a novel subject (Lyons et al.), 90.1 percent (Zhang et al.), 92 percent upper-face AUs and 86 percent lower-face AUs (Pantic and Rothkrantz), 92/88/83 percent for eyebrow/eye/nose-mouth regions (Cohn et al.), 98 percent (Essa and Pentland), 95 percent (Wang et al.), and 88 percent (Black and Yacoob).
  • Most systems rely on frontal-view images of faces without facial hair or glasses, assume constant illumination and immobile subjects, and perform only singular classification into Ekman's six basic emotion categories; only two systems ([62], [106]) perform quantified classification into multiple emotion categories, and none quantifies facial action codes or distinguishes all 44 FACS AUs.

Methods

  • Design: a survey/comparative review of vision-based automatic facial expression analysis, covering work from the mid-1970s onward with emphasis on the late 1990s; systems are tabulated (Tables 2–8) against properties of an "ideal analyzer" (Table 1).
  • Scope: selectively discusses systems addressing face detection, facial expression data extraction (template-based vs. feature-based; holistic, analytic, and hybrid face representations), and classification (template-, neural-network-, and rule-based methods) from both static images and image sequences.
  • Analysis: qualitative comparison of reported techniques, assumptions, and recognition rates; the authors deliberately refrain from ranking systems, arguing that no single common testing database exists.

Limitations

  • Stated by the paper: no single well-defined, commonly used testing database exists, so the authors deliberately do not rank systems objectively; several surveyed methods were tested only on the images used for training, so their person-independence is uncertain; most systems assume frontal views, constant illumination, no facial hair or glasses, and immobile subjects, which is unrealistic for the target application domains; only two systems handle automatic face detection in an arbitrary scene and only three deal to some extent with rigid head motions; only Essa and Pentland's method handles faces with facial hair and/or eyeglasses; no system performs quantification of facial action codes or distinguishes all 44 FACS AUs.
  • Evident from the text: the survey is selective rather than exhaustive, so coverage of the field is incomplete by the authors' own statement; the "ideal analyzer" is a normative construct rather than an implemented benchmark, and the comparison is largely qualitative with heterogeneous, non-comparable evaluation protocols across systems.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/34.895976

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 124 cited sources checked · 0 supported · 0 flagged

Show all 67 claims

  • Probably not a substantial claim. 58
  • It may state two separate claims as one. 47
  • The quote only partly states this claim. 6
93 of 93

  • The paper relies on this without support.

    A face is perceived as a whole, not as a collection of the facial features.

    “Another characteristic of the human visual system is that a face is perceived as a whole, not as a collection of the facial features.”

  • The paper relies on this without support.

    When a face is partially occluded, humans perceive a whole face as if the perceptual system fills in the missing parts.

    “When a face is partially occluded (e.g., by a hand), we perceive a whole face, as if our perceptual system fills in the missing parts.”

  • The paper relies on this without support.

    Each person has his/her own maximal intensity of displaying a particular facial expression.

    “On the other hand, each person has his/her own maximal intensity of displaying a particular facial expression.”

  • The paper relies on this without support.

    In most real-life situations, complete immovability of the observed subject cannot be assumed.

    “In most real-life situations, complete immovability of the observed subject cannot be assumed.”

  • The paper relies on this without support.

    Considering the state-of-the-art in image processing techniques, inaccurate, noisy, and missing data could be expected.

    “Considering the stateof-the-art in image processing techniques, inaccurate, noisy, and missing data could be expected.”

  • The paper relies on this without support.

    Long delays make interaction desynchronized and less efficient.

    “Long delays make the interaction desynchronized and less efficient.”

  • The paper relies on this without support.

    In most real-life situations where an automated facial expression analyzer is to be employed, the location of a face in the image is not known a priori.

    “Yet, in most of the real-life situations where an automated facial expression analyzer is to be employed (e.g., in a multimodal/media HCI), the location of a face in the image is not known a priori.”

  • The paper relies on this without support.

    In most real-life situations, the observed subject will not remain immovable.

    “Also, in most of the real-life situations, it cannot be assumed that the observed subject will remain immovable, as assumed by some methods.”

  • The paper relies on this without support.

    Pure emotional expressions are seldom elicited.

    “First, ªpureº emotional expressions are seldom elicited.”

  • The paper relies on this without support.

    Most of the time, people show blends of emotional expressions.

    “Most of the time, people show blends of emotional expressions.”

  • The paper relies on this without support.

    Some facial actions such as blink, wink, and lips sucked into the mouth are either encountered or not.

    “Second, some facial actions such as blink, wink, and lips sucked into the mouth, are either encountered or not.”

  • The paper relies on this without support.

    Each person has his or her own maximal intensity of displaying a particular facial action.

    “In addition, each person has his/her own maximal intensity of displaying a particular facial action.”

  • The paper relies on this without support.

    FACS provides an option to score intensity on a 3-level intensity scale for only five different AUs.

    “First, there are merely five different AUs for which FACS provides an option to score intensity on a 3-level intensity scale (low, medium, and high).”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conclusion says all systems classify only basic emotions

    Major

    The conclusion says every surveyed approach performs only basic-emotion classification, but the discussion states that four surveyed systems also perform facial-action coding.

  • Citation for Ekman's six categories points to different authors

    Minor

    The statement attributes the six categories to Ekman and cites reference [19], but the paper's reference [19] is by Eisert and Girod.

2 of 2

Citations

Cites

Flagged first, then by citations

Show all 111 references

4 of the 111 references couldn’t be matched to a known paper.

Research works printed but not linked: 13

  1. wiskott, 1995, labelled graphs and dynamic link matching for face recognition and scene analysis, Reihe Physik, 53
  2. cottrell, 1991, empath: fface, emotion, gender recognition using holons, Advances in Neural Information Processing Systems 3, 564
  3. vanger, 1995, applications of synergetics in decoding facial expression of eemotion, Proc Third Int l Conf Automatic Face and Gesture Recognition, 24
  4. izard, 1971, The Face of Emotion
  5. johanstone, 1995, acoustic profiles in prototypical vocal expressions of emotions, Proc Int l Conf Phonetic Science, 4, 2
  6. kawakami, 1995, 3d emotion space for interactive communication, Proc Computer Science Conf, 471
  7. ekman, 1978, Facial Action Coding System (FACS) Manual
  8. fridlund, 1987, facial expressions of emotion: review literature 1970-1983, Nonverbal Behavior and Communication, 143
  9. ellis, 1986, process underlying face recognition, The Neuropsychology of Face Perception and Facial Expression, 1
  10. bruce, 1986, Recognizing Faces
  11. matsuno, 1993, recognition of facial expression with potential net, Proc Asian Conf Computer Vision, 504
  12. zhao, 1996, classifying facial emotions by backpropagation neural networks with fuzzy inputs, Proc Conf Neural Information Processing, 1, 454
  13. picard, 1999, offline and online recognition of emotion expression from physiological data, Emotion-Based Agent Architectures Workshop Notes Int l Conf Autonomous Agents, 135

Cited by

Sorted by citations

Items 901–1,000 of 1,000

Related

See all

Metadata

Authors
Pantie, M., & Rothkrantz, L. J. M.
Affiliations
Delft University of Technology (NL)
ORCID
Rothkrantz, L.J.M.
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2000
Volume
22
Issue
12
Pages
1424-1445
ISSN
0162-8828 (ISSN-L)
DOI
10.1109/34.895976
OpenAlex
W2159017231
Semantic Scholar
087a6f471b5177014b08a342968901e1ef083ed1
Cited by
1,778
References
111
Citations by year
2026: 33 · 2025: 31 · 2024: 30 · 2023: 43 · 2022: 41 · 2021: 43 · 2020: 58 · 2019: 65 · 2018: 72 · 2017: 71 · 2016: 86 · 2015: 87 · 2014: 105 · 2013: 105 · 2012: 108
FWCI
23.88
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • Research Repository (Delft University of Technology) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
facial expression analysis, face detection, facial expression recognition, emotion classification, human visual system, human-machine interaction, real-time systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review