Machine Interpretation of Emotion: Design of a Memory‐Based Expert System for Interpreting Facial Expressions in Terms of Signaled Emotions
Open accessAbstract
As a first step in involving user emotion in human‐computer interaction, a memory‐based expert system (JANUS; Kearney, 1991) was designed to interpret facial expression in terms of the signaled emotion. Anticipating that a VDU‐mounted camera will eventually supply face parameters automatically, JANUS now accepts manually made measurements on a digitized full‐face photograph and returns emotion labels used by college students. An intermediate representation in terms of face actions (e.g., mouth open) is also used. Production rules convert the geometry into these. A dynamic memory (Kolodner, 1984; Schank, 1982) interprets the face actions in terms of emotion labels. The memory is dynamic in the sense that new emotion labels can be learned with experience. A prototype system has been implemented on a Sun 2/120 system using POPLOG. Validation studies on the prototype suggest that the interpretations achieved are generally consistent with those of college students without formal instruction in emotion signals.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although Hong et al. utilize the PersonSpotter system, which deals with real-time processing of video sequences, they perform facial expression analysis from static images.
“Although Hong et al. utilize the PersonSpotter system [81], which deals with real-time processing of video sequences, they perform facial expression analysis from static images.”
From Automatic analysis of facial expressions: the state of the art (2000) · cites this paper as [81]
The PersonSpotter system performs well in the presence of background motion but fails in the case of covered or too much rotated faces.
“Steffens et al. [81] reported that their system performs well in the presence of background motion, but fails in the case of covered or too much rotated faces.”
From Automatic analysis of facial expressions: the state of the art (2000) · cites this paper as [81]
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · see citing works
The use of ranks to avoid the assumption of normality implicit in the analysis of variance
Friedman, M. (1937) · Journal of the American Statistical Association · doi:10.1080/01621459.1937.10503522 · Cited by 7,005 · see citing works
Constants across cultures in the face and emotion.
Ekman, P., & Friesen, W. V. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030377 · Cited by 5,509 · see citing works
Visual perception of biological motion and a model for its analysis
Johansson, G. (1973) · Perception & Psychophysics · doi:10.3758/bf03212378 · Cited by 4,547 · see citing works
A revised version of the Psychoticism scale
Eysenck, S. B. G., Eysenck, H. J., & Barrett, P. (1985) · Personality and Individual Differences · doi:10.1016/0191-8869(85)90026-1 · Cited by 3,094 · see citing works
Dynamic Memory: A Theory of Reminding and Learning in Computers and People
Schank Roger, C. (1983) · Medical Entomology and Zoology · Cited by 1,986 · see citing works
Pan-cultural elements in facial displays of emotion
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969) · Science · doi:10.1126/science.164.3875.86 · Cited by 1,775 · see citing works
The Duchenne smile: Emotional expression and brain physiology: II.
Ekman, P., Davidson, R. J., & Friesen, W. V. (1990) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.58.2.342 · Cited by 1,371 · see citing works
Ekman, P., & Friesen, W. V. (1976) · Journal of Nonverbal Behavior · doi:10.1007/bf01115465 · Cited by 1,242 · see citing works
Distributed memory and the representation of general and specific information.
McClelland, J. L., & Rumelhart, D. (1985) · Journal of experimental psychology. General · doi:10.1037/0096-3445.114.2.159 · Cited by 1,160 · see citing works
Parallel models of associative memory
Heath, R. A. (1982) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(82)90036-0 · Cited by 1,111 · see citing works
Concept of emotion viewed from a prototype perspective.
Fehr, B., & Russell, J. A. (1984) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.113.3.464 · Cited by 1,075 · see citing works
Ellis, H. D., Jeeves, M. A., Newcombe, F., & Young, A. (1986) · Springer eBooks · doi:10.1007/978-94-009-4420-6 · Cited by 693 · see citing works
Perceiving and Remembering Faces
Winograd, E., DAVIES, G. M., Ellis, H., & Shepherd, J. (1983) · The American Journal of Psychology · doi:10.2307/1422222 · Cited by 640 · see citing works
Ekman, P., Friesen, W. V., & O'Sullivan, M. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.3.414 · Cited by 628 · see citing works
Perspectives on Memory Research
Zechmeister, E. B., Nilsson, L., Cermak, L. S., & Craik, F. I. M. (1980) · The American Journal of Psychology · doi:10.2307/1422127 · Cited by 557 · see citing works
An investigation into component and configural processes underlying face perception
Sergent, J. (1984) · British Journal of Psychology · doi:10.1111/j.2044-8295.1984.tb01895.x · Cited by 493 · see citing works
Kohonen, T. (1977) · Communication and cybernetics · doi:10.1007/978-3-642-96384-1 · Cited by 465 · see citing works
Depth of processing pictures of faces and recognition memory.
Bower, G. H., & Karlin, M. B. (1974) · Journal of Experimental Psychology · doi:10.1037/h0037190 · Cited by 438 · see citing works
Recognition of Facial Expression from Optical Flow
Mase, K. (1991) · IEICE Transactions on Information and Systems · Cited by 404 · see citing works
Selectivity of learning caused by affective states.
Bower, G. H., Gilligan, S. G., & Monteiro, K. P. (1981) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.110.4.451 · Cited by 390 · see citing works
Facial motion in the perception of faces and of emotional expression.
Bassili, J. N. (1978) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.4.3.373 · Cited by 369 · see citing works
Retrieval and Organizational Strategies in Conceptual Memory: A Computer Model
Wolff, A. S., & Kolodner, J. L. (1985) · The American Journal of Psychology · doi:10.2307/1422458 · Cited by 313 · see citing works
WISARD·a radical step forward in image recognition
Aleksander, I., Thomas, W., & Bowden, P. (1984) · Sensor Review · doi:10.1108/eb007637 · Cited by 304 · see citing works
Facial Affect Scoring Technique: A First Validity Study
Ekman, P., Friesen, W. V., & Tomkins, S. S. (1971) · Semiotica · doi:10.1515/semi.1971.3.1.37 · Cited by 290 · see citing works
Motives, mechanisms, and emotions
Sloman, A. (1987) · Cognition & Emotion · doi:10.1080/02699938708408049 · Cited by 286 · see citing works
Organization in autobiographical memory
Conway, M., & Bekerian, D. A. (1987) · Memory & Cognition · doi:10.3758/bf03197023 · Cited by 275 · see citing works
Patterson, K. E., & Baddeley, A. (1977) · Journal of Experimental Psychology Human Learning & Memory · doi:10.1037/0278-7393.3.4.406 · Cited by 253 · see citing works
Automatic extraction of face-features
Craw, I., Ellis, H., & Lishman, J. R. (1987) · Pattern Recognition Letters · doi:10.1016/0167-8655(87)90039-0 · Cited by 195 · see citing works
Generalization and memory in an integrated understanding system
Lebowitz, M. (1980) · Defense Technical Information Center (DTIC) · Cited by 183 · see citing works
Memory for faces: Encoding and retrieval operations
Wells, G. L., & Hryciw, B. (1984) · Memory & Cognition · doi:10.3758/bf03198293 · Cited by 117 · see citing works
Situational knowledge and emotions
Conway, M. A., & Bekerian, D. A. (1987) · Cognition & Emotion · doi:10.1080/02699938708408044 · Cited by 111 · see citing works
Recognition Memory for Photographs of Faces
Galper, R. E., & Hochberg, J. E. (1971) · The American Journal of Psychology · doi:10.2307/1420466 · Cited by 98 · see citing works
Comparing the Joint Agreement of Several Raters with Another Rater
Williams, G. W. (1976) · Biometrics · doi:10.2307/2529750 · Cited by 94 · see citing works
Critics: an emerging approach to knowledge-based human-computer interaction
Fischer, G., Lemke, A., Mastaglio, T. W., & Mørch, A. I. (1991) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(05)80184-1 · Cited by 89 · see citing works
Context effects in recognition memory for faces
Watkins, M. J., Ho, E., & Tulving, E. (1976) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/0022-5371(76)90045-1 · Cited by 79 · see citing works
Recognition memory for faces following nine different judgments
Winograd, E. (1976) · Bulletin of the Psychonomic Society · doi:10.3758/bf03335185 · Cited by 76 · see citing works
Tense quantifiers, and contexts
Hinrichs, E. (1988) · doi:10.5555/55056.55057 · Cited by 39 · see citing works
Processing multiple physical features in facial recognition
COURTOIS, M. R., & Mueller, J. H. (1979) · Bulletin of the Psychonomic Society · doi:10.3758/bf03329404 · Cited by 38 · see citing works
Levels of processing in facial recognition memory
Strnad, B. N., & Mueller, J. H. (1977) · Bulletin of the Psychonomic Society · doi:10.3758/bf03336915 · Cited by 31 · see citing works
A microcomputer based approach to the quantification of facial expressions.
Pilowsky, I., Thornton, M., & Stokes, B. B. (1985) · PubMed · Cited by 24 · see citing works
Aspects of Nonverbal Communication
Porchet-Munro, S. (1991) · Recent results in cancer research/Recent Results in Cancer Research · doi:10.1007/978-3-642-84138-5_37 · Cited by 19 · see citing works
Schank, R. C. (1989) · doi:10.21236/ada219875 · Cited by 14 · see citing works
Facial Perception: Holistic or Feature-Analytic?
Jensen, D. G. (1986) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193128603000726 · Cited by 5 · see citing works
Techniques for the analysis of human movement
(1976) · Computers in Biology and Medicine · doi:10.1016/0010-4825(76)90048-2 · Cited by 5 · see citing works
Making IT work: an overview of the Janus Project
Brodie, M. (1989) · Cited by 2 · see citing works
Gda, K. (1991) · Europe PMC (PubMed Central) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · see citing works
Expert system for automatic analysis of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2000) · Image and Vision Computing · doi:10.1016/s0262-8856(00)00034-2 · Cited by 415 · see citing works
An expert system for multiple emotional classification of facial expressions
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings 11th International Conference on Tools with Artificial Intelligence · doi:10.1109/tai.1999.809775 · Cited by 48 · see citing works
- Authors
- Kearney, G. D., & McKenzie, S.
- Affiliations
- University of Greenwich (GB)
- Corresponding author
- Kearney, Garrett D.
- Venue
- Cognitive Science
- Publisher
- Wiley
- Published
- 1993-10
- Volume
- 17
- Issue
- 4
- Pages
- 589-622
- ISSN
- 0364-0213 (ISSN-L)
- 1551-6709
- OpenAlex
- W2097755513
- Cited by
- 36
- References
- 48
- Citations by year
- 2023: 1 · 2019: 1 · 2018: 1 · 2017: 1 · 2016: 1 · 2013: 2 · 2012: 1
- FWCI
- 2.14
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Cognitive Science (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- facial expression interpretation, emotion recognition, dynamic memory, production rules, emotion labeling, face geometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review