CoChat

Multimodal emotion recognition from expressive faces, body gestures and speech

Open access
Caridakis, George, Castellano, Ginevra, Kessous, Loic, Raouzaiou, Amaryllis, Malatesta, Lori, Asteriadis, Stelios, Karpouzis, Kostas · 2007 · IFIP The International Federation for Information Processing

The DOI matches its Crossref record; 33 of the 37 references it cites were found in a registry or an index; 1 of the 2 sources its claims cite were checked for support.

Overview

CoChat Summary

TL;DR

This paper presents a Bayesian multimodal emotion recognition system that combines facial expressions, body gestures, and speech to classify eight acted emotions. Fusing modalities improved recognition over the best unimodal system by more than 10%, with feature-level fusion outperforming decision-level fusion.

Key findings

  • Multimodal feature-level fusion reached 78.3% accuracy when all 240 samples were used and 89.4% when only samples available for all three modalities were used, compared with the best unimodal system (gestures: 67.1% and 83.2%, respectively).
  • Feature-level fusion outperformed decision-level fusion: 78.3% vs. 74.6% with all samples; with complete three-modality samples, feature-level was 89.4%, while decision-level was 85.1% using best probability and 88.20% using majority voting plus best probability.
  • Unimodal performance with all samples was 67.1% for gestures, 57.1% for speech, and 48.3% for facial expressions; with only complete three-modality samples, these rose to 83.2%, 70.8%, and 59.6%, respectively.
  • In the feature-level multimodal confusion matrix, the paper reports that all emotions apart from despair can be recognised with more than 70% accuracy.

Methods

  • Data: a multimodal corpus collected during the Third Summer School of the HUMAINE EU-IST project in Genova, September 2006, based on the GEMEP corpus; 10 participants of five nationalities acted eight emotions (anger, despair, interest, pleasure, sadness, irritation, joy, pride), each emotion acted three times by each actor, yielding 240 posed samples per modality.
  • Features: facial features used MPEG-4 FAPs from 19 feature points; body/gesture features used EyesWeb motion cues (quantity of motion, contraction index, velocity, acceleration, fluidity of hand barycenter) and 80 motion features per gesture; speech features included intensity, pitch, MFCC, Bark spectral bands, voiced segment characteristics, and pause length, totaling 377 features.
  • Analysis: separate Bayesian classifiers were trained per modality in Weka, with normalization, MDL-based discretization, wrapper feature subset selection using best-first forward search, and cross-validation; two models were evaluated (all 240 samples including missing modalities vs. only samples available for all three modalities), and fusion was performed at feature level (single classifier) or decision level (combining posterior probabilities via best probability or majority voting plus best probability).

Limitations

  • Stated: the model using all 240 samples can handle missing modality data but is less precise; the model using only complete three-modality samples is more precise but less flexible.
  • Stated: the study used emotion-specific gestures; the authors note that an alternative approach would be to recognise emotions from different expressivity of the same gesture, not necessarily associated with a specific emotion, and leave this to future work.
  • Evident: the corpus consists of acted emotional expressions from only 10 subjects and 240 samples per modality, and the text states that no user or gender normalization was applied for pitch contour, which may limit generalization.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-0-387-74161-1_41

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 1 Not valid · 2 Not checked · 26 Own results and interpretations

10 more claims

Create a free account to see all 29 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 5
  • It may state two separate claims as one. 10
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 33 references

Research works printed but not linked: 4

  1. Banziger, T., Pirker, H., Scherer, K.: Gemep — geneva multimodal emotion portrayals: a corpus for the study of multimodal emotional expressions. In L. Deviller et al. (Ed.), Proceedings of LREC’06 Workshop on Corpora for Research on Emotion and Affect (pp. 15–019). Genoa. Italy (2006).
  2. Camurri, A., Coletta, P., Massari, A., Mazzarino, B., Peri, M., Ricchetti, M., Ricci, A. and Volpe, G.:Toward real-time multimodal processing: EyesWeb 4.0, in Proc. AISB 2004 Convention: Motion, Emotion and Cognition, Leeds, UK, March 2004.
  3. Castellano, G., Camurri, A., Mazzarino, B., Volpe, G.: A mathematical model to analyse the dynamics of gesture expressivity, in Proc. of AISB 2007 Convention: Artificial and Ambient Intelligence, Newcastle upon Tyne, UK, April 2007.
  4. Kessous, L., Amir, N.: Comparison of feature extraction approaches based on the Bark time/frequency representation for classification of expressive speechpaper submitted to Interspeech 2007.

Cited by

See all 225

Metadata

Authors
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K.
Affiliations
  • National Technical University of Athens (GR)
  • University of Genoa (IT)
Venue
IFIP The International Federation for Information Processing
Publisher
Springer US
Published
2007-11-09
Pages
375-388
OpenAlex
W1606749791
Cited by
225
References
37
Citations by year
2026: 2 · 2025: 8 · 2024: 4 · 2023: 9 · 2022: 28 · 2021: 12 · 2020: 11 · 2019: 14 · 2018: 12 · 2017: 18 · 2016: 16 · 2015: 20 · 2014: 13 · 2013: 11 · 2012: 14
FWCI
4.12
Type
conference paper
Language
en
Licence
other-oa
Locations
  • Springer eBooks (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, other-oa)
  • DSpace - NTUA (National Technical University of Athens) (submittedVersion, other-oa)
  • INFM-OAR (INFN Catania) (submittedVersion, other-oa)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
multimodal emotion recognition, facial expressions, body gestures, speech, Bayesian classifier, feature-level fusion, decision fusion, multimodal corpus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal emotion recognition from expressive faces, body gestures and speech. https://app.cochat.ai/papers/10.1007/978-0-387-74161-1_41
The paper
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K. (2007). Multimodal emotion recognition from expressive faces, body gestures and speech. IFIP The International Federation for Information Processing. https://doi.org/10.1007/978-0-387-74161-1_41

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.