Multimodal emotion recognition from expressive faces, body gestures and speech
Open accessThe DOI matches its Crossref record; 33 of the 37 references it cites were found in a registry or an index; 1 of the 2 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This paper presents a Bayesian multimodal emotion recognition system that combines facial expressions, body gestures, and speech to classify eight acted emotions. Fusing modalities improved recognition over the best unimodal system by more than 10%, with feature-level fusion outperforming decision-level fusion.
Key findings
- Multimodal feature-level fusion reached 78.3% accuracy when all 240 samples were used and 89.4% when only samples available for all three modalities were used, compared with the best unimodal system (gestures: 67.1% and 83.2%, respectively).
- Feature-level fusion outperformed decision-level fusion: 78.3% vs. 74.6% with all samples; with complete three-modality samples, feature-level was 89.4%, while decision-level was 85.1% using best probability and 88.20% using majority voting plus best probability.
- Unimodal performance with all samples was 67.1% for gestures, 57.1% for speech, and 48.3% for facial expressions; with only complete three-modality samples, these rose to 83.2%, 70.8%, and 59.6%, respectively.
- In the feature-level multimodal confusion matrix, the paper reports that all emotions apart from despair can be recognised with more than 70% accuracy.
Methods
- Data: a multimodal corpus collected during the Third Summer School of the HUMAINE EU-IST project in Genova, September 2006, based on the GEMEP corpus; 10 participants of five nationalities acted eight emotions (anger, despair, interest, pleasure, sadness, irritation, joy, pride), each emotion acted three times by each actor, yielding 240 posed samples per modality.
- Features: facial features used MPEG-4 FAPs from 19 feature points; body/gesture features used EyesWeb motion cues (quantity of motion, contraction index, velocity, acceleration, fluidity of hand barycenter) and 80 motion features per gesture; speech features included intensity, pitch, MFCC, Bark spectral bands, voiced segment characteristics, and pause length, totaling 377 features.
- Analysis: separate Bayesian classifiers were trained per modality in Weka, with normalization, MDL-based discretization, wrapper feature subset selection using best-first forward search, and cross-validation; two models were evaluated (all 240 samples including missing modalities vs. only samples available for all three modalities), and fusion was performed at feature level (single classifier) or decision level (combining posterior probabilities via best probability or majority voting plus best probability).
Limitations
- Stated: the model using all 240 samples can handle missing modality data but is less precise; the model using only complete three-modality samples is more precise but less flexible.
- Stated: the study used emotion-specific gestures; the authors note that an alternative approach would be to recognise emotions from different expressivity of the same gesture, not necessarily associated with a specific emotion, and leave this to future work.
- Evident: the corpus consists of acted emotional expressions from only 10 subjects and 240 samples per modality, and the text states that no user or gender normalization was applied for pitch contour, which may limit generalization.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 2 Not checked · 26 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fusing multimodal data increased very much the recognition rates in comparison with the unimodal systems: the multimodal approach gave an improvement of more than 10% with respect to the most successful unimodal system.”
- Also stated in: § 6.4 Feature-level fusion · plain
“The overall performance of this classifier was 78.3 % (it increases up to 89.4 % when only the samples available for all the modalities are used), which is much higher than the performance obtained by the most successful unimodal system, the one based on gestures.”
- Also stated in: § 7 Discussion and conclusions · plain
“Fusing multimodal data increased very much the recognition rates in comparison with the unimodal systems: the multimodal approach gave an improvement of more than 10 % compared to the performance of the system based on gestures, when all the 240 samples are used.”
plainstates a causen = 240- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 6.4 Feature-level fusion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Anger and sadness are classified with high accuracy (93.33 and 76.67% respectively).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have selected the latter since the recognition rate achieved was superior.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Considering the performances of the unimodal emotion recognition systems, the one based on gestures appears to be the most successful, followed by the one based on speech and the one based on facial expressions.”
hedgedstates an associationn = 240- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Anger and pride are recognised with very high accuracy (80 and 96.67 % respectively).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
10 more claims
Create a free account to see all 29 claims and every source verdict.
- Probably not a substantial claim. 5
- It may state two separate claims as one. 10
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“ML also permits integration of multiple sets of these cues to further enhance emotion detection (Caridakis et al., 2007).”
From Applications of machine learning in animal behaviour studies (2017) · cites this paper as Caridakis et al., 2007
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“The overall performance of this classifier was 57.1 (it increases up to 70.8 % when only the samples available for all the modalities are used).”
Reported 57.1 · recomputed 58.3325
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.4 Feature-level fusion
“Anger was the emotion recognised with highest accuracy, as in all the unimodal systems.”
§ 6.4 Feature-level fusion
“| 0 | 0 | 0 | 0 | 93.33 | 0 | 6.67 | 0 | e | Joy |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.4 Feature-level fusion
“Anger was the emotion recognised with highest accuracy, as in all the unimodal systems.”
§ 6.2 Emotion recognition from gestures
“Anger and pride are recognised with very high accuracy (80 and 96.67 % respectively).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.4 Feature-level fusion
“The diagonal components reveal that all the emotions, apart from despair, can be recognised with more than the 70 % of accuracy.”
§ 6.4 Feature-level fusion
“| 0 | 3.33 | 3.33 | 13.33 | 3.33 | 70 | 6.67 | 0 | f | Pleasure |”
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · cited 1×
“Facial expressions [12] [13], vocal features [14] [15], body movements and postures [16] [17] [18], physiological signals [19] have been used as inputs during these attempts, while multimodal emotion recognition is currently gaining ground [20] [21] [22].”
MethodAutomatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1×
“Facial expressions [5], vocal features [6], body movements [7] have been used as inputs during these attempts, while multimodal emotion recognition is currently gaining ground [8].”
MethodHandbook of Discourse Analysis
Dittmar, N., & Forsthoffer, I. (2006) · Journal of Pragmatics · doi:10.1016/j.pragma.2006.03.004 · Cited by 1,549
Acoustic profiles in vocal emotion expression
Banse, R., & Scherer, K. R. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.3.614 · Cited by 1,546
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 1×
“Facial expressions [12] [13], vocal features [14] [15], body movements and postures [16] [17] [18], physiological signals [19] have been used as inputs during these attempts, while multimodal emotion recognition is currently gaining ground [20] [21] [22].”
MethodToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
The Handbook of Discourse Analysis
(2003) · The Handbook of Discourse Analysis · doi:10.1111/b.9780631205968.2003.00029.x · Cited by 604
Research works printed but not linked: 4
- Banziger, T., Pirker, H., Scherer, K.: Gemep — geneva multimodal emotion portrayals: a corpus for the study of multimodal emotional expressions. In L. Deviller et al. (Ed.), Proceedings of LREC’06 Workshop on Corpora for Research on Emotion and Affect (pp. 15–019). Genoa. Italy (2006).
- Camurri, A., Coletta, P., Massari, A., Mazzarino, B., Peri, M., Ricchetti, M., Ricci, A. and Volpe, G.:Toward real-time multimodal processing: EyesWeb 4.0, in Proc. AISB 2004 Convention: Motion, Emotion and Cognition, Leeds, UK, March 2004.
- Castellano, G., Camurri, A., Mazzarino, B., Volpe, G.: A mathematical model to analyse the dynamics of gesture expressivity, in Proc. of AISB 2007 Convention: Artificial and Ambient Intelligence, Newcastle upon Tyne, UK, April 2007.
- Kessous, L., Amir, N.: Comparison of feature extraction approaches based on the Bark time/frequency representation for classification of expressive speechpaper submitted to Interspeech 2007.
Cited by
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Fusing audio, visual and textual clues for sentiment analysis from multimodal content
Poria, S., Cambria, E., Howard, N., Huang, G.-B., & Hussain, A. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.01.095 · Cited by 534
Applications of machine learning in animal behaviour studies
Valletta, J. J., Torney, C., Kings, M., Thornton, A., & Madden, J. (2017) · Animal Behaviour · doi:10.1016/j.anbehav.2016.12.005 · Cited by 505
“ML also permits integration of multiple sets of these cues to further enhance emotion detection (Caridakis et al., 2007).”
Survey on Emotional Body Gesture Recognition
Noroozi, F., Corneanu, C. A., Kaminska, D., Sapinski, T., Escalera, S., & Anbarjafari, G. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2874986 · Cited by 455
“Historical works usually focused on simple fusion techniques of predefined body and face representations [127], [128].”
BackgroundCambria, E., Livingstone, A., & Hussain, A. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-34584-5_11 · Cited by 306
“Few tentative efforts to detect non-basic affective states, such as fatigue, anxiety, satisfaction, confusion, or frustration, have been also made [17–22] (Table.”
Background
Metadata
- Authors
- Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K.
- Affiliations
- National Technical University of Athens (GR)
- University of Genoa (IT)
- Venue
- IFIP The International Federation for Information Processing
- Publisher
- Springer US
- Published
- 2007-11-09
- Pages
- 375-388
- OpenAlex
- W1606749791
- Semantic Scholar
- 08f6350184a7dc97352664979c7f1a2e1bc00725
- Cited by
- 225
- References
- 37
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 4 · 2023: 9 · 2022: 28 · 2021: 12 · 2020: 11 · 2019: 14 · 2018: 12 · 2017: 18 · 2016: 16 · 2015: 20 · 2014: 13 · 2013: 11 · 2012: 14
- FWCI
- 4.12
- Type
- conference paper
- Language
- en
- Open access
- Licence
- other-oa
- Locations
- Springer eBooks (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- DSpace - NTUA (National Technical University of Athens) (submittedVersion, other-oa)
- INFM-OAR (INFN Catania) (submittedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- multimodal emotion recognition, facial expressions, body gestures, speech, Bayesian classifier, feature-level fusion, decision fusion, multimodal corpus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal emotion recognition from expressive faces, body gestures and speech. https://app.cochat.ai/papers/10.1007/978-0-387-74161-1_41
- The paper
- Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., & Karpouzis, K. (2007). Multimodal emotion recognition from expressive faces, body gestures and speech. IFIP The International Federation for Information Processing. https://doi.org/10.1007/978-0-387-74161-1_41
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.