Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 3 of the 31 sources its claims cite were checked for support.

A Multimodal Dataset for Mixed Emotion Recognition

Open access

Abstract

This paper introduces a multimodal dataset for mixed emotion recognition, combining frontal face video with physiological signals such as EEG, GSR, and PPG. It supports recognition of positive, negative, and mixed emotions and provides a benchmark for more realistic affective states.

Overview

CoChat Summary

TL;DR

This paper presents a multimodal dataset for mixed emotion recognition, collected while 80 college students watched 32 selected video clips intended to elicit positive, negative, or mixed emotions. After discarding 7 participants’ data, the final dataset contains EEG, GSR, PPG, frontal face video, and self-assessment data from 73 participants, and the authors report up to 80.96% average accuracy for 3-class emotion classification with SVM using all modalities.

Key findings

  • The dataset includes two versions of the data, raw and time-aligned, and after discarding 7 participants due to deficient physiological data, it contains data from 73 participants.
  • The final stimulus set contains 32 video clips: 8 positive, 8 negative, and 16 mixed.
  • In technical validation, the average accuracy for 3-class classification (positive, negative, mixed) reached 80.96% using SVM with features from all modalities.
  • For self-reports, positive videos had higher amusement than disgust, negative videos showed the opposite, and mixed videos had similar amusement and disgust ratings; PANAS also differed significantly across video types (e.g., positive vs. negative scores: W = 3133, p < 0.001; negative vs. positive: W = 32.5, p < 0.001).

Methods

  • The study used a rule-based video filtering step on the Stanford film library, followed by expert evaluation by four experienced experts to select clips likely to induce positive, negative, and mixed emotions.
  • 80 healthy college students participated; the experiment recorded EEG, GSR, PPG, and frontal face video while participants watched the selected clips and completed PANAS, VAD, amusement, and disgust self-ratings.
  • For validation, the authors performed self-assessment analyses and mixed emotion classification using SVM and random forest in a participant-dependent protocol, with feature extraction from physiological signals and face video.

Limitations

  • The paper states that the stimulus videos were not included in the repository because of copyright issues; only the names and access channel were provided.
  • The dataset is only available upon request and requires a Data Use Agreement for academic research use only.
  • Evident from the text: the classification validation is reported as a technical proof-of-concept, but the paper does not provide a full external replication study or cross-dataset validation; this is not explicitly stated as a limitation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41597-024-03676-4

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Scatter plots of Average NA versus Average PA for selected and non-selected videos. (a) Scatter plot for selected and non-selected positive videos. (b) Scatter plot for selected and non-selected negative videos. (c) Scatter plot for selected and non-selected mixed videos.
Fig. 1 Scatter plots of Average NA versus Average PA for selected and non-selected videos. (a) Scatter plot for selected and non-selected positive videos. (b) Scatter plot for selected and non-selected negative videos. (c) Scatter plot for selected and non-selected mixed videos.CC BY-NC-ND · PubMed Central
Fig. 2 Photos of devices used for signal data collection. (a) Wireless dry electrode EEG device DSI-24. (b) Intelligent wristband ES1 used for GSR and PPG data acquisition.
Fig. 2 Photos of devices used for signal data collection. (a) Wireless dry electrode EEG device DSI-24. (b) Intelligent wristband ES1 used for GSR and PPG data acquisition.CC BY-NC-ND · PubMed Central
Fig. 3 Schematic diagram of channel locations according to international 10-20 system.
Fig. 3 Schematic diagram of channel locations according to international 10-20 system.CC BY-NC-ND · PubMed Central
Fig. 4 Example samples of EEG, GSR, PPG, and Face video.
Fig. 4 Example samples of EEG, GSR, PPG, and Face video.CC BY-NC-ND · PubMed Central
Fig. 5 Timing diagram of the experiment. The experiment procedure mainly consisted of a practice stage, a baseline recording stage, and 4 blocks (each containing 8 trials). A set of arithmetic operations and a 1-minute break were arranged to eliminate the effect of the previous block on the current one.
Fig. 5 Timing diagram of the experiment. The experiment procedure mainly consisted of a practice stage, a baseline recording stage, and 4 blocks (each containing 8 trials). A set of arithmetic operations and a 1-minute break were arranged to eliminate the effect of the previous block on the current one.CC BY-NC-ND · PubMed Central
Fig. 6 Screenshot of EEG recording file 1_raw.csv.
Fig. 6 Screenshot of EEG recording file 1_raw.csv.CC BY-NC-ND · PubMed Central
Fig. 7 Distribution of rating scores for amusement and disgust of all participants for each stimuli video clip. The positive axis represents amusement scores while the negative axis represents disgust scores, both of which are measured through 5-point scales. The tips of the top whisker and bottom whisker mark the maximum and minimum rating scores. Boxes represent quartiles of the rating score distributions. Crosses mark the mean values, and the circle dots are outliers.
Fig. 7 Distribution of rating scores for amusement and disgust of all participants for each stimuli video clip. The positive axis represents amusement scores while the negative axis represents disgust scores, both of which are measured through 5-point scales. The tips of the top whisker and bottom whisker mark the maximum and minimum rating scores. Boxes represent quartiles of the rating score distributions. Crosses mark the mean values, and the circle dots are outliers.CC BY-NC-ND · PubMed Central
Fig. 8 Scatter plot of the average valence, arousal, and dominance rating scores of each video clip. (a) Scatter plot in valence-arousal-dominance space, (b) Projection of (a) onto the valence-arousal plane.
Fig. 8 Scatter plot of the average valence, arousal, and dominance rating scores of each video clip. (a) Scatter plot in valence-arousal-dominance space, (b) Projection of (a) onto the valence-arousal plane.CC BY-NC-ND · PubMed Central
Fig. 9 Distribution of the estimated heart rate for each subject.
Fig. 9 Distribution of the estimated heart rate for each subject.CC BY-NC-ND · PubMed Central
Fig. 10 Boxplot graph for the 3-class classification (i.e., positive, negative, and mixed) results of different combinations of features. (a) Results of using RF as the classifier. (b) Results of using SVM as the classifier.
Fig. 10 Boxplot graph for the 3-class classification (i.e., positive, negative, and mixed) results of different combinations of features. (a) Results of using RF as the classifier. (b) Results of using SVM as the classifier.CC BY-NC-ND · PubMed Central
Fig. 11 Confusion matrix of classification results obtained by using SVM and all features (i.e., features from GSR, PPG, EEG, and Video). Labels 0, 1, and 2 correspond to positive, negative, and mixed emotions. The number under each confusion matrix denotes the participant ID.
Fig. 11 Confusion matrix of classification results obtained by using SVM and all features (i.e., features from GSR, PPG, EEG, and Video). Labels 0, 1, and 2 correspond to positive, negative, and mixed emotions. The number under each confusion matrix denotes the participant ID.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

2 Valid · 1 Not valid · 14 Not checked · 43 Own results and interpretations

25 more claims

Create a free account to see all 60 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 16
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

1 of 1

Citations

Cites

1 retracted

Show all 59 references

Cited by

See all 37

Metadata

Authors
Yang, P., Liu, N., Liu, X., Shu, Y., Ji, W., Ren, Z., Sheng, J., Yu, M., Yi, R., Zhang, D., & Liu, Y.-J.
Affiliations
  • Tsinghua University (CN)
  • Tianjin University (CN)
  • Shanghai Jiao Tong University (CN)
Corresponding author
Liu, Yong-Jin
Venue
Scientific Data
Published
2024-08-05
Volume
11
Issue
1
Pages
847-847
ISSN
2052-4463 (ISSN-L)
OpenAlex
W4401333309
Cited by
37
References
59
Citations by year
2026: 21 · 2025: 15 · 2024: 1
Type
data paper
Language
en
Open access
Licence
Locations
  • Scientific Data (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • National Natural Science Foundation of China (U1736220)
  • Tsinghua University (Tsinghua University Initiative Scientific Research Program)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
Keywords
emotion induction, EEG, skin conductance, photoplethysmography, PANAS, VAD, affective computing
MeSH terms
Electroencephalography, Emotions, Facial Expression, Humans, Video Recording

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Multimodal Dataset for Mixed Emotion Recognition. https://app.cochat.ai/papers/10.1038/s41597-024-03676-4
The paper
Yang, P., Liu, N., Liu, X., Shu, Y., Ji, W., Ren, Z., Sheng, J., Yu, M., Yi, R., Zhang, D., & Liu, Y.-J. (2024). A Multimodal Dataset for Mixed Emotion Recognition. Scientific Data. https://doi.org/10.1038/s41597-024-03676-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.