NNIME: The NTHU-NTUA Chinese interactive multimodal emotion corpus
The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
The increasing availability of large-scale emotion corpus along with the advancement in emotion recognition algorithms have enabled the emergence of next-generation human-machine interfaces. This paper describes a newly-collected multimodal corpus, i.e., the NTHU-NTUA Chinese Interactive Emotion Corpus (NNIME). The database is a result of the collaborative work between engineers and drama experts. This database includes recordings of 44 subjects engaged in spontaneous dyadic spoken interactions. The multimodal data includes approximately 11-hour worth of audio, video, and electrocardiogram data recorded continuously and synchronously. The database is also completed with a rich set of emotion annotations on discrete and continuous-in-time annotation by a total of 49 annotators. Thees emotion annotations include a diverse perspectives: peer-report, director-report, self-report, and observer-report. This carefully-engineered data collection and annotation processes provide an additional valuable resource to quantify and investigate various aspects of affective phenomenon and human communication. To our best knowledge, the NNIME is one of the few large-scale Chinese affective dyadic interaction database that have been systematically collected, organized, and to be publicly-released to the research community.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Structurally, the NNIME dataset comprises 6701 sentences and includes both discrete and continuous emotion labels.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [29]
In the citing paper
“The NTHU-NTUA Chinese Interactive Emotion Corpus (NNIME) [29] is a groundbreaking initiative aimed at creating an interactive multimodal corpus for Chinese emotional expressions.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [29]
Citations
Cites
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Praat, a system for doing phonetics by computer
Boersma, P. (2002) · UvA-DARE (University of Amsterdam) · Cited by 4,417
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 6×
“Furthermore, due to its similarity in the design with the IEMOCAP and the CreativeIT, the idiosyncratic factor in the database design can be much mitigated.”
MethodSelf-citationAction Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Exploiting Generative Models in Discriminative Classifiers
Jaakkola, T., & Haussler, D. H. (1998) · Cited by 1,271
The devil is in the details: an evaluation of recent feature encoding methods
Chatfield, K., Lempitsky, V., Vedaldi, A., & Zisserman, A. (2011) · Procedings of the British Machine Vision Conference 2011 · doi:10.5244/c.25.76 · Cited by 834 · cited 1×
“In order to encode the sequence of LLDs to a fixed-length high-dimensional session-level features, we use the GMM based Fishervector encoding [30], [31], i.”
MethodIntroducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · cited 4×
“RECOLA [17] 46 6 Audio, Video, ECG, EDA Dyadic French Spont.”
BackgroundInfluentialFEELTRACE: an instrument for recording perceived emotion in real time
Cowie, R., Douglas‐Cowie, E., Savvidou, S. M., & McMahon, E. R. (2000) · Research Portal (Queen's University Belfast) · Cited by 539
Effects of Dyadic Interaction on Argumentive Reasoning
Kuhn, D., Shaw, V., & Felton, M. (1997) · Cognition and Instruction · doi:10.1207/s1532690xci1503_1 · Cited by 475 · cited 1×
“Dyadic interaction is a basic unit of face-to-face interaction providing an important gateway for humans to convey emotion, communicate information, and foster mutual understanding [4], [5], [6], [7].”
BackgroundMother and infant at play: The dyadic interaction involving facial, vocal, and gaze behaviors.
Stern, D. N. (1974) · Cited by 459
Impro: Improvisation and the Theatre
Pertalion, A., & Johnstone, K. (1984) · Theatre Journal · doi:10.2307/3206983 · Cited by 418 · cited 1×
“, a creative performance where actors collaborate and coordinate in real-time to create a coherent viewing experiences [8], within a pre-designed scenario setup offers a unique window into understanding the dyadic affective interaction processes.”
BackgroundDouglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., et al. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74889-2_43 · Cited by 321 · cited 4×
“In 2007, Douglas-Cowie et al. summarized an extensive analyses, e.g., on the perspective of scope, naturalness, and context, in the collection of the well-known HUMAINE audio-video emotion database (HUMAINE) [13].”
BackgroundInfluentialEnsemble methods for spoken emotion recognition in call-centres
Morrison, D., Wang, R., & De Silva, L. C. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.004 · Cited by 320
Behavioral Signal Processing: Deriving Human Behavioral Informatics From Speech and Language
Narayanan, S., & Georgiou, P. G. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2236291 · Cited by 294 · cited 1×
“There has already been a tremendous research effort in modeling humans using a variety of measurable signals, which aims at enabling machines to sense emotional states automatically [1], [2].”
BackgroundThe SEMAINE corpus of emotionally coloured character interactions
McKeown, G., Valstar, M. F., Cowie, R., & Pantic, M. (2010) · 2010 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2010.5583006 · Cited by 278 · cited 3×
“SEMAINE [16] 20 8 Audio, Video Dyadic English Script Conti.”
BackgroundInfluentialMutual influence in expressive behavior: Adult-adult and infant-adult dyadic interaction.
Cappella, J. N. (1981) · Psychological Bulletin · doi:10.1037//0033-2909.89.1.101 · Cited by 258 · cited 1×
“Dyadic interaction is a basic unit of face-to-face interaction providing an important gateway for humans to convey emotion, communicate information, and foster mutual understanding [4], [5], [6], [7].”
BackgroundAge and sex differences in dyadic and group interaction.
Benenson, J. F., Apostoleris, N. H., & Parnass, J. (1997) · Developmental Psychology · doi:10.1037//0012-1649.33.3.538 · Cited by 173
Using Actor Portrayals to Systematically Study Multimodal Emotion Expression: The GEMEP Corpus
Bänziger, T., & Scherer, K. R. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_42 · Cited by 157 · cited 1×
“Improvised acting has been considered as a viable research methodology for studying human emotions and communication [9], [10].”
BackgroundA new emotion database: considerations, sources and scope
Douglas‐Cowie, E., Cowie, R., & Schroeder, N. V. (2000) · Cited by 156
CHEAVD: a Chinese natural emotional audio–visual database
Li, Y., Tao, J., Chao, L., Bao, W., & Liu, Y. (2016) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-016-0406-z · Cited by 132 · cited 3×
“CHEAVD [18] 238 4 Audio, Video TV show Chinese Spont.”
BackgroundDesign, recording and verification of a danish emotional speech database
Engberg, I. S., Hansen, A. V., Andersen, O., & Dalsgaard, P. (1997) · 5th European Conference on Speech Communication and Technology (Eurospeech 1997) · doi:10.21437/eurospeech.1997-482 · Cited by 128 · cited 3×
“Some notable exemplary databases include: Engberg et al. recorded 30 minutes of scripted Danish speech to analyze the five major emotion categories, i.e., happy, sad, angry, neutral, and surprise (DES) [11].”
BackgroundInfluentialDifferent Discrete Wavelet Transforms Applied to Denoising Analytical Data
Cai, C., & Harrington, P. D. B. (1998) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci980210j · Cited by 78 · cited 1×
“We first utilized discrete wavelet transform (DWT) to remove baseline variability, motion artifact, and other high frequency noise such as electromyography (EMG) and powerline noise [25].”
MethodSpeech Emotion Recognition Based on Rough Set and SVM
Zhou, J., Wang, G., Yang, Y., & Chen, P. (2006) · IEEE International Conference on Cognitive Informatics · doi:10.1109/coginf.2006.365676 · Cited by 75 · cited 2×
“Zhou et al. collected 1200 utterances from 4 nonprofessional actors for six emotions and named the database as the CLDC [22].”
BackgroundThe USC Creative IT Database: A Multimodal Database of Theatrical Improvisation
Metallinou, A., Lee, C.-C., Busso, C., Carnicke, S. M., & Narayanan, S. S. (2010) · Cited by 74
Self-citationImpro: Improvisation and the Theatre
Johnstone, K., & Wardle, I. (2012) · Routledge eBooks · doi:10.4324/9780203446294 · Cited by 55
Building a Chinese Natural Emotional Audio-Visual Database
Bao, W., Li, Y., Gu, M., Yang, M., Li, H., Chao, L., & Tao, J. (2014) · International Conference on Signal Processing · doi:10.1109/icosp.2014.7015071 · Cited by 44 · cited 1×
“It contained 140 minutes of spontaneous emotional audio-video segments of data extracted from 238 speakers from films, TV plays, and talk shows with 26 nonprototypical emotional states labeled by four Chinese native speakers [18], [23].”
BackgroundSpeaker independent emotion recognition based on SVM/HMMS fusion system
Fu, L., Mao, X., & Chen, L. (2008) · International Conference on Audio, Language and Image Processing · doi:10.1109/icalip.2008.4590144 · Cited by 39 · cited 1×
“recorded 7 actors, and each with 20 spoken utterances of fixed emotion categories [21].”
BackgroundDiscrete Wavelet Soft Threshold Denoise Processing for ECG Signal
Hai-bing, Q., Xiongfei, L., & Pan, C. (2010) · International Conference on Intelligent Computation Technology and Automation · doi:10.1109/icicta.2010.404 · Cited by 15 · cited 1×
“We used eight-level DWT (db4 as the chosen mother wavelet function) decomposition with soft thresholding technique to eliminate high frequency noise [26].”
MethodA patch-sized wearable ECG/respiration recording platform with DSP capability
Jhuang, J.-W., & Ma, H. (2015) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom.2015.7454515 · Cited by 13 · cited 1×
“Electrocardiogram (ECG) Recording: In every session, each actor wore a multichannel physiological integrated circuit from Texas Instrument (ADS1292R) as the analog front-end circuit [24].”
MethodSelf-citation
Research works printed but not linked: 5
- busso, 2008, Recording audio-visual emotional databases from actors: a closer look, Second International Workshop on Emotion Corpora for Research on Emotion and Affect International conference on Language Resources and Evaluation (LREC 2008), 17
- knapp, 2013, Nonverbal communication in human interaction Cengage Learning
- 10.1037/0033-2909.89.1.101 · doi:10.1037/0033-2909.89.1.101 · Paper page
- 10.1037/0012-1649.33.3.538 · doi:10.1037/0012-1649.33.3.538 · Paper page
- nogueiras, 2002, Speech emotion recognition using hidden markov models, speech prosody 2002, an International Conference France, 11
Cited by
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“The NTHU-NTUA Chinese Interactive Emotion Corpus (NNIME) [29] is a ground-breaking initiative aimed at creating an interactive multimodal corpus for Chinese emotional expressions.”
BackgroundSpeech Emotion Recognition Using Deep Neural Network Considering Verbal and Nonverbal Speech Sounds
Huang, K.-Y., Wu, C.-H., Hong, Q.-B., Su, M.-H., & Chen, Y.-H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682283 · Cited by 116
“Different from other scripted emotion speech database or segmented audio from dramas, NTHU-NTUA Chinese Interactive Multimodal Emotion Corpus (NNIME) [18], one of the few large-scale Chinese affective dyadic interaction database which contains various emotional nonverbal sounds, such as laughter,”
BackgroundInfluentialMulti-modal emotion recognition using EEG and speech signals
Wang, Q., Wang, M., Yang, Y., & Zhang, X. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105907 · Cited by 100
George, S. M., & Ilyas, P. M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127015 · Cited by 125
Speech Emotion Recognition Considering Nonverbal Vocalization in Affective Conversations
Hsu, J.-H., Su, M.-H., Wu, C., & Chen, Y. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3076364 · Cited by 84
Metadata
- Authors
- Chou, H.-C., Lin, W.-C., Chang, L.-C., Li, C.-C., Ma, H.-P., & Lee, C.-C.
- Affiliations
- National Tsing Hua University (TW)
- National Taiwan University of Arts (TW)
- Venue
- 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 292-298
- ISSN
- 2156-8103 (ISSN-L)
- OpenAlex
- W2785952417
- Cited by
- 57
- References
- 29
- Citations by year
- 2026: 4 · 2025: 10 · 2024: 13 · 2023: 5 · 2022: 6 · 2021: 9 · 2020: 1 · 2019: 5 · 2018: 2 · 2017: 2
- FWCI
- 3.14
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Affective Computing and Intelligent Interaction and workshops (publishedVersion)
- Funders
- National Taiwan University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Linguistics and Language › Linguistic Studies and Language Acquisition
- Keywords
- dyadic interaction, spontaneous conversation, emotion annotation, electrocardiogram, human-machine interaction, emotional communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.