Context-Dependent Sentiment Analysis in User-Generated Videos
Open accessAbstract
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, Louis-Philippe Morency. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an LSTM-based contextual model for utterance-level multimodal sentiment analysis that lets utterances share information within a video. It reports 5–10% improvement over the state of the art on three benchmark datasets and better generalization in person-independent and cross-dataset settings.
Key findings
- Contextual LSTM variants outperform the context-independent uni-SVM baseline by 2% to 5% on all datasets, and the proposed method outperforms the state of the art by 5–10% on three benchmark datasets.
- Hierarchical fusion outperforms non-hierarchical fusion; bc-LSTM consistently performs better than sc-LSTM, and adding the dense layer improves performance by 0.3%–1.5% on MOSI/MOUD and by 1%–5% over h-LSTM on IEMOCAP.
- Bimodal and trimodal models outperform unimodal models; audio generally performs better than visual, text is best on MOSI and IEMOCAP, and on MOUD text is worse than audio due to translation noise, with Spanish word vectors improving text-CNN by 10%.
- In cross-dataset MOSI→MOUD, visual modality generalizes better than audio and text under language mismatch; for bc-LSTM, visual accuracy is 49.6% vs. audio 47.2% and text 46.9%.
Methods
- Person-independent experiments on MOSI (93 people; 1447 train and 752 test utterances), MOUD (55 persons; 59 train/validation videos), IEMOCAP (10 speakers; 4290 train and 1208 test utterances), plus a MOSI→MOUD cross-dataset split.
- Context-independent unimodal features are extracted with text-CNN using 300-dimensional word2vec and a 50-word window, openSMILE audio features (6373 features), and 3D-CNN visual features.
- These features are fed into contextual LSTM variants (sc-LSTM, h-LSTM, bc-LSTM); hierarchical fusion uses per-modality Level-1 LSTMs and a Level-2 LSTM over concatenated outputs, trained with categorical cross-entropy, dropout, padding/masking, RMSprop, and final SVM classification.
Limitations
- The paper states that acted data like IEMOCAP can suffer from biased labeling and incorrect acting, causing poor generalizability; cross-lingual MOSI→MOUD performance is poor for audio and text due to language mismatch; and no other utterance-level dataset was available for a similar emotion-recognition cross-dataset experiment.
- The paper states that qualitative errors arise from face occlusion or noisy audio, and that weak, non-contextual sentiment can bias predictions toward surrounding utterances.
- The paper states that the hierarchical framework is trained Level-1 and Level-2 successively but separately, not end-to-end; it also uses only two sentiment classes for MOSI and drops the neutral label for MOUD.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 21 cited sources checked · 0 supported · 0 flagged
- Findingown data
The proposed method shows high robustness to generalizability.
“Our method shows 5-10% performance improvement over the state of the art and high robustness to generalizability.”
Quote in the paperSubstantial claim- Rests on an assumption: Most research in multimodal sentiment analysis is performed on datasets with speaker overlap in train and test splits.
- Rests on an assumption: Each individual has a unique way of expressing emotions and sentiments.
- Rests on an assumption: It is very difficult to come up with a generalized model from the behavior of a limited number of individuals.
- Findingown data
The proposed framework outperformed the state of the art on three benchmark datasets by 5-10%.
“Our method shows 5-10% performance improvement over the state of the art and high robustness to generalizability.”
- Also stated in: § 1 Introduction · plain
“Experimental results show that the proposed framework has outperformed the state of the art on three benchmark datasets by 5-10%.”
- Also stated in: § B. Contextual Unimodal and Multimodal Classification · boosted
“We experimentally show that this proposed framework improves the performance of utterance-level sentiment classification over traditional frameworks.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingnone given
On the IEMOCAP dataset, the unimodal and multimodal classifiers obtained poor performance in classifying neutral utterances.
“On IEMOCAP dataset, the unimodal and multimodal classifiers obtained poor performance to classify neutral utterances.”
Quote in the paperSubstantial claim - Findingreasoning
The non-hierarchical model outperforms the baseline uni-SVM.
“The non-hierarchical model outperforms the baseline uni-SVM, which confirms that it is the contextsensitive learning paradigm that plays the key role in improving performance over the baseline.”
Quote in the paperSubstantial claim - Findingreasoning
In the cross-dataset MOSI→MOUD experiment, performance was poor for the audio and textual modalities.
“The performance was poor for audio and textual modality as the MOUD dataset is in Spanish while the model is trained on MOSI dataset, which is in English language.”
Quote in the paperSubstantial claim
- Findingnone given
On the MOSI and IEMOCAP datasets, the textual classifier achieves the best performance over other unimodal classifiers.
“On MOSI and IEMOCAP datasets, the textual classifier achieves the best performance over other unimodal classifiers.”
Quote in the paperNot a substantial claim - Findingnone given
Bimodal and trimodal models outperformed unimodal models in all kinds of experiments.
“As expected, in all kinds of experiments, bimodal and trimodal models have outperformed unimodal models.”
Quote in the paperSubstantial claim - Findingown data
The performance improvement from the dense layer is in the range of 0.3% to 1.5% on the MOSI and MOUD datasets.
“The performance improvement is in the range of 0.3% to 1.5% on MOSI and MOUD datasets.”
Quote in the paperNot a substantial claim - Findingown data
In the cross-lingual MOSI→MOUD experiment, the visual modality performs better than the other two modalities.
“However, notably the visual modality performs better than the other two modalities in this experiment, which means that in cross-lingual scenarios facial expressions carry more generalized, robust information than audio and textual modalities.”
Quote in the paperSubstantial claim - Findingown data
In cases where sentiment is very weak and non-contextual, the proposed approach shows bias towards its surrounding utterances, which leads to wrong predictions.
“Also, in cases where sentiment is very weak and non contextual, the proposed approach shows some bias towards its surrounding utterances, which further leads to wrong predictions.”
Quote in the paperNot a substantial claim - Findingown data
The proposed method outperforms (Poria et al., 2015) by a significant margin.
“Experimental results in Table 4 show that the proposed method outperformes (Poria et al., 2015) by a significant margin.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Poria et al., 2015
Not checked in the free run.
- Findingreasoning
bc-LSTM performs consistently better on all datasets than sc-LSTM.
“Since bc-LSTM has access to both the preceding and following information of the utterance sequence, it performs consistently better on all the datasets over sc-LSTM.”
Quote in the paperSubstantial claim - Findingnone given
In some cases, the predictions of the proposed method are wrong because of face occlusion or noisy audio.
“In some cases, the predictions of the proposed method are wrong because of face occlusion or noisy audio.”
Quote in the paperNot a substantial claim - Findingown data
On the IEMOCAP dataset, the performance improvement of bc-LSTM and sc-LSTM over h-LSTM is in the range of 1% to 5%.
“On the IEMOCAP dataset, the performance improvement of bc-LSTM and sc-LSTM over h-LSTM is in the range of 1% to 5%.”
Quote in the paperNot a substantial claim - Findingnone given
The audio modality performed better than the visual modality on all datasets.
“Overall, audio modality has performed better than visual on all the datasets.”
Quote in the paperNot a substantial claim - Findingnone given
Using Spanish word vectors in text-CNN results in an improvement of 10%.
“Using Spanish word vectorsfoot_1 in text-CNN results in an improvement of 10%.”
Quote in the paperNot a substantial claim - Findingown data
Such contextual relationships are prevalent throughout the dataset.
“Such contextual relationships are prevalent throughout the dataset.”
Quote in the paperSubstantial claim - Findingown data
Every LSTM network variant outperformed the baseline uni-SVM on all datasets by a margin of 2% to 5%.
“Every LSTM network variant has outperformed the baseline uni-SVM on all the datasets by the margin of 2% to 5% (see Table 3).”
Quote in the paperSubstantial claim - Findingreasoning
On the MOUD dataset, the textual modality performs worse than the audio modality.
“On the MOUD dataset, the textual modality performs worse than audio modality due to the noise introduced in translating Spanish utterances to English.”
Quote in the paperSubstantial claim - Findingown data
The dense layer increases performance.
“The usefulness of the dense layer in increasing the performance is evident from the experimental results shown in Table 3.”
Quote in the paperNot a substantial claim - Findingreasoning
The hierarchical fusion framework outperforms the non-hierarchical framework when using the bc-LSTM network.
“As expected, trained contextual unimodal features help the hierarchical fusion framework to outperform the non-hierarchical framework.”
Quote in the paperSubstantial claim - Interpretationreasoning
When classifying one utterance, other utterances can provide important contextual information.
“In particular, we claim that, when classifying one utterance, other utterances can provide important contextual information.”
Quote in the paperNot a substantial claim- Rests on an assumption: In sequence classification, the classification of each member is dependent on the other members.
- Study design and conductreasoning
Train/test splits of the datasets are completely disjoint with respect to speakers.
“To this end, we perform person-independent experiments to study generalization of our model, i.e., our train/test splits of the datasets are completely disjoint with respect to speakers.”
Quote in the paperSubstantial claim - Premisecitation
A combination of text and video data helps to create a more robust emotion and sentiment analysis model.
“Thus, a combination of text and video data helps to create a more robust emotion and sentiment analysis model (Poria et al., 2017a).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Poria et al., 2017a
Not checked in the free run.
- Interpretationreasoning
The proposed LSTM-based model enables utterances to capture contextual information from their surroundings in the same video, thus aiding the classification process.
“In this paper, we propose a LSTM-based model that enables utterances to capture contextual information from their surroundings in the same video, thus aiding the classification process.”
- Also stated in: § 1 Introduction · plain
“Our model preserves the sequential order of utterances and enables consecutive utterances to share information, thus providing contextual information to the utterance-level sentiment classification process.”
- Also stated in: § 3 Method · plain
“In this work, we propose a LSTM network that takes as input the sequence of utterances in a video and extracts contextual unimodal and multimodal features by modeling the dependencies among the input utterances.”
Quote in the paperSubstantial claim- Rests on an assumption: In sequence classification, the classification of each member is dependent on the other members.
- Rests on an assumption: Every utterance in a video is spoken at a distinct time and in a particular order.
- Also stated in: § 1 Introduction · plain
- Premisecitation
The hypothesis of independence of tokens has many limitations.
“The hypothesis of the independence of tokens is quite popular in information retrieval and data mining, e.g., bag-of-words model, but it has a lot limitations (Cambria and White, 2014).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Cambria and White, 2014
Not checked in the free run.
- Premisecitation
Sequential utterances of a video may largely be contextually correlated and influence each other's sentiment distribution.
“Like any other sequence classification problem (Collobert et al., 2011), sequential utterances of a video may largely be contextually correlated and, hence, influence each other's sentiment distribution.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Collobert et al., 2011
Not checked in the free run.
- Interpretationreasoning
The poor cross-dataset performance for the audio and textual modalities is because MOUD is in Spanish while the model is trained on the English MOSI dataset.
“The performance was poor for audio and textual modality as the MOUD dataset is in Spanish while the model is trained on MOSI dataset, which is in English language.”
Quote in the paperSubstantial claim - Interpretationreasoning
The worse performance of the textual modality on MOUD is due to noise introduced in translating Spanish utterances to English.
“On the MOUD dataset, the textual modality performs worse than audio modality due to the noise introduced in translating Spanish utterances to English.”
Quote in the paperSubstantial claim - Interpretationcitation
The higher performance improvement on IEMOCAP indicates the necessity of modeling long-range dependencies among utterances.
“The higher performance improvement on the IEMO-CAP dataset indicates the necessity of modeling long-range dependencies among the utterances as continuous emotion recognition is a multiclass sequential problem where a person does not frequently change emotions (Wöllmer et al., 2008).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wöllmer et al., 2008
Not checked in the free run.
- Interpretationown data
In cross-lingual scenarios, facial expressions carry more generalized, robust information than audio and textual modalities.
“However, notably the visual modality performs better than the other two modalities in this experiment, which means that in cross-lingual scenarios facial expressions carry more generalized, robust information than audio and textual modalities.”
Quote in the paperSubstantial claim - Interpretationown data
Modeling contextual dependencies among utterances improves classification.
“These results prove our initial hypothesis that modeling the contextual dependencies among utterances (which uni-SVM cannot do) improves the classification.”
Quote in the paperSubstantial claim- Rests on an assumption: uni-SVM cannot model contextual dependencies among utterances.
- Interpretationown data
The context-sensitive learning paradigm plays the key role in improving performance over the baseline.
“The non-hierarchical model outperforms the baseline uni-SVM, which confirms that it is the contextsensitive learning paradigm that plays the key role in improving performance over the baseline.”
Quote in the paperSubstantial claim- Rests on an assumption: uni-SVM cannot model contextual dependencies among utterances.
- Interpretationown data
Considering context dependency is of prime importance for utterance-level sentiment classification.
“The need for considering context dependency (see Section 1) is of prime importance for utterancelevel sentiment classification.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 29
- May combine two claims. 10
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenUsing a gated recurrent unit (GRU) instead of LSTM did not improve the performance.
“It should be noted that using a gated recurrent unit (GRU) instead of LSTM did not improve the performance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone givenThe proposed method has outperformed the state of the art and showed significant performance improvement over the baseline.
“The proposed method has outperformed the state of the art and showed significant performance improvement over the baseline.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone givenThe baseline unimodal-SVM and the state of the art fail to classify the utterance 'What would have been a better name for the movie' correctly.
“The baseline unimodal-SVM and state of the art fail to classify this utterance correctlyfoot_2 .”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe IEMOCAP database contains categorical labels anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and other.
“The database contains the following categorical labels: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and other, but we take only the first four so as to compare with the state of the art (Rozgic et al., 2012).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Busso et al., 2008
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe MOSI dataset contains 93 people reviewing topics in English.
“MOSI The MOSI dataset (Zadeh et al., 2016) is a dataset rich in sentimental expressions where 93 people review topics in English.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zadeh et al., 2016
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenHyper-parameters tuning is done on the training set by splitting it into train and validation components with 80 20% split.
“Hyper-parameters tuning is done on the training set by splitting it into train and validation components with 80 20% split.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationMOSI videos are segmented and each segment's sentiment label is scored between +3 and -3 by 5 annotators.
“The videos are segmented with each segments sentiment label scored between +3 (strong positive) to -3 (strong negative) by 5 annotators.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zadeh et al., 2016
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoningPadding is introduced to serve as neutral utterances, and bit masking is done on these padded utterances to eliminate their effect in the network.
“As the videos do not have the same number of utterances, padding is introduced to serve as neutral utterances.To avoid the proliferation of noise within the network, bit masking is done on these padded utterances to eliminate their effect in the network.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMOUD utterances are labeled positive, negative, or neutral.
“The utterances are labeled to be either positive, negative or neutral.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Pérez-Rosas et al., 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenDropout between the LSTM cell and dense layer is introduced to avoid overfitting.
“As a regularization method, dropout between the LSTM cell and dense layer is introduced to avoid overfitting.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationCurrent research indicates the benefit of using such networks to incorporate contextual information in the classification process.
“Current research (Zhou et al., 2016) indicates the benefit of using such networks to incorporate contextual information in the classification process.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Zhou et al., 2016
Not checked in the free run.
May combine two claims.
PremisecitationState-of-the-art multimodal sentiment analysis approaches treat utterances independently and ignore the order and dependencies among utterances in a video.
“Current research considers utterances as independent entities, i.e., ignores the interdependencies and relations among the utterances of a video.”
- Also stated in: § 1 Introduction · plain
“State-of-the-art approaches in this area treat utterances independently and ignore the order of utterances in a video (Cambria et al., 2017b).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Cambria et al., 2017b
Not checked in the free run.
- Also stated in: § 1 Introduction · plain
Classified as probably not a substantial claim. May combine two claims.
PremisecitationLSTM cells are capable of modeling long-range dependencies, which other traditional RNNs fail to do given the vanishing gradient issue.
“Specifically, LSTM cells are capable of modeling long-range dependencies, which other traditional RNNs fail to do given the vanishing gradient issue.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hochreiter and Schmidhuber, 1997
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationIEMOCAP contains the acts of 10 speakers in a two-way conversation segmented into utterances.
“IEMOCAP The IEMOCAP (Busso et al., 2008) contains the acts of 10 speakers in a twoway conversation segmented into utterances.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Busso et al., 2008
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe MOUD reviews are in Spanish.
“The reviews are in Spanish (we used Google Translate APIfoot_0 to get the English transcripts).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Pérez-Rosas et al., 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe convolution of the CNN over the utterance learns abstract representations of phrases equipped with implicit semantic information, which with each successive layer spans over increasing number of words and ultimately the entire utterance.
“The convolution of the CNN over the utterance learns abstract representations of the phrases equipped with implicit semantic information, which with each successive layer spans over increasing number of words and ultimately the entire utterance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe MOUD dataset contains product review videos provided by 55 persons.
“MOUD This dataset (Pérez-Rosas et al., 2013) contains product review videos provided by 55 persons.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Pérez-Rosas et al., 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe conversations in IEMOCAP are in English.
“The medium of the conversations in all the videos is English.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Busso et al., 2008
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationBoth early bimodal audio-visual systems yielded higher accuracy than any unimodal system.
“Both works showed that a bimodal system yielded a higher accuracy than any unimodal system.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites De Silva et al., 1997
Not checked in the free run.
- cites Chen et al., 1998
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThe performance of the second level of the hierarchical network banks on the quality of the features from the previous level, with better features aiding the fusion process.
“The performance of the second level banks on the quality of the features from the previous level, with better features aiding the fusion process.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
InterpretationreasoningContextual unimodal features can further improve performance of the multimodal fusion framework.
“Contextual unimodal features can further improve performance of the multimodal fusion framework explained in Section 3.3.1.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenNo other utterance-level dataset apart from IEMOCAP was available at the time of the experiments for emotion recognition.
“We could not carry out a similar experiment for emotion recognition as no other utterance-level dataset apart from the IEMO-CAP was available at the time of our experiments.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenTo curate IEMOCAP, subjects were provided affect-related scripts and asked to act.
“To curate the IEMOCAP dataset, instead, subjects were provided affect-related scripts and asked to act.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Premisenone givenBoth MOSI and MOUD are used for sentiment classification but consist of review videos spoken in different languages, English and Spanish respectively.
“Both MOSI and MOUD are used for the sentiment classification task but they consist of review videos spoken in different languages, i.e., English and Spanish, respectively.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductcitationThe authors report results using translated utterances as opposed to utterances trained on Spanish word vectors in order to make a fair comparison with (Poria et al., 2015).
“Nonetheless, we report results using these translated utterances as opposed to utterances trained on Spanish word vectors, in order to make fair comparison with (Poria et al., 2015).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Poria et al., 2015
Not checked in the free run.
May combine two claims.
PremisecitationContinuous emotion recognition is a multiclass sequential problem where a person does not frequently change emotions.
“The higher performance improvement on the IEMO-CAP dataset indicates the necessity of modeling long-range dependencies among the utterances as continuous emotion recognition is a multiclass sequential problem where a person does not frequently change emotions (Wöllmer et al., 2008).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wöllmer et al., 2008
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductnone givenIn Table 3, the results in bold are statistically significant (p < 0.05) compared to uni-SVM.
“In Table 3, the results in bold are statistically significant (p < 0.05) in compare to uni-SVM.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenIEMOCAP is annotated with emotion labels, unlike MOSI and MOUD.
“IEMOCAP dataset is different from MOSI and MOUD since it is annotated with emotion labels.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationActed datasets like IEMOCAP can suffer from biased labeling and incorrect acting, which can further cause poor generalizability of models trained on acted datasets.
“As pointed out by Poria et al. (Poria et al., 2017a), acted dataset like IEMOCAP can suffer from biased labeling and incorrect acting which can further cause the poor generalizability of the models trained on the acted datasets.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Poria et al., 2017a
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductnone givenThe datasets' individual configuration and splits are the same throughout all experiments.
“It should be noted that the datasets' individual configuration and splits are same throughout all the experiments (i.e., context-independent unimodal feature extraction, LSTM-based contextdependent unimodal and multimodal feature extraction and classification).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenUsing the proposed model on reviews from different languages allows analysis of its robustness and generalizability.
“The proposed model being used on reviews from different languages allows us to analyze its robustness and generalizability.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenMOSI and MOUD were developed by crawling consumers' spontaneous online product review videos from popular social websites and later labeled with sentiment labels.
“These two datasets were developed by crawling consumers' spontaneous online product review videos from popular social websites and later labeled with sentiment labels.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Most research in multimodal sentiment analysis is performed on datasets with speaker overlap in train and test splits.
“Most of the research in multimodal sentiment analysis is performed on datasets with speaker overlap in train and test splits.”
The paper relies on this without support.
Each individual has a unique way of expressing emotions and sentiments.
“Because each individual has a unique way of expressing emotions and sentiments, however, finding generic, personindependent features for sentiment analysis is very important.”
The paper relies on this without support.
The primary advantage of analyzing videos over text is the surplus of behavioral cues present in vocal and visual modalities.
“The primary advantage of analyzing videos over text is the surplus of behavioral cues present in vocal and visual modalities.”
The paper relies on this without support.
Vocal modulations and facial expressions in visual data, along with textual data, provide important cues to better identify affective states of the opinion holder.
“The vocal modulations and facial expressions in the visual data, along with textual data, provide important cues to better identify affective states of the opinion holder.”
The paper relies on this without support.
In sequence classification, the classification of each member is dependent on the other members.
“In sequence classification, the classification of each member is dependent on the other members.”
The paper relies on this without support.
Every utterance in a video is spoken at a distinct time and in a particular order.
“Every utterance in a video is spoken at a distinct time and in a particular order.”
The paper relies on this without support.
It is very difficult to come up with a generalized model from the behavior of a limited number of individuals.
“In real-world applications, the model should be robust to person idiosyncrasy but it is very difficult to come up with a generalized model from the behavior of a limited number of individuals.”
The paper relies on this without support.
The contextual relationship among utterances in a video is mostly ignored in the literature.
“The contextual relationship among utterances in a video is mostly ignored in the literature.”
The paper relies on this without support.
uni-SVM cannot model contextual dependencies among utterances.
“These results prove our initial hypothesis that modeling the contextual dependencies among utterances (which uni-SVM cannot do) improves the classification.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Poria et al. developed an LTSM-based network to extract contextual features from video for multimodal sentiment analysis.
“Interesting work is realized by Poria et al., who developed a LTSM-based network to extract contextual features from the video for multimodal sentiment analysis [10].”
From Survey on AI-Based Multimodal Methods for Emotion Detection (2019) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Reported improvement over uni-SVM is not always 2-5%
MinorTable 3 reports a MOSI trimodal accuracy of 77.9% for uni-SVM and 78.1% for h-LSTM, an improvement of only 0.2 percentage points. The corresponding MOUD results, 66.1% and 66.4%, likewise fall below the claimed range.
“Every LSTM network variant has outperformed the baseline uni-SVM on all the datasets by the margin of 2% to 5% (see Table 3).”
“Modality T V A T + V T + A V + A T + V + A 77.9 78.1 78.6 80.3 78.1 66.1 66.4 67.3 68.1 67.0 72.5 73.3 74.2 76.1 73.5”
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
(2010) · Cited by 8,261 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,222 · see citing works
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification
Zhou, P., Shi, W. Y., Tian, J., Qi, Z., Li, B., Hao, H., & Xu, B. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2034 · Cited by 2,108 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Self-citationFrom Utterance to Text: The Bias of Language in Speech and Writing
Olson, D. (1977) · Harvard Educational Review · doi:10.17763/haer.47.3.8840364413869005 · Cited by 1,065 · see citing works
Aspect extraction for opinion mining with a deep convolutional neural network
Poria, S., Cambria, E., & Gelbukh, A. F. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.06.009 · Cited by 862 · see citing works
Self-citationMultimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · see citing works
Self-citationDeep Learning-Based Document Modeling for Personality Detection from Text
Majumder, N., Poria, S., Gelbukh, A. F., & Cambria, E. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.23 · Cited by 671 · see citing works
Self-citationConvolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016) · Proceedings · doi:10.1109/icdm.2016.0055 · Cited by 624 · see citing works
Self-citationPoria, S., Cambria, E., & Gelbukh, A. F. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1303 · Cited by 553 · see citing works
Self-citationJumping NLP Curves: A Review of Natural Language Processing Research
Cambria, E., & White, B. (2014) · Cited by 455 · see citing works
Self-citationA Practical Guide to Sentiment Analysis
Feraco, A., Bandyopadhyay, S., Das, D., & Cambria, E. (2017) · Socio-affective computing · doi:10.1007/978-3-319-55394-8 · Cited by 410 · see citing works
Self-citationYoutube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403 · see citing works
Self-citationA Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural NetworksPreprint
Poria, S., Cambria, E., Hazarika, D., & Vij, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08815 · Cited by 368 · see citing works
Self-citationUniversal facial expressions in emotion.
Ekman, P. (1970) · Cited by 340 · see citing works
Wöllmer, M., Eyben, F., Reiter, S. A., Schuller, B. W., Cox, C., Douglas‐Cowie, E., & Cowie, R. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-192 · Cited by 329 · see citing works
Facial emotion recognition using multi-modal information
De Silva, L. C., Miyasato, T., & Nakatsu, R. (2002) · Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat. No.97TH8237) · doi:10.1109/icics.1997.647126 · Cited by 319 · see citing works
Wu, C., & Liang, W.-B. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.16 · Cited by 265 · see citing works
SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives
Cambria, E., Poria, S., Bajpai, R., & Schuller, B. W. (2016) · OPUS (Augsburg University) · Cited by 260 · see citing works
Self-citationKessous, L., Castellano, G., & Caridakis, G. (2010) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0025-5 · Cited by 256 · see citing works
(2013) · Cited by 236 · see citing works
Long short-term memory in recurrent neural networks
Gers, F. A. (2001) · doi:10.5075/epfl-thesis-2366 · Cited by 235 · see citing works
Poria, S., Peng, H., Hussain, A., Howard, N., & Cambria, E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.09.117 · Cited by 214 · see citing works
Self-citationMultimodal human emotion/expression recognition
Chen, L. S., Huang, T. S., Miyasato, T., & Nakatsu, R. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670976 · Cited by 183 · see citing works
Sentic LDA: Improving on LDA with semantic similarity for aspect-based sentiment analysis
Poria, S., Chaturvedi, I., Cambria, E., & Bisio, F. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727784 · Cited by 163 · see citing works
Self-citationEyben, F., Wöllmer, M., Graves, A., Schuller, B. W., Douglas‐Cowie, E., & Cowie, R. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0032-6 · Cited by 137 · see citing works
Rate-coded Restricted Boltzmann Machines for Face Recognition
Teh, Y. W., & Hinton, G. E. (2000) · Cited by 134 · see citing works
Label Embedding for Zero-shot Fine-grained Named Entity Typing.
Ma, Y., Cambria, E., & Gao, S. (2016) · International Conference on Computational Linguistics · Cited by 116 · see citing works
Self-citationStatistical Learning Theory and ELM for Big Social Data Analysis
Oneto, L., Bisio, F., Cambria, E., & Anguita, D. (2016) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2016.2572540 · Cited by 114 · see citing works
Self-citationEnsemble of SVM trees for multimodal emotion recognition
Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., & Prasad, R. (2012) · Cited by 112 · see citing works
Automatic Video Genre Categorization using Hierarchical SVM
Yuan, X., Lai, W., Mei, T., Hua, X., Wu, X., & Li, S. (2006) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2006.313037 · Cited by 110 · see citing works
A graph-based approach to commonsense concept extraction and semantic similarity detection
Rajagopal, D., Cambria, E., Olsher, D. J., & Kwok, K. (2013) · International World Wide Web Conference (WWW) · doi:10.1145/2487788.2487995 · Cited by 108 · see citing works
Self-citationAudio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice
Metallinou, A., Lee, S., & Narayanan, S. S. (2008) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2008.40 · Cited by 97 · see citing works
Semantic Audiovisual Data Fusion for Automatic Emotion Recognition
Datcu, D., & Rothkrantz, L. J. M. (2015) · Wiley eBooks · doi:10.1002/9781118910566.ch16 · Cited by 91 · see citing works
Recognizing Affect from Linguistic Information in 3D Continuous Space
Schuller, B. W. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.17 · Cited by 69 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921 · see citing works
Self-citationMultimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · see citing works
Self-citationMultimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · see citing works
Self-citationMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073 · see citing works
Self-citationMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855 · see citing works
Self-citationDeep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Mai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237 · see citing works
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · see citing works
Self-citationReal-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge
Zhang, K., Li, Y., Wang, J., Cambria, E., & Li, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3072412 · Cited by 184 · see citing works
Self-citationTransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
Wang, Z., Wan, Z., & Wan, X. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380000 · Cited by 156 · see citing works
Towards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155 · see citing works
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137 · see citing works
Shou, Y., Meng, T., Ai, W., Yang, S., & Li, K. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.072 · Cited by 132 · see citing works
Multi-Modal Attention for Speech Emotion Recognition
Pan, Z., Luo, Z., Yang, J., & Li, H. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1653 · Cited by 95 · see citing works
Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3378570 · Cited by 96 · see citing works
Dilated Residual Network with Multi-head Self-attention for Speech Emotion Recognition
Li, R., Wu, Z., Jia, J., Zhao, S., & Meng, H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682154 · Cited by 82 · see citing works
Do Deepfakes Feel Emotions? A Semantic Approach to Detecting Deepfakes Via Emotional Inconsistencies
Hosler, B., Salvi, D., Murray, A., Antonacci, F., Bestagini, P., Tubaro, S., & Stamm, M. C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00112 · Cited by 78 · see citing works
Multimodal Emotion Recognition With Temporal and Semantic Consistency
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3129331 · Cited by 71 · see citing works
Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Liang, J., Li, R., & Jin, Q. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413579 · Cited by 73 · see citing works
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · see citing works
Mai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26 · see citing works
Quan, Z., Sun, T., Su, M., & Wei, J. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4767437 · Cited by 25 · see citing works
- Authors
- Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P.
- Affiliations
- Temasek Laboratories@NTU (SG)
- National University of Singapore (SG)
- National Institute of Technology Warangal (IN)
- Instituto Politécnico Nacional (MX)
- Carnegie Mellon University (US)
- ORCID
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 873-883
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2740550900
- Semantic Scholar
- 75d17e8fa5165a849ebe5f0475bdf77bf0b6be74
- Cited by
- 939
- References
- 42
- Citations by year
- 2026: 81 · 2025: 102 · 2024: 132 · 2023: 149 · 2022: 110 · 2021: 105 · 2020: 115 · 2019: 77 · 2018: 47 · 2017: 20 · 2016: 1
- FWCI
- 32.30
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- sentiment analysis, user-generated video
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
