The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
WASSA-2017 Shared Task on Emotion Intensity
Open accessAbstract
We present the first shared task on detecting the intensity of emotion felt by the speaker of a tweet.We create the first datasets of tweets annotated for anger, fear, joy, and sadness intensities using a technique called best-worst scaling (BWS).We show that the annotations lead to reliable fine-grained intensity scores (rankings of tweets by intensity).The data was partitioned into training, development, and test sets for the competition.Twenty-two teams participated in the shared task, with the best system obtaining a Pearson correlation of 0.747 with the gold intensity scores.We summarize the machine learning setups, resources, and tools used by the participating teams, with a focus on the techniques and resources that are particularly useful for the task.The emotion intensity dataset and the shared task are helping improve our understanding of how we convey more or less intense emotions through language.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1×
“We implemented a package called Affec-tiveTweets (Mohammad and Bravo-Marquez, 2017) for the Weka machine learning workbench (Hall et al., 2009).”
MethodAn argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 2×
“On going work with annotations on the same data for valence , arousal, and dominance aims to better understand the relationships between the cir-cumplex model of emotions (Russell, 2003) and the categorical model of emotions (Ekman, 1992; Plutchik, 1980).”
BackgroundMining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793
Core affect and the psychological construction of emotion.
Russell, J. A. (2003) · Psychological Review · doi:10.1037/0033-295x.110.1.145 · Cited by 6,113
A law of comparative judgment.
Thurstone, L. L. (1927) · Psychological Review · doi:10.1037/h0070288 · Cited by 5,210
Recognizing contextual polarity in phrase-level sentiment analysis
Wilson, T., Wiebe, J., & Hoffmann, P. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220619 · Cited by 3,345
CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON
Mohammad, S. M., & Turney, P. D. (2013) · Computational Intelligence · doi:10.1111/j.1467-8640.2012.00460.x · Cited by 2,634
Self-citationSENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining
Esuli, A., & Sebastiani, F. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/54vo5wjya4cw · Cited by 2,480
The Theory of the Estimation of Test Reliability
Kuder, G. F., & Richardson, M. W. (1937) · Psychometrika · doi:10.1007/bf02288391 · Cited by 2,002
Norms of valence, arousal, and dominance for 13,915 English lemmas
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0314-x · Cited by 1,959
Frijda, N. H. (1988) · American Psychologist · doi:10.1037/0003-066x.43.5.349 · Cited by 1,528
The Method of Paired Comparisons.
Curnow, R. N., & David, H. A. (1964) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344014 · Cited by 1,271
Sentiment strength detection for the social web
Thelwall, M., Buckley, K., & Paltoglou, G. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21662 · Cited by 1,062
Sentiment Analysis of Short Informal Texts
Kiritchenko, S., Zhu, X., & Mohammad, S. M. (2014) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4272 · Cited by 880
Self-citationTemporal patterns of happiness and information in a global social network: hedonometrics and Twitter
Dodds, P. S., Harris, K. D., Kloumann, I. M., Bliss, C. A., & Danforth, C. M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026752 · Cited by 809 · cited 1×
“The gold training words are taken from existing affective norms and emotion lexicons: NRC Hashtag Emotion Lexicon (Mohammad, 2012b; Mohammad and Kir-itchenko, 2015), affective norms from Warriner et al. (2013), Brysbaert et al. (2014), and ratings for happiness from Dodds et al. (2011).”
BackgroundEmotions from text: machine learning for text-based emotion prediction
Alm, C. O., Roth, D., & Sproat, R. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220648 · Cited by 781
SemEval-2018 Task 1: Affect in Tweets
Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1001 · Cited by 746
Self-citationNRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets
Mohammad, S. M., Kiritchenko, S., & Zhu, X. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1308.6242 · Cited by 451
Self-citationStance and Sentiment in Tweets
Mohammad, S. M., Sobhani, P., & Kiritchenko, S. (2017) · ACM Transactions on Internet Technology · doi:10.1145/3003433 · Cited by 449
Self-citationIdentifying Expressions of Emotion in Text
Aman, S., & Szpakowicz, S. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74628-7_27 · Cited by 447
Best-Worst Scaling: Theory, Methods and Applications
Louviere, J. J., Flynn, T. N., & Marley, A. A. J. (2015) · Thammasat University Digital Collections · Cited by 435
Using Hashtags to Capture Fine Emotion Categories from Tweets
Mohammad, S. M., & Kiritchenko, S. (2015) · Computational Intelligence · doi:10.1111/coin.12024 · Cited by 408
Self-citationMohammad, S. M. (2012) · Joint Conference on Lexical and Computational Semantics · Cited by 355
Self-citationModeling public mood and emotion: Twitter sentiment and socio-economic phenomena
Bollen, J., Pepe, A., & Mao, H. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0911.1583 · Cited by 316
emoji2vec: Learning Emoji Representations from their Description
Eisner, B., Rocktäschel, T., Augenstein, I., Bosnjak, M., & Riedel, S. (2016) · Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w16-6208 · Cited by 291
YATES, F. (1936) · Annals of Eugenics · doi:10.1111/j.1469-1809.1936.tb02134.x · Cited by 278
Mohammad, S., & Bravo-Marquez, F. (2017) · Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017) · doi:10.18653/v1/s17-1007 · Cited by 220
Self-citationSemEval-2012 Task 2 : measuring degrees of relational similarity
Jurgens, D., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012) · NPARC · Cited by 182
Self-citationMohammad, S. M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/58igjzkfg9k5 · Cited by 179
Self-citationBest-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation
Kiritchenko, S., & Mohammad, S. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2074 · Cited by 170
Self-citationBest-worst scaling: theory and methods
Flynn, T. N., & Marley, A. A. J. (2014) · Edward Elgar Publishing eBooks · doi:10.4337/9781781003152.00014 · Cited by 161
Petrović, S., Osborne, M., & Lavrenko, V. P. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 158
Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels
Jurgens, D. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 71
Statistical affect detection in collaborative chat
Brooks, M., Kuksenok, K., Torkildson, M. K., Perry, D., Robinson, J. J., Scott, T. J., Anicello, O., Zukowski, A., Harris, P. G., & Aragon, C. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441813 · Cited by 63
Cited by
SemEval-2018 Task 1: Affect in Tweets
Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1001 · Cited by 746
Self-citationObtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words
Mohammad, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1017 · Cited by 613
Self-citationExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384
Self-citationMohammad, S. M. (2016) · Emotion Measurement · doi:10.1016/b978-0-08-100508-8.00009-6 · Cited by 391
“…lexicons are also commonly used in conjunction with neural representations for detecting emotions or for detecting affect-related classes such as personality traits and states of well-being (Mohammad & Bravo-Marquez, 2017; Mohammad et al., 2018; Chatterjee et al., 2019; Agrawal & Suri, 2019).”
MethodSelf-citationUnderstanding Emotions in Text Using Deep Learning and Big Data
Chatterjee, A., Gupta, U., Chinnakotla, M. K., Srikanth, R., Galley, M., & Agrawal, P. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.029 · Cited by 384
Metadata
- Authors
- Mohammad, S., & Bravo-Marquez, F.
- Venue
- Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis
- Published
- 2017
- Pages
- 34-49
- OpenAlex
- W2747541555
- Cited by
- 270
- References
- 35
- FWCI
- 18.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.