CoChat
Sign Up
Cited by 468Sentiment Analysis and Opinion MiningHate Speech and Cyberbullying DetectionEmotion and Mood Recognition

SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods

Open access
Ribeiro, Filipe N, Araújo, Matheus, Gonçalves, Pollyanna, André Gonçalves, Marcos, Benevenuto, Fabrício·2016·EPJ Data Science
DOI 10.1140/epjds/s13688-016-0085-1
Open this paper in CoChat — free accountOpen PDF

Abstract

In the last few years thousands of scientific papers have investigated sentiment analysis, several startups that measure opinions on real data have emerged and a number of innovative products related to this theme have been developed. There are multiple methods for measuring sentiments, including lexical-based and supervised machine learning methods. Despite the vast interest on the theme and wide popularity of some methods, it is unclear which one is better for identifying the polarity (i.e., positive or negative) of a message. Accordingly, there is a strong need to conduct a thorough apple-to-apple comparison of sentiment analysis methods, as they are used in practice , across multiple datasets originated from different data sources. Such a comparison is key for understanding the potential limitations, advantages, and disadvantages of popular methods. This article aims at filling this gap by presenting a benchmark comparison of twenty-four popular sentiment analysis methods (which we call the state-of-the-practice methods). Our evaluation is based on a benchmark of eighteen labeled datasets, covering messages posted on social networks, movie and product reviews, as well as opinions and comments in news articles. Our results highlight the extent to which the prediction performance of these methods varies considerably across datasets. Aiming at boosting the development of this research area, we open the methods’ codes and datasets used in this article, deploying them in a benchmark system, which provides an open API for accessing and comparing sentence-level sentiment analysis methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1140/epjds/s13688-016-0085-1

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Neutral reviews are difficult to detect in most sentiment analysis methods.

    “It appears that neutral reviews are difficult to detect in most of the sentiment analysis methods (Ribeiro et al., 2016).”

    From Sentiment Analysis in Tourism: Capitalizing on Big Data (2019) · cites this paper as Ribeiro et al., 2016

    Section 5.3 says neutral messages were harder for most methods to detect, supporting the general point. But its review datasets have only two-class results, and Section 5.1 says product reviews were expected to contain polarized opinions; the paper does not establish this finding specifically for neutral reviews. No page number is available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • In the data mining process, different types of sentiment analysis methods can be distinguished: machine learning, rule/dictionary-based and hybrid approaches.

    “In the data mining process, different types of sentiment analysis methods can be distinguished in the literature; namely (i) machine learning, (ii) rule/dictionary-based and (iii) hybrid approaches (Feldman, 2013;Ribeiro et al., 2016).”

    From Sentiment Analysis in Tourism: Capitalizing on Big Data (2019) · cites this paper as Ribeiro et al., 2016

  • Sentiment analysis in tourism is only beginning to gain in popularity.

    “Whilst highly relevant for tourism, sentiment analysis in tourism is only beginning to gain in popularity (Feldman, 2013, Gao et al., 2015, Ribeiro et al., 2016).”

    From Sentiment Analysis in Tourism: Capitalizing on Big Data (2019) · cites this paper as Ribeiro et al., 2016

    The Introduction discusses growing interest in sentiment analysis generally, and the Concluding remarks describe its use on social-network content. Neither addresses whether sentiment analysis in tourism was only beginning to gain popularity; searches for tourism, travel, and hotel applications found no relevant passage. Our copy therefore cannot verify the tourism-specific claim. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 60

  1. Feldman R (2013) Techniques and applications for sentiment analysis. Commun ACM 56(4):82-89. doi:10.1145/2436256.2436274 · DOI · Paper page
  2. Hu M, Liu B (2004) Mining and summarizing customer reviews. In: KDD'04, pp 168-177. http://doi.acm.org/10.1145/1014052.1014073 · DOI · Paper page
  3. Oliveira N, Cortez P, Areal N (2013) On the predictability of stock market behavior using stocktwits sentiment and posting volume. In: Progress in artificial intelligence. Lecture notes in computer science, vol 8154. Springer, Heidelberg, pp 355-365
  4. Tumasjan A, Sprenger TO, Sandner PG, Welpe IM (2010) Predicting elections with Twitter: what 140 characters reveal about political sentiment. In: 4th international AAAI conference on weblogs and social media (ICWSM)
  5. Pang B, Lee L, Vaithyanathan S (2002) Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the 2002 conference on empirical methods in natural language processing (EMNLP '02), pp 79-86
  6. Gonçalves P, Benevenuto F, Cha M (2013) PANAS-t: a pychometric scale for measuring sentiments on Twitter. arXiv:1308.1857v1 · arXiv
  7. Bollen J, Pepe A, Mao H (2009) Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. arXiv:0911.1583 · arXiv
  8. Kramer ADI, Guillory JE, Hancock JT (2014) Experimental evidence of massive-scale emotional contagion through social networks. Proc Natl Acad Sci USA 111(24):8788-8790. doi:10.1073/pnas.1320040111 · DOI · Paper page
  9. Reis J, Goncalves P, Vaz de Melo P, Prates R, Benevenuto F (2014) Magnet news: you choose the polarity of what you read. In: 8th international AAAI conference on weblogs and social media (ICWSM)
  10. Reis J, Benevenuto F, Vaz de Melo P, Prates R, Kwak H, An J (2015) Breaking the news: first impressions matter on online news. In: 9th international AAAI conference on weblogs and social media (ICWSM)
  11. Tamersoy A, De Choudhury M, Chau DH (2015) Characterizing smoking and drinking abstinence from social media. In: Proceedings of the 26th ACM conference on hypertext and social media (HT)
  12. Tsytsarau M, Palpanas T (2012) Survey on mining subjective data on the web. Data Min Knowl Discov 24(3):478-514. doi:10.1007/s10618-011-0238-6 · DOI · Paper page
  13. Levallois C (2013) Umigon: sentiment analysis for tweets based on terms lists and heuristics. In: The second joint conference on lexical and computational semantics (*SEM), volume 2: proceedings of the seventh international workshop on semantic evaluation (SemEval 2013), pp 414-417. http://www.aclweb.org/anthology/S13-2068
  14. Abbasi A, Hassan A, Dhar M (2014) Benchmarking Twitter sentiment analysis tools. In: 9th international conference on language resources and evaluation (LREC)
  15. Gonçalves P, Araujo M, Benevenuto F, Cha M (2013) Comparing and combining sentiment analysis methods. In: Proceedings of the 1st ACM conference on online social networks (COSN'13)
  16. Araujo M, Diniz JP, Bastos L, Soares E, Júnior M, Ferreira M, Ribeiro F, Benevenuto F (2016) iFeel 2.0: a multilingual benchmarking system for sentence-level sentiment analysis. In: 10th international AAAI conference on weblogs and social media (ICWSM)
  17. Wilson T, Hoffmann P, Somasundaran S, Kessler J, Wiebe J, Choi Y, Cardie C, Riloff E, Patwardhan S (2005) OpinionFinder: a system for subjectivity analysis. In: HLT/EMNLP on interactive demonstrations, pp 34-35
  18. Wilson T, Wiebe J, Hoffmann P (2005) Recognizing contextual polarity in phrase-level sentiment analysis. In: Proceedings of the conference on human language technology and empirical methods in natural language processing (HLT '05), pp 347-354
  19. Esuli A, Sebastiani F (2006) SentiWordNet: a publicly available lexical resource for opinion mining. In: 5th international conference on language resources and evaluation (LREC), pp 417-422
  20. Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In: 7th international conference on language resources and evaluation (LREC), pp 2200-2204
  21. Cambria E, Olsher D, Rajagopal D (2014) SenticNet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis. In: 28th AAAI conference on artificial intelligence, pp 1515-1521
  22. Nielsen F (2011) A new ANEW: evaluation of a word list for sentiment analysis in microblogs. arXiv:1103.2903 · arXiv
  23. Taboada M, Brooke J, Tofiloski M, Voll K, Stede M (2011) Lexicon-based methods for sentiment analysis. Comput Linguist 37(2):267-307
  24. Hannak A, Anderson E, Barrett LF, Lehmann S, Mislove A, Riedewald M (2012) Tweetin' in the rain: exploring societal-scale effects of weather on mood. In: 6th international AAAI conference on weblogs and social media (ICWSM)
  25. Mohammad S (2012) #emotional tweets. In: The first joint conference on lexical and computational semantics - volume 1: proceedings of the main conference and the shared task, and volume 2: proceedings of the sixth international workshop on semantic evaluation (SemEval 2012), pp 246-255. http://www.aclweb.org/anthology/S12-1033
  26. De Smedt T, Daelemans W (2012) Pattern for Python. J Mach Learn Res 13(1):2063-2067
  27. Wang H, Can D, Kazemzadeh A, Bar F, Narayanan S (2012) A system for real-time Twitter sentiment analysis of 2012 U.S. presidential election cycle. In: ACL system demonstrations, pp 115-120
  28. Watson D, Clark L (1985) Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 54(1):1063-1070
  29. Mohammad S, Turney PD (2013) Crowdsourcing a word-emotion association lexicon. Comput Intell 29(3):436-465
  30. Plutchik R (1980) A general psychoevolutionary theory of emotion. Academic Press, New York, pp 3-33
  31. Pappas N, Katsimpras G, Stamatatos E (2013) Distinguishing the popularity between topics: a system for up-to-date opinion retrieval and mining in the web. In: 14th international conference on intelligent text processing and computational linguistics
  32. Mohammad SM, Kiritchenko S, Zhu X (2013) NRC-Canada: building the state-of-the-art in sentiment analysis of tweets. In: Proceedings of the seventh international workshop on semantic evaluation exercises (SemEval 2013)
  33. Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng AY, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP '13), pp 1631-1642
  34. Brysbaert M, New B (2009) Moving beyond Kučera and Francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behav Res Methods 41(4):977-990
  35. Lexalytics (2015) Sentiment extraction -measuring the emotional tone of content. Technical report, Lexalytics 45. Wiebe J, Wilson T, Cardie C (2005) Annotating expressions of opinions and emotions in language. Lang Resour Eval 39(2):165-210
  36. Stone PJ, Dunphy DC, Smith MS, Ogilvie DM (1966) The general inquirer: a computer approach to content analysis. MIT Press, Cambridge
  37. Biever C (2010) Twitter mood maps reveal emotional states of America. New Sci 207(2771):14
  38. Taboada M, Anthony C, Voll K (2006) Methods for creating semantic orientation dictionaries. In: 5th international conference on language resources and evaluation (LREC), pp 427-432
  39. Mohammad S, Dunne C, Dorr B (2009) Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus. In: Proceedings of the 2009 conference on empirical methods in natural language processing (EMNLP '09), pp 599-608. http://dl.acm.org/citation.cfm?id=1699571.1699591
  40. Taboada M, Anthony C, Voll K (2006) Methods for creating semantic orientation dictionaries. In: 5th international conference on language resources and evaluation (LREC), pp 427-432
  41. Strapparava C, Mihalcea R (2007) SemEval-2007 task 14: affective text. In: Proceedings of the 4th international workshop on semantic evaluations (SemEval '07), pp 70-74. http://dl.acm.org/citation.cfm?id=1621474.1621487
  42. Nakov P, Kozareva Z, Ritter A, Rosenthal S, Stoyanov V, Wilson T (2013) SemEval-2013 task 2: sentiment analysis in Twitter. In: The second joint conference on lexical and computational semantics (*SEM), volume 2: proceedings of the seventh international workshop on semantic evaluation (SemEval 2013), pp 312-320
  43. Pang B, Lee L (2004) A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of the 42nd annual meeting of the Association for Computational Linguistics, pp 271-278
  44. Cambria E, Speer R, Havasi C, Hussain A (2010) SenticNet: a publicly available semantic resource for opinion mining. In: AAAI fall symposium series
  45. Liu B (2012) Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, vol 5(1). doi:10.2200/s00416ed1v01y201204hlt016 · DOI
  46. Godbole N, Srinivasaiah M, Skiena S (2007) Large-scale sentiment analysis for news and blogs. In: 1st international AAAI conference on weblogs and social media (ICWSM)
  47. Kouloumpis E, Wilson T, Moore J (2011) Twitter sentiment analysis: the good the bad and the OMG! In: 5th international AAAI conference on weblogs and social media (ICWSM)
  48. Tang D, Wei F, Yang N, Zhou M, Liu T, Qin B (2014) Learning sentiment-specific word embedding for Twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics, pp 1555-1565
  49. Kalchbrenner N, Grefenstette E, Blunsom P (2014) A convolutional neural network for modelling sentences. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics, pp 655-665
  50. Johnson R, Zhang T (2015) Effective use of word order for text categorization with convolutional neural networks. In: Human language technologies: the 2015 annual conference of the North American chapter of the ACL, pp 103-112.
  51. Valitutti R (2004) WordNet-affect: an affective extension of WordNet. In: 4th international conference on language resources and evaluation (LREC), pp 1083-1086
  52. Dodds PS, Danforth CM (2009) Measuring the happiness of large-scale written expression: songs, blogs, and presidents. J Happiness Stud 11(4):441-456. doi:10.1007/s10902-009-9150-9 · DOI
  53. Snow R, O'Connor B, Jurafsky D, Ng AY (2008) Cheap and fast -but is it good?: evaluating non-expert annotations for natural language tasks. In: Proceedings of the 2008 conference on empirical methods in natural language processing (EMNLP '08)
  54. Pappas N, Popescu-Belis A (2013) Sentiment analysis of user comments for one-class collaborative filtering over TED talks. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval, pp 773-776
  55. Diakopoulos NA, Shamma DA (2010) Characterizing debate performance via aggregated Twitter sentiment. In: Proceedings of the 28th international conference on human factors in computing systems, pp 1195-1198
  56. Narr S, Hülfenhaus M, Albayrak S (2012) Language-independent Twitter sentiment analysis. In: Workshop on knowledge discovery, data mining and machine learning (KDML-2012)
  57. Berenson ML, Levine DM, Szabat KA (2014) In: Basic business statistics -concepts and applications, 13th edn. Pearson Education, Upper Saddle River
  58. Jain R (1991) The art of computer systems performance analysis -techniques for experimental design, measurement, simulation, and modeling. Wiley, New York
  59. Garcia D, Garas A, Schweitzer F (2012) Positive words carry less information than negative words. EPJ Data Sci 1(1):3
  60. Dodds PS, Clark EM, Desu S, Frank MR, Reagan AJ, Williams JR, Mitchell L, Harris KD, Kloumann IM, Bagrow JP, Megerdoomian K, McMahon MT, Tivnan BF, Danforth CM (2015) Human language reveals a universal positivity bias. Proc Natl Acad Sci USA 112(8):2389-2394. doi:10.1073/pnas.1411678112 http://www.pnas.org/content/112/8/2389.full.pdf · DOI · Paper page

Cited by

Sorted by citations

Authors
Ribeiro, F. N., Araújo, M., Gonçalves, P., André Gonçalves, M., & Benevenuto, F.
Affiliations
  • Universidade Federal de Ouro Preto (BR)
  • Universidade Federal de Minas Gerais (BR)
Corresponding author
Ribeiro, Filipe N
ORCID
Venue
EPJ Data Science
Publisher
Springer Science and Business Media LLC
Published
2016-07-07
Volume
5
Issue
1
ISSN
2193-1127 (ISSN-L)
DOI
10.1140/epjds/s13688-016-0085-1
OpenAlex
W2471350540
Cited by
468
References
8
Citations by year
2026: 19 · 2025: 36 · 2024: 51 · 2023: 52 · 2022: 60 · 2021: 68 · 2020: 54 · 2019: 39 · 2018: 47 · 2017: 30 · 2016: 10 · 2015: 2
FWCI
34.60
Type
journal article
Language
en
Open access
Licence
Locations
  • EPJ Data Science (publishedVersion, cc-by)
  • Repository of Samara University (Samara National Research University) (submittedVersion, cc-by)
  • ENPUIR (National Pedagogical Dragomanov University) (submittedVersion)
  • Repository of Samara University (Samara National Research University) (submittedVersion)
Funders
  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (BR)
  • Fundação de Amparo à Pesquisa do Estado de Minas Gerais (FAPEMIG/PRONEX APQ-01400-14)
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
sentiment analysis, benchmark, polarity prediction, supervised learning, social media posts, product reviews, movie reviews, news comments, open APIs, method comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review