Comparing automated text classification methods
Open accessAbstract
Online social media drive the growth of unstructured text data. Many marketing applications require structuring this data at scales non-accessible to human coding, e.g., to detect communication shifts in sentiment or other researcher-defined content categories. Several methods have been proposed to automatically classify unstructured text. This paper compares the performance of ten such approaches (five lexicon-based, five machine learning algorithms) across 41 social media datasets covering major social media platforms, various sample sizes, and languages. So far, marketing research relies predominantly on support vector machines (SVM) and Linguistic Inquiry and Word Count (LIWC). Across all tasks we study, either random forest (RF) or naive Bayes (NB) performs best in terms of correctly uncovering human intuition. In particular, RF exhibits consistently high performance for three-class sentiment, NB for small samples sizes. SVM never outperform the remaining methods. All lexicon-based approaches, LIWC in particular, perform poorly compared with machine learning. In some applications, accuracies only slightly exceed chance. Since additional considerations of text classification choice are also in favor of NB and RF, our results suggest that marketing research can benefit from considering these alternatives.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine learning algorithms and lexicon-based text classification can be used to analyze various social media datasets.
“Machine learning algorithms and lexicon-based text classification can be used to analyze various social media datasets (Hartmann et al. 2019).”
From A strategic framework for artificial intelligence in marketing (2021) · cites this paper as Hartmann et al. 2019
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
Schölkopf, B., & Smola, A. J. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/4175.001.0001 · Cited by 11,934 · see citing works
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026683 · Cited by 8,134 · see citing works
Machine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,927 · see citing works
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · see citing works
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · see citing works
Top 10 algorithms in data mining
Wu, X., Kumar, V., Ross Quinlan, J., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G. J., Ng, A., Liu, B., Yu, P. S., et al. (2008) · Knowledge and Information Systems · doi:10.1007/s10115-007-0114-2 · Cited by 5,667 · see citing works
A few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326 · see citing works
Pang, B., & Lee, L. (2004) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1218955.1218990 · Cited by 3,317 · see citing works
What Makes Online Content Viral?
Berger, J., & Milkman, K. L. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0353 · Cited by 3,178 · see citing works
On the Optimality of the Simple Bayesian Classifier under Zero-One Loss
Domingos, P., & Pazzani, M. J. (1997) · Machine Learning · doi:10.1023/a:1007413511361 · Cited by 3,115 · see citing works
An empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780 · see citing works
A re-examination of text categorization methods
Yang, Y., & Liu, X. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312647 · Cited by 2,670 · see citing works
Image-Based Recommendations on Styles and Substitutes
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767755 · Cited by 2,348 · see citing works
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Atiya, A. F. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.848998 · Cited by 2,248 · see citing works
Thumbs up? Sentiment Classification using Machine Learning TechniquesPreprint
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205070 · Cited by 2,211 · see citing works
Adaptive Control Processes: A Guided Tour.
Page, E. S., & Bellman, R. (1962) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2343225 · Cited by 2,208 · see citing works
Do we need hundreds of classifiers to solve real world classification problems
Fernández-Delgado, M., Cernadas, E., Barro, S., & AmorimDinani (2014) · Journal of Machine Learning Research · doi:10.5555/2627435.2697065 · Cited by 2,131 · see citing works
An Evaluation of Statistical Approaches to Text Categorization
Yang, Y. (1999) · Information Retrieval · doi:10.1023/a:1009982220290 · Cited by 1,945 · see citing works
(2001) · Cited by 1,932 · see citing works
Graph-based approach for airborne light detection and ranging segmentation
Vilariño, D. L., Cabaleiro, J. C., Sánchez, J. M., Rivera, F. F., & Pena, T. F. (2017) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.11.015020 · Cited by 1,760 · see citing works
Digital marketing: a framework, review, and research agenda
Kannan, P. K., & Li, H. “. (2017) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2016.11.006 · Cited by 1,707 · see citing works
The Lack of A Priori Distinctions Between Learning Algorithms
Wolpert, D. H. (1996) · Neural Computation · doi:10.1162/neco.1996.8.7.1341 · Cited by 1,652 · see citing works
Inductive learning algorithms and representations for text categorization
Dumais, S., Platt, J. C., Heckerman, D. E., & Sahami, M. (1998) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/288627.288651 · Cited by 1,463 · see citing works
†, F. J. M. (2017) · Lexikon des gesamten Buchwesens Online · doi:10.1163/9789004337862_lgbo_com_161117 · Cited by 1,435 · see citing works
Yahoo! for Amazon: Sentiment Extraction from Small Talk on the Web
Das, S. R., & Chen, M. Y. (2007) · Management Science · doi:10.1287/mnsc.1070.0704 · Cited by 1,370 · see citing works
Advertising Content and Consumer Engagement on Social Media: Evidence from Facebook
Lee, D., Hosanagar, K., & Nair, H. S. (2018) · Management Science · doi:10.1287/mnsc.2017.2902 · Cited by 1,069 · see citing works
Adaptive Control Processes: A Guided Tour
Morgenstern, D., & Bellman, R. (1962) · Econometrica · doi:10.2307/1909901 · Cited by 1,018 · see citing works
Computer Age Statistical Inference
Efron, B., & Hastie, T. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316576533 · Cited by 926 · see citing works
Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon
Mohammad, S. M., & Turney, P. D. (2010) · NPARC · Cited by 882 · see citing works
A Survey of Text Classification Algorithms
Aggarwal, C. C., & Zhai, C. (2012) · Springer eBooks · doi:10.1007/978-1-4614-3223-4_6 · Cited by 877 · see citing works
Sentiment analysis using product review data
Fang, X., & Zhan, J. Z. (2015) · Journal Of Big Data · doi:10.1186/s40537-015-0015-2 · Cited by 791 · see citing works
A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum CutsPreprint
Pang, B., & Lee, L. (2004) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0409058 · Cited by 789 · see citing works
Mine Your Own Business: Market-Structure Surveillance Through Text Mining
Netzer, O., Feldman, R., Goldenberg, J., & Fresko, M. (2012) · Marketing Science · doi:10.1287/mksc.1120.0713 · Cited by 741 · see citing works
Document-level sentiment classification: An empirical comparison between SVM and ANN
Moraes, R., Valiati, J. F., & Neto, W. P. G. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.07.059 · Cited by 734 · see citing works
A new ANEW: Evaluation of a word list for sentiment analysis in microblogsPreprint
Nielsen, F. Å. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.2903 · Cited by 733 · see citing works
Bennett, K. P., & Campbell, C. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/380995.380999 · Cited by 689 · see citing works
Ghose, A., Ipeirotis, P. G., & Li, B. (2012) · Marketing Science · doi:10.1287/mksc.1110.0700 · Cited by 649 · see citing works
Automated Text Analysis for Consumer Research
Humphreys, A., & Wang, R. J.-H. (2018) · Journal of Consumer Research · doi:10.1093/jcr/ucx104 · Cited by 636 · see citing works
Does Chatter Really Matter? Dynamics of User-Generated Content and Stock Performance
Tirunillai, S., & Tellis, G. J. (2011) · Marketing Science · doi:10.1287/mksc.1110.0682 · Cited by 622 · see citing works
Ensemble of feature sets and classification algorithms for sentiment classification
Xia, R., Zong, C., & Li, S. (2011) · Information Sciences · doi:10.1016/j.ins.2010.11.023 · Cited by 583 · see citing works
Artificial intelligence: the next digital frontier?
Bughin, J., Hazan, E., Ramaswamy, S., Chui, M., Allas, T., Dahlström, P., Henke, N., & Trench, M. (2017) · Cited by 569 · see citing works
Sentiment analysis of blogs by combining lexical knowledge with text classification
Melville, P., Gryc, W., & Lawrence, R. D. (2009) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557156 · Cited by 509 · see citing works
Sentiment analysis in twitter using machine learning techniques
Neethu, M. S., & Rajasree, R. (2013) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt.2013.6726818 · Cited by 502 · see citing works
Contributions to the study of SMS spam filtering
Almeida, T. A., Hidalgo, J. M. G., & Yamakami, A. (2011) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/2034691.2034742 · Cited by 466 · see citing works
Does Twitter matter? The impact of microblogging word of mouth on consumers’ adoption of new movies
Hennig-Thurau, T., Wiertz, C., & Feldhaus, F. (2015) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-014-0388-3 · Cited by 434 · see citing works
Hewett, K., Rand, W., Rust, R. T., & van Heerde, H. J. (2016) · Journal of Marketing · doi:10.1509/jm.15.0033 · Cited by 366 · see citing works
Broadcasting and Narrowcasting: How Audience Size Affects What People Share
Barasch, A., & Berger, J. A. (2014) · Journal of Marketing Research · doi:10.1509/jmr.13.0238 · Cited by 361 · see citing works
Stochastic gradient boosted distributed decision trees
Ye, J., Chow, J.-H., Chen, J., & Zheng, Z. (2009) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1645953.1646301 · Cited by 343 · see citing works
Ordenes, F. J. V., Grewal, D., Ludwig, S., de Ruyter, K., Mahr, D., & Wetzels, M. G. M. (2018) · Journal of Consumer Research · doi:10.1093/jcr/ucy032 · Cited by 331 · see citing works
Classifying sentiment in microblogs
Bermingham, A., & Smeaton, A. F. (2010) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1871437.1871741 · Cited by 328 · see citing works
Listening in on Social Media: A Joint Model of Sentiment and Venue Format Choice
Schweidel, D. A., & Moe, W. W. (2014) · Journal of Marketing Research · doi:10.1509/jmr.12.0424 · Cited by 323 · see citing works
From Group to Individual Labels Using Deep Features
Kotzias, D., Denil, M., de Freitas, N., & Smyth, P. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783380 · Cited by 314 · see citing works
Public Opinion and Executive Compensation
Kuhnen, C. M., & Niessen‐Ruenzi, A. (2012) · Management Science · doi:10.1287/mnsc.1110.1490 · Cited by 299 · see citing works
Cavanaugh, L. A., Bettman, J. R., & Luce, M. F. (2015) · Journal of Marketing Research · doi:10.1509/jmr.10.0219 · Cited by 281 · see citing works
Akpinar, E., & Berger, J. (2017) · Journal of Marketing Research · doi:10.1509/jmr.13.0350 · Cited by 267 · see citing works
PSVM: Parallelizing Support Vector Machines on Distributed Computers
Chang, E. Y. (2011) · Springer eBooks · doi:10.1007/978-3-642-20429-6_10 · Cited by 246 · see citing works
Villarroel Ordenes, F., Ludwig, S., de Ruyter, K., Grewal, D., & Wetzels, M. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucw070 · Cited by 245 · see citing works
Measuring and Managing Consumer Sentiment in an Online Community Environment
Homburg, C., Ehm, L., & Artz, M. (2015) · Journal of Marketing Research · doi:10.1509/jmr.11.0448 · Cited by 196 · see citing works
A Comparison of Sentiment Analysis Techniques: Polarizing Movie Blogs
Annett, M., & Kondrak, G. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-68825-9_3 · Cited by 174 · see citing works
Hansen, N., Kupfer, A.-K., & Hennig‐Thurau, T. (2018) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2018.08.001 · Cited by 164 · see citing works
The Role of Emotions for the Perceived Usefulness in Online Customer Reviews
Felbermayr, A., & Νανόπουλος, Α. (2016) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2016.05.004 · Cited by 163 · see citing works
Search Personalization Using Machine Learning
Yoganarasimhan, H. (2019) · Management Science · doi:10.1287/mnsc.2018.3255 · Cited by 159 · see citing works
Neural Networks as Statistical Tools for Business Researchers
DeTienne, K. B., Detienne, D. H., & Joshi, S. A. (2003) · Organizational Research Methods · doi:10.1177/1094428103251907 · Cited by 158 · see citing works
Modeling the role of message content and influencers in social media rebroadcasting
Zhang, Y., Moe, W. W., & Schweidel, D. A. (2016) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2016.07.003 · Cited by 156 · see citing works
Puranam, D., Narayan, V., & Kadiyali, V. (2017) · Marketing Science · doi:10.1287/mksc.2017.1048 · Cited by 102 · see citing works
Huang, D., & Luo, L. (2016) · Marketing Science · doi:10.1287/mksc.2015.0946 · Cited by 89 · see citing works
When Words Sweat: Identifying Signals for Loan Default in the Text of Loan ApplicationsPreprint
Netzer, O., Lemaire, A., & Herzenstein, M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2865327 · Cited by 52 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A strategic framework for artificial intelligence in marketing
Huang, M.-H., & Rust, R. T. (2021) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-020-00749-9 · Cited by 2,059 · see citing works
“At the feeling level, more real-time and accurate emotion sensing from posted messages can better engage customers and provide a better interaction experience (Hartmann et al. 2019; Lee et al. 2018).”
BackgroundUniting the Tribes: Using Text for Marketing Insight
Berger, J., Humphreys, A., Ludwig, S., Moe, W. W., Netzer, O., & Schweidel, D. A. (2020) · Journal of Marketing · doi:10.1177/0022242919873106 · Cited by 742 · see citing works
Machine learning and AI in marketing–Connecting computing power to human insights
Ma, L., & Sun, B. (2020) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2020.04.005 · Cited by 602 · see citing works
Brave New World? On AI and the Management of Customer Relationships
Libai, B., Bart, Y., Gensler, S., Hofacker, C. F., Kaplan, A., Kötterheinrich, K., & Kroll, E. B. (2020) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2020.04.002 · Cited by 358 · see citing works
Sentiment Analysis in the Age of Generative AI
Krugmann, J. O., & Hartmann, J. (2024) · Customer Needs and Solutions · doi:10.1007/s40547-024-00143-4 · Cited by 189 · see citing works
“The field of sentiment analysis employs diverse methods, ranging from lexicon-based approaches to traditional machine learning models, and extending to the more advanced transfer learning techniques, as detailed by Hartmann et al. (2019 & 2023) [24, 25].”
InfluentialNatural Language Processing in Marketing
Hartmann, J., & Netzer, O. (2023) · Artificial Intelligence in Marketing · doi:10.1108/s1548-643520230000020011 · Cited by 38 · see citing works
Self-citation
- Authors
- Hartmann, J., Huppertz, J., Schamp, C., & Heitmann, M.
- Affiliations
- Universität Hamburg (DE)
- Corresponding author
- Hartmann, Jochen
- Venue
- International Journal of Research in Marketing
- Publisher
- Elsevier BV
- Published
- 2019-03
- Volume
- 36
- Issue
- 1
- Pages
- 20-38
- ISSN
- 0167-8116 (ISSN-L)
- OpenAlex
- W2898339904
- Cited by
- 399
- References
- 74
- Citations by year
- 2026: 16 · 2025: 58 · 2024: 54 · 2023: 63 · 2022: 81 · 2021: 71 · 2020: 38 · 2019: 18
- FWCI
- 17.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Research in Marketing (publishedVersion, cc-by-nc-nd)
- WU Research (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion)
- MADOC (University of Mannheim) (submittedVersion, cc-by-nc-nd)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by-nc-nd)
- Funders
- German Research Foundation (HE 6703/1-2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- text classification, social media text, sentiment analysis, LIWC, lexicon-based methods, support vector machine, random forest, naive Bayes, method comparison, market analysis, unstructured text, small sample size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
