Sentiment Analysis in the Age of Generative AI
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index; 9 of the 35 sources its claims cite were checked for support.
Abstract
In the rapidly advancing age of Generative AI, Large Language Models (LLMs) such as ChatGPT stand at the forefront of disrupting marketing practice and research. This paper presents a comprehensive exploration of LLMs’ proficiency in sentiment analysis, a core task in marketing research for understanding consumer emotions, opinions, and perceptions. We benchmark the performance of three state-of-the-art LLMs, i.e., GPT-3.5, GPT-4, and Llama 2, against established, high-performing transfer learning models. Despite their zero-shot nature, our research reveals that LLMs can not only compete with but in some cases also surpass traditional transfer learning methods in terms of sentiment classification accuracy. We investigate the influence of textual data characteristics and analytical procedures on classification accuracy, shedding light on how data origin, text complexity, and prompting techniques impact LLM performance. We find that linguistic features such as the presence of lengthy, content-laden words improve classification performance, while other features such as single-sentence reviews and less structured social media text documents reduce performance. Further, we explore the explainability of sentiment classifications generated by LLMs. The findings indicate that LLMs, especially Llama 2, offer remarkable classification explanations, highlighting their advanced human-like reasoning capabilities. Collectively, this paper enriches the current understanding of sentiment analysis, providing valuable insights and guidance for the selection of suitable methods by marketing researchers and practitioners in the age of Generative AI.
Overview
CoChat Summary
TL;DR
This paper benchmarks GPT-3.5, GPT-4, and Llama 2 against fine-tuned transfer learning models for sentiment analysis, finding that LLMs are competitive in zero-shot binary and three-class tasks but not uniformly superior. It also shows that data origin, text characteristics, prompting method, and explainability shape LLM performance.
Key findings
- In binary zero-shot sentiment classification, SiEBERT had the highest average accuracy at 95.8%, followed by GPT-4 at 93.1%, Llama 2 at 90.9%, and GPT-3.5 at 87.7%. In three-class classification, GPT-4 averaged 82.6%, while GPT-3.5 and RoBERTa each averaged 77.7%, and Llama 2 averaged 75.7%.
- Logistic regression across 7,096 documents found that Twitter origin lowered odds of correct prediction by about 65%, three-class tasks lowered odds by approximately 54% to 63%, and “Big Words” positively predicted accuracy across all LLMs. Single-sentence documents reduced accuracy, especially for GPT-4 and Llama 2.
- Prompting effects varied: GPT-3.5 benefited from contextual prompts (odds ratio 1.382, p < 0.001), while GPT-4 and Llama 2 did not significantly benefit. Few-shot prompting improved GPT-4 in seven of eight datasets and GPT-3.5 in five of eight, but reduced Llama 2 accuracy across all datasets, by as much as 8.6 percentage points.
- In explainability, 15 marketing academics rated 96 LLM-generated explanations (1,440 votes). Llama 2 received the highest mean ratings, closely followed by GPT-4, while GPT-3.5 scored lowest; trustworthiness was the lowest-rated dimension for all LLMs.
Methods
- Experiment 1 compared GPT-3.5, GPT-4, and Llama 2 in zero-shot binary and three-class sentiment classification against transfer learning models including SiEBERT and RoBERTa. Binary used 3,120 unique documents from 16 datasets; three-class used 792 documents from four datasets, with 20 datasets total and over 3,900 unique documents overall.
- Experiment 2 tested contextual versus instructive prompts and few-shot versus zero-shot prompting on eight datasets and 1,592 documents. It extracted LIWC and TextAnalyzer linguistic features and ran logistic regressions on 7,096 documents across experiments, with prediction correctness as the dependent variable.
- Experiment 3 surveyed 15 marketing academics who rated 96 classification explanations on understandability, level of detail, and trustworthiness using five-point Likert scales. An OLS regression examined determinants of explainability, including word count and citations.
Limitations
- The paper states that it evaluates only off-the-shelf zero-shot and few-shot LLM use, without fine-tuning or task-specific training, so results likely represent a lower bound of LLM potential. It also notes that aspect-based sentiment analysis and non-text modalities were not tested.
- The paper states reproducibility concerns: temperature zero yields only near-deterministic outputs, closed-source models can change over time, and prompt adjustments can substantially affect performance. It also notes practical constraints around fine-tuning access, proprietary data, and computational cost.
- Evident from the text: the explainability experiment relies on a small survey sample of 15 academics rating 96 explanations, and the benchmark covers a limited set of datasets and model versions, which may constrain generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 17 Not checked · 139 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Demonstrating a significant advancement, Hartmann et al. (2023) find a superior performance of transfer learning models in sentiment analysis tasks compared to lexicon-based and traditional machine learning approaches, by 20 and 10 percentage points in accuracy, respectively [24].”
§ Sentiment Analysis in the Age of Generative AI > 1 Introduction
What the sources say
- Not checked yetcites Hartmann et al. (2023)
- Supportedcites [24] Hartmann J, Heitmann M, Siebert C et al (2023) More than a Feeling: Accuracy and Application of Sentiment Analysis. Int J Res Mark 40(1):75–87. https://doi.org/10.1016/j.ijresmar.2022.05.005 source
The Data Description & Model-Free Results section reports that transfer learning outperformed lexicons by 20.22 percentage points and traditional machine learning by 10.77 percentage points in average classification accuracy (p < .01 for both). The citing sentence rounds those figures to 20 and 10 percentage points, respectively. (LLM, conf=0.95)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“RoBERTa, specifically fine-tuned on a corpus of over 5,000 social media posts [23], exhibits the highest accuracy in one of the Twitter dataset (see Table 2, Twitter Various II).”
§ Sentiment Analysis in the Age of Generative AI > 3.3 Results Three-Class Sentiment Analysis
What the sources say
- Partly supportedcites [23] Hartmann J, Heitmann M, Schamp C et al (2021) The Power of Brand Selfies. J Mark Res 58(6):1159–1177. https://doi.org/10.1177/00222437211037258 sourceExperimental — accuracy not yet measured
Our copy supports the identification half of the claim: Automated Text Classification (p. 9) states they 'classify the sentiment of the caption as positive, neutral, or negative by fine-tuning the RoBERTa language model on 5,304 manually annotated posts (86.1% accuracy for the three classes on 1,327 holdout observations)', i.e. RoBERTa fine-tuned on a corpus of over 5,000 social media posts. However, nothing in our copy supports the comparative assertion that RoBERTa 'exhibits the highest accuracy in one of the Twitter dataset': the paper reports no cross-model or cross-dataset accuracy ranking (searched: classifier accuracy comparison, benchmark/other models). The only comparison it draws is that LIWC-based sentiment 'often produces relatively low levels of accuracy' (p. 9, footnote 5), which credits superior accuracy to their own model rather than establishing a highest-accuracy result on a named Twitter dataset — that ranking appears to come from the citing paper's own Table 2. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Recent studies have indicated that the efficiency of LLMs varies with the nature of the prompts employed [8,57,71].”
§ Sentiment Analysis in the Age of Generative AI > Hartmann et al. (2019 & 2023
What the sources say
- Not checked yetcites [8] Brown T, Mann B, Ryder N et al (2020) Language Models are Few-Shot Learners. In: Larochelle H, Ranzato M, Hadsell R et al (eds) Advances in Neural Information Processing Systems, vol 33. Curran Associates Inc, pp 1877–1901
- Not checked yetcites [57] Reynolds L, McDonell K (2021) Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In: Kitamura Y, Quigley A, Isbister K et al (eds) Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, pp 1–7
- Supportedcites [71] Zhou Y, Muresanu AI, Han Z et al. (2022) Large Language Models Are Human-Level Prompt Engineers. arXiv. https://doi.org/10.48550/arXiv.2211.01910 source
The Abstract (p. 1) states that 'task performance depends significantly on the quality of the prompt used to steer the model,' and the Introduction (pp. 1-2) adds that 'plain language prompts do not always produce the desired results, even when those results are possible to produce with alternative instructions' and that instruction quality 'can only be measured when executing these instructions on a downstream task.' This matches the attributed point that LLM outcomes vary with the prompt used; only the word 'efficiency' is looser than the paper's 'performance/accuracy'. (LLM, conf=0.85)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Conversely, transfer learning models such as SiEBERT only provide explainability when used in conjunction with other models, such as LIME [21].”
§ Sentiment Analysis in the Age of Generative AI > 6.1 Summary
What the sources say
- Partly supportedcites [21] Hartmann J, Bergner A, Hildebrand C (2023) MindMiner: Uncovering linguistic markers of mind perception as a new lens to understand consumer–smart object relationships. J Consum Psychol 33(4):645–667. https://doi.org/10.1002/jcpy.1381 sourceExperimental — accuracy not yet measured
Our copy supports the general proposition: 'Detecting mind perception using language models' states that language models are 'often considered a "black box," sacrificing interpretability for prediction accuracy' and that 'the current research combines the predictive power of language models with the interpretability of other text-mining methods (i.e., ... semi-supervised LDA, LIME, and LASSO),' and 'Methodological contributions' adds that 'even though language models are often thought of as black boxes ... it is possible to obtain post hoc explanations of such models through interpretable machine learning techniques,' with 'Model-free evidence' showing LIME approximating RoBERTa's nonlinear boundary locally. However, the paper's transfer-learning model is MindMiner, a fine-tuned RoBERTa (BERT-derived) language model; it never mentions SiEBERT/sentence-BERT, and its benchmarked alternatives are word2vec and fastText, so attributing the explainability-with-LIME point specifically to SiEBERT goes beyond what our copy says. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to the findings of other researchers [56,58], this variability in results, which depends on the prompts chosen, poses a reproducibility challenge that is particularly important for researchers in the field of Generative AI.”
§ Sentiment Analysis in the Age of Generative AI > 4.5 Discussion
What the sources say
- Supportedcites [56] Reiss MV (2023) Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark. arXiv. https://doi.org/10.48550/arXiv.2304.11085 source
The Conclusion says the exact prompt is decisive and that even tiny wording changes can produce inconsistent outputs. The Discussion calls this a problem for research using ChatGPT for text annotation and classification, supporting the cited reproducibility concern. (LLM, conf=0.87)
- Partly supportedcites [58] Rytting CM, Sorensen T, Argyle L et al. (2023) Towards Coding Social Science Datasets with Language Models. arXiv. https://doi.org/10.48550/arXiv.2306.02177 sourceExperimental — accuracy not yet measured
The Additional information section says small prompt-format changes can significantly alter token probabilities, and Accuracy with Additional Exemplars reports that one choice of response wording significantly worsened performance. But the paper also says prompt engineering generally made little difference to coding performance and that results across many format variations were relatively stable. It does not characterize prompt-dependent results as a particular reproducibility challenge for generative-AI researchers. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
98 more claims
Create a free account to see all 164 claims and to verify the 26 references still open.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 42
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A growing body of research suggests that the recent generation of large language model (LLMs) excel, and in many cases outpace humans, at writing persuasively and empathetically, at inferring user traits from text, and at mimicking human-like conversation believably and effectively—without possessing any true empathy or social understanding.”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [54]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition to persuasive writing, LLMs are now capable of reliably inferring emotion and sentiment from human-written text. In one study, a fine-tuned LLM was able to detect emotions in social media posts with 84% accuracy (55). Another study found precision levels of over 95% for LLMs assessing sentiment across multiple languages, with the ability to identify nuanced feelings such as nostalgia”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sentiment Analysis in the Age of Generative AI > 3.2 Results Binary Sentiment Analysis
“It surpasses all other traditional transfer learning models by a margin of at least four percentage points in 15 out of the 16 datasets.”
§ Sentiment Analysis in the Age of Generative AI > 7 Conclusion
“| Amazon (Various) | 94.0 | 94.0 | 95.6 | 95.9 | 96.5 | 90.5 | 94.5 | 95.0 | | Amazon Titles (Various) | 84.8 | 87.6 | 86.7 | 87.8 | 86.5 | 82.0 | 88.5 | 87.0 | | Yelp (Various) | 96.8 | 95.6 | 97.2 | 97.0 | 98.5 | 96.5 | 97.5 | 98.5 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sentiment Analysis in the Age of Generative AI > 6.1 Summary
“Interestingly, among the three LLMs, only GPT-3.5 benefited from few-shot and contextual prompting.”
§ Sentiment Analysis in the Age of Generative AI > 4.3 Results for Few-Shot Prompting
“For GPT-4, the introduction of few-shot prompting leads to an increase in prediction accuracy in seven out of eight datasets.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Sentiment Analysis in the Age of Generative AI > 4.4 Results of Logistic Regression Analysis: Impact of Prompting Method and Data Characteristics
“Third, sentiment prediction accuracy correlates positively and statistically significantly with word length, as evidenced by the presence of 'Big Words', across all LLMs (p < 0.001).”
§ Sentiment Analysis in the Age of Generative AI > 7 Conclusion
“| log(Big Words) | 1.082** (.03) | 1.137* (.05) | 1.117* (.046) |”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“5 [8], and an estimated 1.8 trillion for GPT-4 [59], full parameter updates are often impractical and highly cost-inefficient.”
InfluentialAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Thus, parameter-efficient fine-tuning methods, such as Low-Rank Adoption (LoRA) of LLMs, are increasingly utilized, optimizing only a subset of the model’s parameters [32].”
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 17×
“This tendency is more noticeable with datasets comprising Tweets, reflecting the colloquial and sometimes offensive language used on Twitter.”
InfluentialDwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., et al. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102642 · Cited by 4,584
Kincaid, J. P., Fishburne, Jr., Robert P., R., Richard L., C., & Brad, S. (1975) · doi:10.21236/ada006655 · Cited by 3,497 · cited 1×
“We extract features including word count, the frequency of lengthy words (more than seven letters), the proportion of capitalized words (e.g., “GREAT”), netspeak (a measure of informal language usage), and the grade level score (an indicator of text complexity) [37].”
What Makes Online Content Viral?
Berger, J., & Milkman, K. L. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0353 · Cited by 3,178 · cited 1×
“Applications of sentiment analysis include generating market insights from user-generated online content [45], predicting virality from linguistic features of newspaper articles [3], or identifying top performing individuals by analyzing the language style used in emails [67].”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 4×
“Meta has specifically focused on selecting and weighting its training data based on the truthfulness, toxicity, and bias inherent in the language when training Llama 2 [63].”
InfluentialExperimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150 · cited 2×
“First, Generative AI is used in content creation, including creative writing tasks [47, 55], conversational customer support [9], or market research based on emulated consumers [7, 40].”
Advances in natural language processing
Hirschberg, J., & Manning, C. D. (2015) · Science · doi:10.1126/science.aaa8685 · Cited by 1,712 · cited 1×
“Sentiment analysis based on natural language processing stands out as a prevalent use case in marketing, drawing significant attention across application contexts [2, 28, 65].”
Explainable AI: From black box to glass box
Rai, A. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00710-5 · Cited by 1,317 · cited 2×
“This distinction highlights the advanced natural language understanding and explanation synthesis inherent in the latest LLMs, challenging the notion of deep learning models as “black boxes” [53].”
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31161 · Cited by 995 · cited 1×
“Brynjolfsson et al. (2023) report a 14% productivity increase using Generative AI-based conversational assistants for customer service representatives [9].”
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 2×
“Recent studies have indicated that the efficiency of LLMs varies with the nature of the prompts employed [8, 57, 71].”
GoEmotions: A Dataset of Fine-Grained Emotions
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.372 · Cited by 873 · cited 1×
“Additionally, in a seven-class sentiment classification study on the Google Emotions Dataset [15], the accuracy gap between a fine-tuned transfer learning model (Emotion English DistilRoBERTa-base) [20] and the lowest-perform-ing LLM (Llama 2) was 16.4 percentage points, compared to 8.1 percentage…”
InfluentialGoEmotions: A Dataset of Fine-Grained Emotions
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020) · arXiv · doi:10.48550/arxiv.2005.00547 · Cited by 873
A review on sentiment analysis and emotion detection from text
Nandwani, P., & Verma, R. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00776-6 · Cited by 869 · cited 2×
“At the document level, sentiment is assessed for the entire text; at the sentence level, it is evaluated for individual sentences; and at the aspect level, sentiment is determined for specific elements within the text, e.g., certain product features [44].”
Prompt Engineering with ChatGPT: A Guide for Academic Writers
Giray, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03272-4 · Cited by 776
Mine Your Own Business: Market-Structure Surveillance Through Text Mining
Netzer, O., Feldman, R., Goldenberg, J., & Fresko, M. (2012) · Marketing Science · doi:10.1287/mksc.1120.0713 · Cited by 750 · cited 2×
“Recall is defined as the number of true positives over the number of false negatives plus the number of true positives [45].”
Uniting the Tribes: Using Text for Marketing Insight
Berger, J., Humphreys, A., Ludwig, S., Moe, W. W., Netzer, O., & Schweidel, D. A. (2020) · Journal of Marketing · doi:10.1177/0022242919873106 · Cited by 742 · cited 3×
“The role of automated text analysis in marketing, under-scored by Berger et al.(2020) [2], is set to further expand with the adoption of Generative AI, as it is expected to boost not only the accuracy but also the accessibility of text mining methods.”
Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks
You, Q., Luo, J., Jin, H., & Yang, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9179 · Cited by 548 · cited 2×
“Additionally, sentiment analysis can be conducted on images [69] and multi-modal data, such as videos combining image, audio and text [68], or social media posts that feature both images and text [31].”
Metrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518 · cited 1×
“…works” ), level of detail (“The explanation of the LLM is sufficiently detailed” ), and the trustworthiness (“The explanation lets me know how trustworthy the LLM is” ) on a five-point Likert scale, building on the “Explanation Good-ness Checklist” for explainable AI by Hoffmann et al. (2018) [29].”
Identifying Customer Needs from User-Generated Content
Timoshenko, A., & Hauser, J. R. (2019) · Marketing Science · doi:10.1287/mksc.2018.1123 · Cited by 513 · cited 1×
“Aspect-based sentiment analysis can help firms understand consumer needs [62] and motivations [11].”
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · cited 2×
“Differentiating model outputs under the same external parameter settings are particularly pronounced in closed-source models, which often undergo model updates that can significantly impact their performance [12].”
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507
Knowing What, How and Why: A Near Complete Solution for Aspect-Based Sentiment Analysis
Peng, H., Xu, L., Bing, L., Huang, F., Lu, W., & Si, L. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6383 · Cited by 439 · cited 1×
“Aspect-based sentiment analysis aims to extract specific aspects or features of the subject of the review (WHAT), classify the sentiment (HOW), and optionally explain the rationale (WHY) [52].”
Comparing automated text classification methods
Hartmann, J., Huppertz, J., Schamp, C., & Heitmann, M. (2019) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2018.09.009 · Cited by 399 · cited 4×
“The field of sentiment analysis employs diverse methods, ranging from lexicon-based approaches to traditional machine learning models, and extending to the more advanced transfer learning techniques, as detailed by Hartmann et al. (2019 & 2023) [24, 25].”
InfluentialSelf-citationMore than a Feeling: Accuracy and Application of Sentiment Analysis
Hartmann, J., Heitmann, M., Siebert, C., & Schamp, C. (2023) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2022.05.005 · Cited by 356 · cited 9×
“The field of sentiment analysis employs diverse methods, ranging from lexicon-based approaches to traditional machine learning models, and extending to the more advanced transfer learning techniques, as detailed by Hartmann et al. (2019 & 2023) [24, 25].”
InfluentialSelf-citationLarge Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 2×
“Recent studies have indicated that the efficiency of LLMs varies with the nature of the prompts employed [8, 57, 71].”
The Role of Marketer-Generated Content in Customer Engagement Marketing
Meire, M., Hewett, K., Ballings, M., Kumar, V., & Van den Poel, D. (2019) · Journal of Marketing · doi:10.1177/0022242919873903 · Cited by 298 · cited 1×
“This decision can vary from binary tasks like the prediction of positive or negative sentiment from social media posts [42] or the detection of firestorms [19] to more complex multi-class tasks like identifying emotions from email headlines [46].”
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2308950121 · Cited by 297 · cited 4×
“The emergence of Generative AI is revolutionizing the marketing landscape through its dual role : it serves to both generate [9, 22, 55] and analyze content [36, 38, 41, 54, 66].”
InfluentialThe power of generative marketing: Can generative AI create superhuman visual marketing content?
Hartmann, J., Exner, Y., & Domdey, S. (2025) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.09.002 · Cited by 229
Self-citationMeasuring and Managing Consumer Sentiment in an Online Community Environment
Homburg, C., Ehm, L., & Artz, M. (2015) · Journal of Marketing Research · doi:10.1509/jmr.11.0448 · Cited by 196
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190 · cited 1×
“First, Generative AI is used in content creation, including creative writing tasks [47, 55], conversational customer support [9], or market research based on emulated consumers [7, 40].”
Hartmann, J., Heitmann, M., Schamp, C., & Netzer, O. (2021) · Journal of Marketing Research · doi:10.1177/00222437211037258 · Cited by 184 · cited 1×
“RoBERTa, specifically fine-tuned on a corpus of over 5,000 social media posts [23], exhibits the highest accuracy in one of the Twitter dataset (see Table 2, Twitter Various II).”
Self-citationHansen, N., Kupfer, A.-K., & Hennig-Thurau, T. (2018) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2018.08.001 · Cited by 163 · cited 2×
“This decision can vary from binary tasks like the prediction of positive or negative sentiment from social media posts [42] or the detection of firestorms [19] to more complex multi-class tasks like identifying emotions from email headlines [46].”
Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis
Li, P., Castelo, N., Katona, Z., & Sarvary, M. (2024) · Marketing Science · doi:10.1287/mksc.2023.0454 · Cited by 159 · cited 1×
“First, Generative AI is used in content creation, including creative writing tasks [47, 55], conversational customer support [9], or market research based on emulated consumers [7, 40].”
Kheiri, K., & Karimi, H. R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10234 · Cited by 159 · cited 5×
“Second, Generative AI is pioneered in zero-shot content analysis, which includes areas like visual analysis [38] and automated textual analysis [36, 41, 54, 66].”
InfluentialJiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143
Multimodal Sentiment Analysis To Explore the Structure of Emotions
Hu, A., & Flaxman, S. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219853 · Cited by 137 · cited 2×
“Additionally, the application of sentiment analysis to real-time data streams such as social media posts [31] and live-streamed comments [13] presents opportunities for leveraging Generative AI in capturing immediate public sentiment.”
Global evidence of expressed sentiment alterations during the COVID-19 pandemic
Wang, J., Fan, Y., Palacios, J., Chai, Y., Guetta-Jeanrenaud, N., Obradovich, N., Zhou, C., & Zheng, S. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01312-y · Cited by 133 · cited 1×
“Sentiment analysis based on natural language processing stands out as a prevalent use case in marketing, drawing significant attention across application contexts [2, 28, 65].”
Video sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 131
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C., & Van Bavel, J. J. (2023) · doi:10.31234/osf.io/sekf5 · Cited by 129
Frontiers: Supporting Content Marketing with Natural Language Generation
Reisenbichler, M., Reutterer, T., Schweidel, D. A., & Dan, D. (2022) · Marketing Science · doi:10.1287/mksc.2022.1354 · Cited by 126 · cited 1×
“For instance, Reisenbichler et al. (2022) demonstrate that natural language generation for search engine optimization (SEO) outperforms content created by human writers in search engine rankings, increasing over-all campaign performance while reducing production costs [55].”
Zhang, Q., Wang, W., & Chen, Y. (2020) · Marketing Science · doi:10.1287/mksc.2019.1215 · Cited by 104 · cited 1×
“Sentiment analysis techniques can also be applied to process dynamic data for real-time information extraction [70].”
Auditing Elon Musk's Impact on Hate Speech and Bots
Hickey, D., Schmitz, M., Fessler, D., Smaldino, P. E., Muric, G., & Burghardt, K. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22222 · Cited by 100 · cited 17×
“This tendency is more noticeable with datasets comprising Tweets, reflecting the colloquial and sometimes offensive language used on Twitter.”
InfluentialChakraborty, I., Kim, M., & Sudhir, K. (2022) · Journal of Marketing Research · doi:10.1177/00222437211052500 · Cited by 84
Large language models predict human sensory judgments across six modalities
Marjieh, R., Sucholutsky, I., van Rijn, P., Jacoby, N., & Griffiths, T. L. (2024) · Scientific Reports · doi:10.1038/s41598-024-72071-1 · Cited by 73 · cited 2×
“The emergence of Generative AI is revolutionizing the marketing landscape through its dual role : it serves to both generate [9, 22, 55] and analyze content [36, 38, 41, 54, 66].”
The dangers of using proprietary LLMs for research
Ollion, É., Shen, R., Macanovic, A., & Chatelain, A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00783-6 · Cited by 62 · cited 1×
“Finally, reproducibility of results is a key concern in the application of LLMs in any research context, as high-lighted by Ollion et al. (2024) [48].”
Marketing insights from text analysis
Berger, J. A., Packard, G. M., Boghrati, R., Hsu, M., Humphreys, A., Luangrath, A. W., Moore, S. G., Nave, G., Olivola, C. Y., & Rocklage, M. D. (2022) · Marketing Letters · doi:10.1007/s11002-022-09635-6 · Cited by 53 · cited 1×
“Methods like lexicons offer explainability by assigning scores to individual words, enabling a weighted sum that provides a rationale for the final sentiment classification [4].”
Finding top performers through email patterns analysis
Wen, Q., Gloor, P. A., Colladon, A. F., Tickoo, P., & Joshi, T. (2019) · Journal of Information Science · doi:10.1177/0165551519849519 · Cited by 42 · cited 2×
“Applications of sentiment analysis include generating market insights from user-generated online content [45], predicting virality from linguistic features of newspaper articles [3], or identifying top performing individuals by analyzing the language style used in emails [67].”
Hartmann, J., Bergner, A., & Hildebrand, C. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1381 · Cited by 42 · cited 2×
“Conversely, transfer learning models such as SiEBERT only provide explainability when used in conjunction with other models, such as LIME [21].”
Self-citationNatural Language Processing in Marketing
Hartmann, J., & Netzer, O. (2023) · Artificial Intelligence in Marketing · doi:10.1108/s1548-643520230000020011 · Cited by 38
Self-citationThe power of generative marketing: Can generative AI create superhuman visual marketing content?Preprint
Hartmann, J., Exner, Y., & Domdey, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4597899 · Cited by 34 · cited 1×
“The emergence of Generative AI is revolutionizing the marketing landscape through its dual role : it serves to both generate [9, 22, 55] and analyze content [36, 38, 41, 54, 66].”
Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
Reiss, M. V. (2023) · arXiv · doi:10.48550/arxiv.2304.11085 · Cited by 34 · cited 1×
“Similar to the findings of other researchers [56, 58], this variability in results, which depends on the prompts chosen, poses a reproducibility challenge that is particularly important for researchers in the field of Generative AI.”
Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations
Huang, S., Mamidanna, S., Jangam, S., Zhou, Y., & Gilpin, L. H. (2023) · arXiv · doi:10.48550/arxiv.2310.11207 · Cited by 22 · cited 2×
“Given the human-like reasoning abilities of LLMs, the authors suggest reevaluating explain-ability assessments and recommend an alternative qualitative approach involving human subject studies [33].”
Unlimited Testing: Let’s Test Your Emails with AI
Nguyen, N., Johnson, J., & Tsiros, M. (2023) · Marketing Science · doi:10.1287/mksc.2021.0126 · Cited by 21 · cited 1×
“This decision can vary from binary tasks like the prediction of positive or negative sentiment from social media posts [42] or the detection of firestorms [19] to more complex multi-class tasks like identifying emotions from email headlines [46].”
Towards Coding Social Science Datasets with Language Models
Rytting, C. M., Sorensen, T., Argyle, L., Busby, E., Fulda, N., Gubler, J., & Wingate, D. (2023) · arXiv · doi:10.48550/arxiv.2306.02177 · Cited by 18 · cited 1×
“Similar to the findings of other researchers [56, 58], this variability in results, which depends on the prompts chosen, poses a reproducibility challenge that is particularly important for researchers in the field of Generative AI.”
Large language models for aspect-based sentiment analysis
Simmering, P. F., & Huoviala, P. (2023) · arXiv · doi:10.48550/arxiv.2310.18025 · Cited by 17 · cited 2×
“5 and GPT-4 in aspect-based sentiment analysis tasks when provided with six in-context examples, with no significant performance gain observed with more examples [61].”
Discovering Public Opinions by Performing Sentimental Analysis on Real Time Twitter Data
(2018) · doi:10.1109/iccsdet.2018.8821105 · Cited by 15 · cited 1×
“This includes the real-time analysis of Tweets, such as during elections to gauge public sentiment [10, 35], and the extraction of sentiment from live stream comments, which can assist content creators in adjusting their content in response to viewer feedback [13].”
Sentiment Analysis of Twitch.tv Livestream Messages using Machine Learning Methods
Chouhan, A., Halgekar, A., Rao, A., et al. (2021) · International Conference on Electrical, Computer and Communication Technologies (ICECCT) · doi:10.1109/icecct52121.2021.9616932 · Cited by 8 · cited 2×
“This includes the real-time analysis of Tweets, such as during elections to gauge public sentiment [10, 35], and the extraction of sentiment from live stream comments, which can assist content creators in adjusting their content in response to viewer feedback [13].”
Castellanos, M., Ghosh, R., Lu, Y., Zhang, L., Ruiz, P., Dekhil, M., Umeshwar Dayal, B., & Hsu, M. (2011) · doi:10.1145/1963192.1963287 · Cited by 4 · cited 1×
“This includes the real-time analysis of Tweets, such as during elections to gauge public sentiment [10, 35], and the extraction of sentiment from live stream comments, which can assist content creators in adjusting their content in response to viewer feedback [13].”
Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study
Wang, Z., Xie, Q., Feng, Y., Ding, Z., Yang, Z., & Xia, R. (2023) · arXiv · doi:10.48550/arxiv.2304.04339 · cited 5×
“Second, Generative AI is pioneered in zero-shot content analysis, which includes areas like visual analysis [38] and automated textual analysis [36, 41, 54, 66].”
Influential
Research works printed but not linked: 3
- Boyd RL, Ashokkumar A, Seraj S et al (2022) The development and psychometric properties of LIWC-22. University of Texas at Austin, Austin, TX, pp 1–47. https://www.liwc.app/static/documents/LIWC-22%20Manual%20-%20Development%20and%20Psychometrics.pdf
- Konrad M, Hartmann J (2023) One model fits all? Exploring the zero-shot capabilities of multimodal large language models for automated marketing image analytics. In: Proceeding of the 2023 Marketing Dynamics Conference
- Larochelle H, Dumitro E, Yoshua B (2008) Zero-Data Learning of New Tasks. AAAI 1(2):646–651
Cited by
The power of generative marketing: Can generative AI create superhuman visual marketing content?
Hartmann, J., Exner, Y., & Domdey, S. (2025) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.09.002 · Cited by 229
Self-citationUsing natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140
“LLMs offer zero-shot functionality, which means that LLMs do not need task-specific training but can simply annotate a text following a prompt (for example, a prompt to classify the sentiment of a social media post as positive, neutral or negative 41 ).”
Self-citationA review of sentiment analysis: tasks, applications, and deep learning techniques
Sharma, N. A., Ali, A. B. M. S., & Kabir, M. A. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00594-x · Cited by 242
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
A Comprehensive Survey of Artificial Intelligence Techniques for Talent Analytics
Qin, C., Zhang, L., Cheng, Y., Zha, R., Shen, D., Zhang, Q., Chen, X., Sun, Y., Zhu, C., Zhu, H., et al. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3572744 · Cited by 174
“However, recent advancements in LLMs have propelled sentiment analysis forward significantly [277].”
Metadata
- Authors
- Krugmann, J. O., & Hartmann, J.
- Affiliations
- Technical University of Munich (DE)
- Corresponding author
- Krugmann, Jan Ole
- Venue
- Customer Needs and Solutions
- Published
- 2024-03-05
- Volume
- 11
- Issue
- 1
- ISSN
- 2196-291X (ISSN-L)
- 2196-2928
- OpenAlex
- W4392462461
- Cited by
- 189
- References
- 55
- Citations by year
- 2026: 59 · 2025: 102 · 2024: 26
- FWCI
- 27.54
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Customer Needs and Solutions (publishedVersion, cc-by)
- Econstor (Econstor) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Technische Universität München
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- sentiment analysis, large language models, generative AI, sentiment classification, GPT-4, Llama 2, ChatGPT, consumer emotions, social media text, text complexity, prompting techniques, explainability, market analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.