Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index; 9 of the 35 sources its claims cite were checked for support.

Sentiment Analysis in the Age of Generative AI

Open access
Krugmann, Jan Ole, Hartmann, Jochen · 2024 · Customer Needs and Solutions

Abstract

In the rapidly advancing age of Generative AI, Large Language Models (LLMs) such as ChatGPT stand at the forefront of disrupting marketing practice and research. This paper presents a comprehensive exploration of LLMs’ proficiency in sentiment analysis, a core task in marketing research for understanding consumer emotions, opinions, and perceptions. We benchmark the performance of three state-of-the-art LLMs, i.e., GPT-3.5, GPT-4, and Llama 2, against established, high-performing transfer learning models. Despite their zero-shot nature, our research reveals that LLMs can not only compete with but in some cases also surpass traditional transfer learning methods in terms of sentiment classification accuracy. We investigate the influence of textual data characteristics and analytical procedures on classification accuracy, shedding light on how data origin, text complexity, and prompting techniques impact LLM performance. We find that linguistic features such as the presence of lengthy, content-laden words improve classification performance, while other features such as single-sentence reviews and less structured social media text documents reduce performance. Further, we explore the explainability of sentiment classifications generated by LLMs. The findings indicate that LLMs, especially Llama 2, offer remarkable classification explanations, highlighting their advanced human-like reasoning capabilities. Collectively, this paper enriches the current understanding of sentiment analysis, providing valuable insights and guidance for the selection of suitable methods by marketing researchers and practitioners in the age of Generative AI.

Overview

CoChat Summary

TL;DR

This paper benchmarks GPT-3.5, GPT-4, and Llama 2 against fine-tuned transfer learning models for sentiment analysis, finding that LLMs are competitive in zero-shot binary and three-class tasks but not uniformly superior. It also shows that data origin, text characteristics, prompting method, and explainability shape LLM performance.

Key findings

  • In binary zero-shot sentiment classification, SiEBERT had the highest average accuracy at 95.8%, followed by GPT-4 at 93.1%, Llama 2 at 90.9%, and GPT-3.5 at 87.7%. In three-class classification, GPT-4 averaged 82.6%, while GPT-3.5 and RoBERTa each averaged 77.7%, and Llama 2 averaged 75.7%.
  • Logistic regression across 7,096 documents found that Twitter origin lowered odds of correct prediction by about 65%, three-class tasks lowered odds by approximately 54% to 63%, and “Big Words” positively predicted accuracy across all LLMs. Single-sentence documents reduced accuracy, especially for GPT-4 and Llama 2.
  • Prompting effects varied: GPT-3.5 benefited from contextual prompts (odds ratio 1.382, p < 0.001), while GPT-4 and Llama 2 did not significantly benefit. Few-shot prompting improved GPT-4 in seven of eight datasets and GPT-3.5 in five of eight, but reduced Llama 2 accuracy across all datasets, by as much as 8.6 percentage points.
  • In explainability, 15 marketing academics rated 96 LLM-generated explanations (1,440 votes). Llama 2 received the highest mean ratings, closely followed by GPT-4, while GPT-3.5 scored lowest; trustworthiness was the lowest-rated dimension for all LLMs.

Methods

  • Experiment 1 compared GPT-3.5, GPT-4, and Llama 2 in zero-shot binary and three-class sentiment classification against transfer learning models including SiEBERT and RoBERTa. Binary used 3,120 unique documents from 16 datasets; three-class used 792 documents from four datasets, with 20 datasets total and over 3,900 unique documents overall.
  • Experiment 2 tested contextual versus instructive prompts and few-shot versus zero-shot prompting on eight datasets and 1,592 documents. It extracted LIWC and TextAnalyzer linguistic features and ran logistic regressions on 7,096 documents across experiments, with prediction correctness as the dependent variable.
  • Experiment 3 surveyed 15 marketing academics who rated 96 classification explanations on understandability, level of detail, and trustworthiness using five-point Likert scales. An OLS regression examined determinants of explainability, including word count and citations.

Limitations

  • The paper states that it evaluates only off-the-shelf zero-shot and few-shot LLM use, without fine-tuning or task-specific training, so results likely represent a lower bound of LLM potential. It also notes that aspect-based sentiment analysis and non-text modalities were not tested.
  • The paper states reproducibility concerns: temperature zero yields only near-deterministic outputs, closed-source models can change over time, and prompt adjustments can substantially affect performance. It also notes practical constraints around fine-tuning access, proprietary data, and computational cost.
  • Evident from the text: the explainability experiment relies on a small survey sample of 15 academics rating 96 explanations, and the benchmark covers a limited set of datasets and model versions, which may constrain generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s40547-024-00143-4

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

8 Valid · 0 Not valid · 17 Not checked · 139 Own results and interpretations

98 more claims

Create a free account to see all 164 claims and to verify the 26 references still open.

See all claims — free account
  • Probably not a substantial claim. 25
  • It may state two separate claims as one. 42
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 55 references

Research works printed but not linked: 3

  1. Boyd RL, Ashokkumar A, Seraj S et al (2022) The development and psychometric properties of LIWC-22. University of Texas at Austin, Austin, TX, pp 1–47. https://www.liwc.app/static/documents/LIWC-22%20Manual%20-%20Development%20and%20Psychometrics.pdf
  2. Konrad M, Hartmann J (2023) One model fits all? Exploring the zero-shot capabilities of multimodal large language models for automated marketing image analytics. In: Proceeding of the 2023 Marketing Dynamics Conference
  3. Larochelle H, Dumitro E, Yoshua B (2008) Zero-Data Learning of New Tasks. AAAI 1(2):646–651

Cited by

See all 195

Metadata

Authors
Krugmann, J. O., & Hartmann, J.
Affiliations
Technical University of Munich (DE)
Corresponding author
Krugmann, Jan Ole
Venue
Customer Needs and Solutions
Published
2024-03-05
Volume
11
Issue
1
ISSN
  • 2196-291X (ISSN-L)
  • 2196-2928
OpenAlex
W4392462461
Cited by
189
References
55
Citations by year
2026: 59 · 2025: 102 · 2024: 26
FWCI
27.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Customer Needs and Solutions (publishedVersion, cc-by)
  • Econstor (Econstor) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
Technische Universität München
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
sentiment analysis, large language models, generative AI, sentiment classification, GPT-4, Llama 2, ChatGPT, consumer emotions, social media text, text complexity, prompting techniques, explainability, market analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Sentiment Analysis in the Age of Generative AI. https://app.cochat.ai/papers/10.1007/s40547-024-00143-4
The paper
Krugmann, J. O., & Hartmann, J. (2024). Sentiment Analysis in the Age of Generative AI. Customer Needs and Solutions. https://doi.org/10.1007/s40547-024-00143-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.