CoChat

Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages

Open access
Törnberg, Petter · 2025 · Social Science Computer Review

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; none of the 11 sources its claims cite is checked for support yet.

Abstract

Instruction-tuned Large Language Models (LLMs) have recently emerged as a powerful new tool for text analysis. As these models are capable of zero-shot annotation based on instructions written in natural language, they obviate the need of large sets of training data—and thus bring potential paradigm-shifting implications for using text as data. While the models show substantial promise, their relative performance compared to human coders and supervised models remains poorly understood and subject to significant academic debate. This paper assesses the strengths and weaknesses of popular fine-tuned AI models compared to both conventional supervised classifiers and manual annotation by experts and crowd workers. The task used is to identify the political affiliation of politicians based on a single X/Twitter message, focusing on data from 11 different countries. The paper finds that GPT-4 achieves higher accuracy than both supervised models and human coders across all languages and country contexts. In the US context, it achieves an accuracy of 0.934 and an inter-coder reliability of 0.982. Examining the cases where the models fail, the paper finds that the LLM—unlike the supervised models—correctly annotates messages that require interpretation of implicit or unspoken references, or reasoning on the basis of contextual knowledge—capacities that have traditionally been understood to be distinctly human. The paper thus contributes to our understanding of the revolutionary implications of LLMs for text analysis within the social sciences.

Overview

CoChat Summary

TL;DR

This paper compares zero-shot GPT-4 annotation of politicians’ party affiliation from single X/Twitter messages against supervised classifiers and human coders across 11 countries. GPT-4 achieves the highest accuracy in all countries, including 0.934 accuracy and 0.982 inter-coder reliability in the US, and succeeds on interpretive cases where supervised models fail.

Key findings

  • In the US case, GPT-4 has the highest accuracy and macro F1 of all methods: GPT-4 0.934/0.934, Naïve Bayes 0.792/0.791, BERT 0.808/0.808, MTurk average 0.804/0.803, MTurk plurality 0.854/0.853, experts average 0.830/0.830, experts plurality 0.860/0.860; GPT-4 is statistically significantly better than all other models with 95% confidence.
  • Across 11 countries and languages, GPT-4 has the highest accuracy of all classifiers for all countries, though accuracy is lower outside the United States; the differences between GPT-4 and supervised models are not significant for all countries.
  • Failure analysis shows supervised models fail on messages lacking party-associated words but trivial for humans, while GPT-4 correctly annotated all identified cases where human coders agreed on a correct answer and the LLM was incorrect; GPT-4 also correctly interpreted implicit references, contextual knowledge, and author intentions.
  • The model results were found to be stable to variations in instructions (see Figure SI.2).

Methods

  • Task: binary, balanced classification of a politician’s party from a single X/Twitter message, using van Vliet et al.’s database across 11 countries; two largest parties per country; random sample of 250 messages per party (500 total) for most countries, and 100 per party for Sweden, New Zealand, and Poland; retweets, replies, URLs, and messages under 100 characters were removed.
  • Models: zero-shot GPT-4 (gpt-4-0314) via API with a natural-language prompt; Naïve Bayes bag-of-words baseline; fine-tuned BERT (bert-base-uncased, 3 epochs, batch size 16 training/64 evaluation, AdamW, warmup 500 steps, weight decay 0.01); for the US only, MTurk Master workers (10 independent workers per question, with control questions) and three PhD expert coders.
  • Analysis: accuracy and macro F1; Krippendorff’s Alpha for inter-coder reliability; manual examination of all model failures, with follow-up requests for GPT-4 to “motivate your response.”

Limitations

  • Paper-stated limitations: the task may be differently challenging across countries due to political differences, and further research is needed to compare LLM capacity across country contexts; the model’s capacity on this specific task may not generalize to other text annotation tasks.
  • Evident in the text: human comparison (MTurk and experts) is only available for the US; some country samples are smaller (100 messages per party); LLM motivations are described as potentially post-hoc rationalizations; the GPT-4 version is fixed at gpt-4-0314; and MTurk was run in early March 2023, before LLM use among crowd workers was widespread.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/08944393241286471

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 A comparison of accuracy between the models for the US case, with 95% con fi dence interval. As can be seen, the LLM statistically signi fi cantly outperforms all other classi fi ers in terms of accuracy.
Figure 1 A comparison of accuracy between the models for the US case, with 95% con fi dence interval. As can be seen, the LLM statistically signi fi cantly outperforms all other classi fi ers in terms of accuracy.CC-BY · From the PDF
Figure 2 Comparing the accuracy of the model across 11 countries. The LLM has the highest accuracy in the United States, but the LLM has the highest accuracy of all classi fi ers across all countries and languages. The graph shows the 95% con fi dence interval.
Figure 2 Comparing the accuracy of the model across 11 countries. The LLM has the highest accuracy in the United States, but the LLM has the highest accuracy of all classi fi ers across all countries and languages. The graph shows the 95% con fi dence interval.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 1 Not valid · 12 Not checked · 83 Own results and interpretations

20 more claims

Create a free account to see all 96 claims and to verify the 6 references still open.

See all claims — free account
  • It may state two separate claims as one. 40
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 49

Create a free account to see these claims.

Citations

Cites

Show all 50 references

Cited by

See all 249

Metadata

Authors
Törnberg, P.
Affiliations
University of Neuchâtel (CH)
Corresponding author
Törnberg, Petter
Venue
Social Science Computer Review
Published
2024-09-22
Volume
43
Issue
6
Pages
1181-1195
ISSN
  • 0894-4393 (ISSN-L)
  • 1552-8286
OpenAlex
W4402740791
Cited by
249
References
43
Citations by year
2026: 166 · 2025: 80 · 2024: 3
FWCI
76.58
Type
journal article
Language
en
Open access
Licence
Locations
  • Social Science Computer Review (publishedVersion, cc-by)
  • UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
Funders
Sociale en Geesteswetenschappen, NWO (VI.Veni.201S.006)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, Twitter, GPT-4, intercoder reliability, supervised classifiers, implicit references, contextual reasoning, text-as-data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. https://app.cochat.ai/papers/10.1177/08944393241286471
The paper
Törnberg, P. (2025). Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. Social Science Computer Review. https://doi.org/10.1177/08944393241286471

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.