Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; none of the 11 sources its claims cite is checked for support yet.
Abstract
Instruction-tuned Large Language Models (LLMs) have recently emerged as a powerful new tool for text analysis. As these models are capable of zero-shot annotation based on instructions written in natural language, they obviate the need of large sets of training data—and thus bring potential paradigm-shifting implications for using text as data. While the models show substantial promise, their relative performance compared to human coders and supervised models remains poorly understood and subject to significant academic debate. This paper assesses the strengths and weaknesses of popular fine-tuned AI models compared to both conventional supervised classifiers and manual annotation by experts and crowd workers. The task used is to identify the political affiliation of politicians based on a single X/Twitter message, focusing on data from 11 different countries. The paper finds that GPT-4 achieves higher accuracy than both supervised models and human coders across all languages and country contexts. In the US context, it achieves an accuracy of 0.934 and an inter-coder reliability of 0.982. Examining the cases where the models fail, the paper finds that the LLM—unlike the supervised models—correctly annotates messages that require interpretation of implicit or unspoken references, or reasoning on the basis of contextual knowledge—capacities that have traditionally been understood to be distinctly human. The paper thus contributes to our understanding of the revolutionary implications of LLMs for text analysis within the social sciences.
Overview
CoChat Summary
TL;DR
This paper compares zero-shot GPT-4 annotation of politicians’ party affiliation from single X/Twitter messages against supervised classifiers and human coders across 11 countries. GPT-4 achieves the highest accuracy in all countries, including 0.934 accuracy and 0.982 inter-coder reliability in the US, and succeeds on interpretive cases where supervised models fail.
Key findings
- In the US case, GPT-4 has the highest accuracy and macro F1 of all methods: GPT-4 0.934/0.934, Naïve Bayes 0.792/0.791, BERT 0.808/0.808, MTurk average 0.804/0.803, MTurk plurality 0.854/0.853, experts average 0.830/0.830, experts plurality 0.860/0.860; GPT-4 is statistically significantly better than all other models with 95% confidence.
- Across 11 countries and languages, GPT-4 has the highest accuracy of all classifiers for all countries, though accuracy is lower outside the United States; the differences between GPT-4 and supervised models are not significant for all countries.
- Failure analysis shows supervised models fail on messages lacking party-associated words but trivial for humans, while GPT-4 correctly annotated all identified cases where human coders agreed on a correct answer and the LLM was incorrect; GPT-4 also correctly interpreted implicit references, contextual knowledge, and author intentions.
- The model results were found to be stable to variations in instructions (see Figure SI.2).
Methods
- Task: binary, balanced classification of a politician’s party from a single X/Twitter message, using van Vliet et al.’s database across 11 countries; two largest parties per country; random sample of 250 messages per party (500 total) for most countries, and 100 per party for Sweden, New Zealand, and Poland; retweets, replies, URLs, and messages under 100 characters were removed.
- Models: zero-shot GPT-4 (gpt-4-0314) via API with a natural-language prompt; Naïve Bayes bag-of-words baseline; fine-tuned BERT (bert-base-uncased, 3 epochs, batch size 16 training/64 evaluation, AdamW, warmup 500 steps, weight decay 0.01); for the US only, MTurk Master workers (10 independent workers per question, with control questions) and three PhD expert coders.
- Analysis: accuracy and macro F1; Krippendorff’s Alpha for inter-coder reliability; manual examination of all model failures, with follow-up requests for GPT-4 to “motivate your response.”
Limitations
- Paper-stated limitations: the task may be differently challenging across countries due to political differences, and further research is needed to compare LLM capacity across country contexts; the model’s capacity on this specific task may not generalize to other text annotation tasks.
- Evident in the text: human comparison (MTurk and experts) is only available for the US; some country samples are smaller (100 messages per party); LLM motivations are described as potentially post-hoc rationalizations; the GPT-4 version is fixed at gpt-4-0314; and MTurk was run in early March 2023, before LLM use among crowd workers was widespread.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
0 Valid · 1 Not valid · 12 Not checked · 83 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, the Naïve Bayes has the worst performance of all models-but still achieves a respectable 79.2% accuracy”
§ Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The LLM also correctly interprets messages that require making inferences based on the assumptions of the aims and intentions of the poster.”
§ Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages > In-Depth Examination of Model Failures
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, there are several examples of the LLM outperforming both MTurkers and experts on tasks involving subjective and contextual interpretation-seemingly as the LLM simply does a better job of scoping the motivations of the poster and making valid inductions.”
§ Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages > In-Depth Examination of Model Failures
hedgedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The LLM has the highest accuracy in the United States, but the LLM has the highest accuracy of all classifiers across all countries and languages.”
§ Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages > Discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The examination reveals clearly that the models display different patterns of failure.”
§ Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages > In-Depth Examination of Model Failures
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 96 claims and to verify the 6 references still open.
- It may state two separate claims as one. 40
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
- Probably not a substantial claim. 49
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 1×
“However, manual annotation is also not without its own issues (Murthy et al., 2019; Van Atteveldt et al., 2021), as humans can be slow, costly, biased, and prone to fatigue, all of which can undermine data quality and limit the scope of analysis (Grimmer & Stewart, 2013; Neuendorf, 2017).”
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
The Structure of Scientific Revolutions
Hall, M. B., & Kuhn, T. S. (1963) · The American Historical Review · doi:10.2307/1847040 · Cited by 7,479
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Cited by
Best Practices for Text Annotation with Large Language Models
Törnberg, P. (2024) · Sociologica · doi:10.48550/arxiv.2402.05129 · Cited by 230
Self-citationWe Reject the Use of Generative Artificial Intelligence for Reflexive Qualitative Research
Jowsey, T., Braun, V., Clarke, V., Lupton, D. A., & Fine, M. (2025) · Qualitative Inquiry · doi:10.1177/10778004251401851 · Cited by 188
Generative Artificial Intelligence in Qualitative Data Analysis: Analyzing—Or Just Chatting?
Nguyen, D. C., & Welch, C. (2025) · Organizational Research Methods · doi:10.1177/10944281251377154 · Cited by 124
Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text AnnotationPreprint
Baumann, J., Röttger, P., Urman, A., Wendsjö, A., Plaza-del-Arco, F. M., Gruber, J. B., & Hovy, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.08825 · Cited by 107
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 84
Metadata
- Authors
- Törnberg, P.
- Affiliations
- University of Neuchâtel (CH)
- Corresponding author
- Törnberg, Petter
- ORCID
- Törnberg, Petter
- Venue
- Social Science Computer Review
- Publisher
- SAGE Publications
- Published
- 2024-09-22
- Volume
- 43
- Issue
- 6
- Pages
- 1181-1195
- ISSN
- 0894-4393 (ISSN-L)
- 1552-8286
- OpenAlex
- W4402740791
- Cited by
- 249
- References
- 43
- Citations by year
- 2026: 166 · 2025: 80 · 2024: 3
- FWCI
- 76.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (vor)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Social Science Computer Review (publishedVersion, cc-by)
- UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
- Funders
- Sociale en Geesteswetenschappen, NWO (VI.Veni.201S.006)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, Twitter, GPT-4, intercoder reliability, supervised classifiers, implicit references, contextual reasoning, text-as-data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. https://app.cochat.ai/papers/10.1177/08944393241286471
- The paper
- Törnberg, P. (2025). Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. Social Science Computer Review. https://doi.org/10.1177/08944393241286471
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.