CoChat

Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research

Open access
Lin, Hao, Zhang, Yongjun · 2026 · Social Science Computer Review

The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index; none of the 22 sources its claims cite is checked for support yet.

Abstract

Large language models (LLMs) have the potential to revolutionize computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks associated with using LLMs for text classification tasks, using social movement studies as an example. We propose a framework for social scientists to incorporate LLMs into text annotation, either as the primary coding decision-maker or as a coding assistant. This framework offers researchers tools to develop the potential best-performing prompt, and to systematically examine and report the validity and reliability of LLMs as a methodological tool. Additionally, we evaluate and discuss its epistemic risks associated with validity, reliability, replicability, and transparency. We conclude with several practical guidelines for using LLMs in text annotation tasks and offer recommendations for more effectively communicating epistemic risks in research.

Overview

CoChat Summary

TL;DR

This paper systematically evaluates OpenAI GPT-4 (and Meta Llama3-8B) as text annotators for social movement coding tasks, benchmarking against the human-coded Dynamics of Collective Action (DoCA) dataset. It proposes a workflow in which LLMs act either as primary coders or as coding assistants, and argues that validity, reliability, replicability, and transparency risks grow with task complexity.

Key findings

  • GPT-4 accuracy/F1 declined as task complexity rose: binary police presence scored 0.84/0.63 (zero-shot), 0.85/0.64 (zero-shot CoT), and 0.82/0.61 (zero-shot ToT); multiclass protest size scored 0.499/0.464, 0.484/0.449, and 0.461/0.426; multilabel protest activity scored 0.548, 0.409, and 0.564 (subset accuracy, no F1 reported).
  • Reliability also tracked complexity: across 25 repeated runs on 200 articles, ICC was 0.934 (95% CI [0.921, 0.946]) for police presence (excellent) but only 0.518 (95% CI [0.463, 0.577]) for protest size (moderate).
  • Prompting strategy mattered: zero-shot CoT outperformed others on binary classification, while zero-shot ToT worked better for multiclass classification.
  • Llama3-8B (4-bit quantized, CoT, temperature 0.2, run locally via ollama) performed worse than GPT-4: 0.762/0.566 (binary), 0.367/0.284 (multiclass), and 0.099 (multilabel), with runtimes of 1.6, 2.2, and 5.5 hours respectively.
  • Human review of 100 articles found GPT-4's step-by-step CoT reasoning factually correct, while Llama3 made inference errors (e.g., inferring police presence not stated in the article); most police-presence predictions had token probabilities above 95%, with 137 cases below that threshold flagged for further investigation.

Methods

  • Design: systematic evaluation of LLM coding against human-coded ground truth, using social movement studies as an illustrative domain; three task types (binary police presence, multiclass protest size on a 1–6 scale, multilabel protest activity scored by subset accuracy).
  • Data: the DoCA project's annotated New York Times articles (1960–1995) aligned with the Linguistic Data Consortium NYT corpus; a 200-article random subset for reliability testing and 100 articles for manual evaluation of reasoning outputs.
  • Analysis: GPT-4-turbo queried with zero-shot role-play, zero-shot CoT, and zero-shot ToT prompts; intraclass correlation (ICC) across 25 repeated runs treated as raters, using Koo and Li's (2016) thresholds; logprob values inspected for prediction confidence; Llama3-8B compared under the same zero-shot CoT prompt.

Limitations

  • Stated by the paper: only zero-shot classification with different prompting strategies was tested (few-shot prompting and fine-tuning left untested); rapid advances in generative AI may mitigate some risks; findings are specific to social movement studies and may not generalize across domains; only GPT-4 and Llama3-8B were compared, not the newer Llama3.1-405B.
  • Evident from the text: the reliability demonstration used only two tasks (police presence and protest size) on a 200-article subset, and the manual reasoning audit covered only 100 articles, so the transparency findings rest on a small sample; the paper reports accuracy and F1 for some tasks but only subset accuracy for the multilabel task, limiting comparability across tasks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/08944393251366243

Cites

78 of 78 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 The Workflow of Using LLM for Text Annotation. Note: the LLM icon was created using GPT-4o.
Figure 1 The Workflow of Using LLM for Text Annotation. Note: the LLM icon was created using GPT-4o.CC-BY · From the PDF
Figure 2 The Consistency of 25 Repeated Outcomes.
Figure 2 The Consistency of 25 Repeated Outcomes.CC-BY · From the PDF
Figure 3 Histogram of Reported Probability for Police Presence
Figure 3 Histogram of Reported Probability for Police PresenceCC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 1 Not valid · 18 Not checked · 38 Own results and interpretations

25 more claims

Create a free account to see all 57 claims and to verify the 20 references still open.

See all claims — free account
  • It may state two separate claims as one. 15
  • Probably not a substantial claim. 23
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 78 references

Research works printed but not linked: 1

  1. Do S., Ollion E., Shen R. (2022). The augmented social scientist: Using sequential transfer learning to annotate millions of texts with human-level accuracy. Sociological Methods & Research, 53(3), Article 004912412211345. https://doi.org/10.1177/00491241221134526 · DOI · Paper page

Cited by

See all 33

Metadata

Authors
Lin, H., & Zhang, Y.
Affiliations
Stony Brook University (US)
Corresponding author
Zhang, Yongjun
Venue
Social Science Computer Review
Published
2025-08-08
Volume
44
Issue
3
Pages
403-427
ISSN
  • 0894-4393 (ISSN-L)
  • 1552-8286
OpenAlex
W4413121431
Cited by
33
References
69
Citations by year
2026: 26 · 2025: 6
FWCI
15.15
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Social Science Computer Review (publishedVersion)
  • SocArXiv (OSF Preprints) (submittedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
large language models, textual annotation, text classification, social movement studies, automated coding, prompt engineering, epistemic risk, reliability and validity, replicability, transparency, computational social science, research guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research. https://app.cochat.ai/papers/10.1177/08944393251366243
The paper
Lin, H., & Zhang, Y. (2026). Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research. Social Science Computer Review. https://doi.org/10.1177/08944393251366243

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.