Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Open accessThe DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index; none of the 22 sources its claims cite is checked for support yet.
Abstract
Large language models (LLMs) have the potential to revolutionize computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks associated with using LLMs for text classification tasks, using social movement studies as an example. We propose a framework for social scientists to incorporate LLMs into text annotation, either as the primary coding decision-maker or as a coding assistant. This framework offers researchers tools to develop the potential best-performing prompt, and to systematically examine and report the validity and reliability of LLMs as a methodological tool. Additionally, we evaluate and discuss its epistemic risks associated with validity, reliability, replicability, and transparency. We conclude with several practical guidelines for using LLMs in text annotation tasks and offer recommendations for more effectively communicating epistemic risks in research.
Overview
CoChat Summary
TL;DR
This paper systematically evaluates OpenAI GPT-4 (and Meta Llama3-8B) as text annotators for social movement coding tasks, benchmarking against the human-coded Dynamics of Collective Action (DoCA) dataset. It proposes a workflow in which LLMs act either as primary coders or as coding assistants, and argues that validity, reliability, replicability, and transparency risks grow with task complexity.
Key findings
- GPT-4 accuracy/F1 declined as task complexity rose: binary police presence scored 0.84/0.63 (zero-shot), 0.85/0.64 (zero-shot CoT), and 0.82/0.61 (zero-shot ToT); multiclass protest size scored 0.499/0.464, 0.484/0.449, and 0.461/0.426; multilabel protest activity scored 0.548, 0.409, and 0.564 (subset accuracy, no F1 reported).
- Reliability also tracked complexity: across 25 repeated runs on 200 articles, ICC was 0.934 (95% CI [0.921, 0.946]) for police presence (excellent) but only 0.518 (95% CI [0.463, 0.577]) for protest size (moderate).
- Prompting strategy mattered: zero-shot CoT outperformed others on binary classification, while zero-shot ToT worked better for multiclass classification.
- Llama3-8B (4-bit quantized, CoT, temperature 0.2, run locally via ollama) performed worse than GPT-4: 0.762/0.566 (binary), 0.367/0.284 (multiclass), and 0.099 (multilabel), with runtimes of 1.6, 2.2, and 5.5 hours respectively.
- Human review of 100 articles found GPT-4's step-by-step CoT reasoning factually correct, while Llama3 made inference errors (e.g., inferring police presence not stated in the article); most police-presence predictions had token probabilities above 95%, with 137 cases below that threshold flagged for further investigation.
Methods
- Design: systematic evaluation of LLM coding against human-coded ground truth, using social movement studies as an illustrative domain; three task types (binary police presence, multiclass protest size on a 1–6 scale, multilabel protest activity scored by subset accuracy).
- Data: the DoCA project's annotated New York Times articles (1960–1995) aligned with the Linguistic Data Consortium NYT corpus; a 200-article random subset for reliability testing and 100 articles for manual evaluation of reasoning outputs.
- Analysis: GPT-4-turbo queried with zero-shot role-play, zero-shot CoT, and zero-shot ToT prompts; intraclass correlation (ICC) across 25 repeated runs treated as raters, using Koo and Li's (2016) thresholds; logprob values inspected for prediction confidence; Llama3-8B compared under the same zero-shot CoT prompt.
Limitations
- Stated by the paper: only zero-shot classification with different prompting strategies was tested (few-shot prompting and fine-tuning left untested); rapid advances in generative AI may mitigate some risks; findings are specific to social movement studies and may not generalize across domains; only GPT-4 and Llama3-8B were compared, not the newer Llama3.1-405B.
- Evident from the text: the reliability demonstration used only two tasks (police presence and protest size) on a 200-article subset, and the manual reasoning audit covered only 100 articles, so the transparency findings rest on a small sample; the paper reports accuracy and F1 for some tasks but only subset accuracy for the multilabel task, limiting comparability across tasks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 1 Not valid · 18 Not checked · 38 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Multilabel (Activity) | 0.099, - |”
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Conclusion and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Binary (Police) | 0.762,0.566 |”
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Conclusion and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the accuracy scores fluctuate for both measures when repeated 25 times.”
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Are Generative AI coding results reliable?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments show that LLM performance varies across different types of prompting strategies.”
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Are Generative AI coding results replicable?
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Multiclass (Size) | 0.367,0.284 |”
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Conclusion and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 57 claims and to verify the 20 references still open.
- It may state two separate claims as one. 15
- Probably not a substantial claim. 23
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Are Generative AI coding results replicable?
“But for binary classification, we show that the zero-shot CoT outperforms others, while for multiclass classification, zero-shot ToT works better.”
§ The Risks of Using Large Language Models for Text Annotation in Social Science Research > Conclusion and Discussion
“| Multiclass (Size) | 0.499, 0.464 | 0.484, 0.449 | 0.461, 0.426 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“It is important to note that we are not AI-Luddites; as advancements continue, we anticipate improvements that may mitigate these limitations and challenges, enabling more effective and ethical use of LLMs in social science research.”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Structural Topic Models for Open‐Ended Survey Responses
Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder‐Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014) · American Journal of Political Science · doi:10.1111/ajps.12103 · Cited by 1,952
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
Research works printed but not linked: 1
- Do S., Ollion E., Shen R. (2022). The augmented social scientist: Using sequential transfer learning to annotate millions of texts with human-level accuracy. Sociological Methods & Research, 53(3), Article 004912412211345. https://doi.org/10.1177/00491241221134526 · DOI · Paper page
Cited by
Generative AI has lowered the barriers to computational social sciencesPreprint
Zhang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.10833 · Cited by 19
Self-citationEl‐Bassel, N., David, J., Mukherjee, T. I., Aggarwal, M., Wu, E., Gilbert, L., Walters, S. T., Chandler, R. K., Hunt, T., Frye, V. A., et al. (2025) · Implementation Science · doi:10.1186/s13012-025-01447-2 · Cited by 11
Gender disparity in U.S. patenting
Wang, J., & Maynard, A. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06038-6 · Cited by 10
Benchmarking Debiasing Methods for LLM-based Parameter Estimates
de Pieuchon, N. A., Daoud, A., Jerzak, C., Johansson, M., & Johansson, R. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1000 · Cited by 10
Vibe researching: can AI agents with skills replace or augment social scientists?
Zhang, Y. (2026) · Chinese Sociological Review · doi:10.1080/21620555.2026.2707167 · Cited by 8
Self-citation
Metadata
- Authors
- Lin, H., & Zhang, Y.
- Affiliations
- Stony Brook University (US)
- Corresponding author
- Zhang, Yongjun
- Venue
- Social Science Computer Review
- Publisher
- SAGE Publications
- Published
- 2025-08-08
- Volume
- 44
- Issue
- 3
- Pages
- 403-427
- ISSN
- 0894-4393 (ISSN-L)
- 1552-8286
- OpenAlex
- W4413121431
- Cited by
- 33
- References
- 69
- Citations by year
- 2026: 26 · 2025: 6
- FWCI
- 15.15
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Social Science Computer Review (publishedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- large language models, textual annotation, text classification, social movement studies, automated coding, prompt engineering, epistemic risk, reliability and validity, replicability, transparency, computational social science, research guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research. https://app.cochat.ai/papers/10.1177/08944393251366243
- The paper
- Lin, H., & Zhang, Y. (2026). Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research. Social Science Computer Review. https://doi.org/10.1177/08944393251366243
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


