Can Large Language Models Transform Computational Social Science?
Open accessThe DOI matches its Crossref record; 320 of the 324 references it cites were found in a registry or an index; none of the 111 sources its claims cite is checked for support yet.
Abstract
Large language models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the computational social science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers’ gold references. We conclude that the performance of today’s LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.
Overview
CoChat Summary
TL;DR
This paper evaluates whether zero-shot large language models can augment the Computational Social Science pipeline by benchmarking 13 LLMs on 25 English CSS tasks. It concludes that LLMs can assist with annotation and generation but cannot fully replace fine-tuned models or human analysis.
Key findings
- On classification, zero-shot LLMs fail to outperform the best fine-tuned models but still achieve fair agreement with humans: 8/17 tasks reach moderate-to-good agreement (κ = 0.40–0.65), while 5/17 (29.5%) show poor agreement. Best zero-shot performance reaches 77.4 F1 on misinformation detection, with κ = 0.55 versus the original paper’s κ = 0.51 inter-human agreement.
- On generation, LLMs often exceed crowdworkers’ gold references: model generations outrank the dataset’s gold human reference at least 38% of the time across five tasks. For COVIDET, Misinformation Explanation, and Social Bias Inference, gpt-3.5-turbo produces more faithful, coherent, and fluent text than both gold references and the fine-tuned baseline; for Positive Reframing and Figurative Language explanation, text-davinci-003 matches gold-standard levels.
- Model selection matters: larger instruction-tuned open-source FLAN models are preferable for classification, while gpt-3.5-turbo is the clear leading LLM for generation. FLAN performance scales nearly logarithmically, with a median improvement of 5.0 absolute percentage points per order-of-magnitude size increase; OpenAI models do not monotonically benefit from scale, and RLHF improves average F1 by 3.5 absolute points over text-davinci-002.
- Performance is not tied to academic discipline, but document-level tasks are hardest, with all F1 scores near or below 50%; complex expert taxonomies also perform poorly, e.g., implicit hate 32.3 F1, empathy 35.9 F1, and dialect feature detection 32.9 F1.
Methods
- The authors evaluate zero-shot performance of 13 language models on 25 representative English CSS benchmarks: 10 utterance-level classification tasks, 6 conversation-level tasks, 4 document-level tasks, and 5 generation tasks. Models include FLAN-T5 sizes from 80M to 11B, FLAN-UL2 (20B), OpenAI GPT-3 text-001/002/003, gpt-3.5-turbo, and GPT-4.
- They use prompt engineering guidelines, generate five semantically equivalent prompt perturbations per task, set temperature to zero, and constrain decoding to valid output classes or use logit bias. Classification and structured parsing are evaluated with macro F1 averaged over prompt perturbations; supervised baselines use RoBERTa-large for classification and T5-base for generation, tuned by grid search and averaged over three random seeds.
- For each task, they evaluate a class-stratified sample of at most 500 instances from the designated test set. Generation is evaluated by domain experts on 1–5 Likert scales for faithfulness, relevance, coherence, and fluency, and by ranking model outputs against gold human references; ranking annotations were performed by two authors blind, with unanimous voting.
Limitations
- The paper states that its task selection is not exhaustive; sensitive domains such as mental health and cultural studies are largely outside scope, and available data resources are dominated by standard dialects and WEIRD populations. It also notes data leakage concerns for proprietary models, that it explores predictive rather than causal explanations, that bias and fairness could not be systematically analyzed with available datasets, and that proprietary LLMs carry opacity, privacy, and intellectual-property risks.
- The paper states that generation diversity may be limited and that human evaluation is costly and hard to reproduce; it also notes that exact model sizes and training data for OpenAI models are trade secrets, so scaling analyses rely on estimates.
- Evident from the text: classification results are based on class-stratified samples capped at 500 instances per task, prompt variance is addressed with only five perturbations, and the human ranking evaluation used the authors as blind annotators rather than independent crowdworkers. Some dataset categories with fewer than 10 examples were omitted from evaluation, and automatic generation metrics such as BLEU, BERTScore, and BLEURT produced uninterpretable scores.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
0 Valid · 14 Not valid · 77 Not checked · 179 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references.”
§ Can Large Language Models Transform Computational Social Science? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other document-level accuracies are at or below 50%.”
§ Can Large Language Models Transform Computational Social Science? > 5.3 Domain-Utility (RQ3)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is confirmed by our error analysis in Figure 2 where GPT3.5 often defaults to the neutral, more colloquially recognizable label stereotype (64% of errors) rather than use a more taxonomy-specific label like white grievance.”
§ Can Large Language Models Transform Computational Social Science? > 5.1 Viability (RQ1) 5.1.1 Zero-Shot Viability.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 3 shows FLAN classification performances scaling nearly logarithmically with the parameter count.”
§ Can Large Language Models Transform Computational Social Science? > 5.2 Model-Selection (RQ2)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, documents encode complex sequences of ideas or temporal events, and overall, the two lowest task performances are on the document-level tasks: character trope classification and event argument extraction.”
§ Can Large Language Models Transform Computational Social Science? > 5.3 Domain-Utility (RQ3)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
114 more claims
Create a free account to see all 270 claims and to verify the 92 references still open.
- It may state two separate claims as one. 80
- Probably not a substantial claim. 91
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the domain of psychology, LLM-based agents can be leveraged for conducting simulation experiments, providing mental health support and so on [102-105].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research assistant: Beyond their application in specialized domains, LLM-based agents are increasingly adopted as versatile assistants in the broad field of social science research [105,114].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Political science and economy: LLM-based agents can also be utilized to study political science and economy [29,105,106].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Researchers have also recently noted GPT’s ability to help with computational social science tasks (49–54), detect misinformation (55), infer politicians’ ideologies (56), write persuasive political arguments (57), respond to patient questions (58), simulate human research participants (59–61), and model collective behavior (56).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [49]
In the citing paper
“Specifically, commentary has focused on large language models (LMs), which rely on minimal researcher input and model training for the classification of different types of data [Ziems et al., 2024, Törnberg, 2023a].”
From Prompt Stability Scoring for Text Annotation with Large Language Models (2024) · cites this paper as Ziems et al., 2024
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Can Large Language Models Transform Computational Social Science? > Causality and Explanations.
“Stance | gpt-3.5-turbo 72.0 0.58 | moderate”
§ Can Large Language Models Transform Computational Social Science? > 5.1 Viability (RQ1) 5.1.1 Zero-Shot Viability.
“These tasks also correspond to the highest absolute performances on Stance (76.0 F1)”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Can Large Language Models Transform Computational Social Science? > Causality and Explanations.
“Power Toxicity | gpt-4 gpt-4 | 59.7 0.26 55.4 0.11”
§ Can Large Language Models Transform Computational Social Science? > 5. Classification Results
“Power 49.5 72.7 46.6 48.0 40.8 55.6 52.6 56.9 43.1 39.8 37.5 36.9 39.2 51.9 56.5 42.0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Can Large Language Models Transform Computational Social Science? > Causality and Explanations.
“Ideology Event Det. gpt-4 gpt-4 | 51.5 0.51 23.0 n/a”
§ Can Large Language Models Transform Computational Social Science? > 5. Classification Results
“Event Det.0.4 75.8 9.8 7.0 1.0 10.9 41.8 50.6 29.8 47.3 47.4 44.4 48.8 52.4 51.3 14.8”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Can Large Language Models Transform Computational Social Science? > 5.3 Domain-Utility (RQ3)
“All other document-level accuracies are at or below 50%.”
§ Can Large Language Models Transform Computational Social Science? > 5. Classification Results
“Event Det.0.4 75.8 9.8 7.0 1.0 10.9 41.8 50.6 29.8 47.3 47.4 44.4 48.8 52.4 51.3 14.8”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Framing: toward clarification of a fractured paradigm
Entman, R. M. (1993) · Journal of Communication · doi:10.1111/j.1460-2466.1993.tb01304.x · Cited by 16,203 · cited 1×
“Political scientists study how political actors move agendas (Grimmer, 2010) by persuasively framing their discourse “to promote a particular problem definition, causal interpretation, moral evaluation, and/or treatment recommendation” (Entman, 1993).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Politeness: Some Universals in Language Usage
Brown, P., Levinson, S. C., & Gumperz, J. J. (1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813085 · Cited by 10,438
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“LLMs generally follow scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) where performance increases with the size of the model and training data.”
BackgroundThe spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“…analyses of social norms and conventions (Centola et al., 2018; Bicchieri, 2005), information diffusion (Leskovec et al., 2009; Tan et al., 2014; Vosoughi et al., 2018; Cheng et al., 2016), emotional contagion (Bail, 2016), collective behaviors (Barberá et al., 2015), and social movements…”
BackgroundSpeech Genres and Other Late Essays
Bakhtin, M. M. (1986) · University of Texas Press eBooks · doi:10.7560/720466 · Cited by 8,470
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Research works printed but not linked: 12
- Cialdini, 2003, Influence
- 73. Dang, Hoa Trang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1-12, Citeseer.
- 77. Ziems, Held, Shaikh, Chen, Zhang, Yang Can LLMs Transform CSS? Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 188-198, Association for Computational Linguistics, Berlin, Germany.
- 100. Garfinkel, Harold. 2016. Studies in ethnomethodology. In Social Theory Re-Wired. Routledge, pages 85-95.
- 109. Gross, Justin H, Brice Acree, Yanchuan Sim, and Noah A Smith. 2013. Testing the etch-a-sketch hypothesis: a computational analysis of mitt romney's ideological makeover during the 2012 primary vs. general elections. In APSA 2013 Annual Meeting Paper, American Political Science Association 2013 Annual Meeting.
- 152. Kuhn, Thomas S. 1962. The structure of Scientific Revolutions. The University of Chicago Press.
- 156. Ziems, Held, Shaikh, Chen, Zhang, Yang Can LLMs Transform CSS? Conference on Human Factors in Computing Systems, CHI 2015, Seoul, Republic of Korea, April 18-23, 2015, pages 3739-3748, ACM.
- 172. Luo, Yiwei, Dallas Card, and Dan Jurafsky. 2020b. Detecting stance in media on global warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296-3315, Association for Computational Linguistics, Online. · DOI
- 188. Murray, Michael et al. 2015. Narrative psychology. Qualitative psychology: A practical guide to research methods, pages 85-107.
- Omitaomu, 2022, Empathic conversations: A multi-level dataset of contextualized conversations, ArXiv preprint
- Rothbaum, 2000, Cognitive-behavioral therapy, 320
- 227. Sacks, Harvey. 1992. Lectures on conversation: Volume i. Malden, Massachusetts: Blackwell.
Cited by
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Self-citationA survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Self-citationThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · Science · doi:10.1126/science.adq1814 · Cited by 354
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2308950121 · Cited by 297
Ferrara, E. (2023) · doi:10.2139/ssrn.4614223 · Cited by 266
Ferrara, E. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00250-1 · Cited by 259
Zhou, H., Hu, C., Yuan, Y., Cui, Y., Jin, Y., Chen, C., Wu, H., Yuan, D., Jiang, L., Wu, D., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3465447 · Cited by 244
Törnberg, P. (2025) · Social Science Computer Review · doi:10.1177/08944393241286471 · Cited by 249
AI–Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators
Arora, N., Chakraborty, I., & Nishimura, Y. (2025) · Journal of Marketing · doi:10.1177/00222429241276529 · Cited by 162
Harnessing the power of AI in qualitative research: Exploring, using and redesigning ChatGPT
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100144 · Cited by 402
Explainable Generative AI (GenXAI): a survey, conceptualization, and research agenda
Schneider, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10916-x · Cited by 156
Finding love in algorithms: deciphering the emotional contexts of close encounters with AI chatbots
Li, H., & Zhang, R. (2024) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmae015 · Cited by 143
Finding Love in Algorithms: Deciphering the Emotional Contexts of Close Encounters with AI Chatbots
Li, H., & Zhang, R. (2024) · doi:10.31219/osf.io/xd4k7 · Cited by 150
Sycophantic AI decreases prosocial intentions and promotes dependence
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026) · Science · doi:10.1126/science.aec8352 · Cited by 185
“…we prompt GPT-4o with detailed instructions to assign a binary label to each prompt–response pair, following best practices for the LLM-as-a-judge paradigm (Zheng et al., 2023; Dubois et al., 2023; Gilardi et al., 2023; Ziems et al., 2024); full details of the LLM judges are in Appendix B.”
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 128
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C., & Van Bavel, J. J. (2023) · doi:10.31234/osf.io/sekf5 · Cited by 129
AI for social science and social science of AI: A survey
Xu, R. F., Sun, Y., Ren, M., Guo, S., Pan, R., Lin, H., Sun, L., & Han, X. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103665 · Cited by 226
Wang, A., Morgenstern, J., & Dickerson, J. P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00986-z · Cited by 528
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Dai, S.-C., Xiong, A., & Ku, L.-W. (2023) · arXiv · doi:10.48550/arxiv.2310.15100 · Cited by 110
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Dai, S.-C., Xiong, A., & Ku, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.669 · Cited by 273
Foundation Models: A New Paradigm for Artificial Intelligence
Schneider, J., Meske, C., & Kuss, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00851-0 · Cited by 104
Hayes, A. S. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251322346 · Cited by 184
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K., Cramer, M., Soni, S., & Bamman, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 83
Huang, C., Tu, Y., Wang, Q., Li, M., He, T., & Zhang, D. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105213 · Cited by 94
Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00345-9 · Cited by 80
Benefits and Harms of Large Language Models in Digital Mental Health
De Choudhury, M., Pendse, S. R., & Kumar, N. (2023) · doi:10.31234/osf.io/y8ax9 · Cited by 78
Benefits and Harms of Large Language Models in Digital Mental Health
De Choudhury, M., Pendse, S. R., & Kumar, N. (2023) · arXiv · doi:10.48550/arxiv.2311.14693 · Cited by 78
Thapa, S., Shiwakoti, S., Shah, S. B., Adhikari, S., Veeramani, H., Nasim, M., & Naseem, U. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01428-9 · Cited by 74
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73
“…an ever-increasing body of work illustrates the good performance of LLMs as proxies for human labeling (Wang et al., 2023; Gilardi et al., 2023; Ziems et al., 2023) and for generating high-quality text (which is, however, often factually inaccurate) that has captured the imagination of the…”
Lossio-Ventura, J. A., Weger, R., Lee, A. Y., Guinee, E. P., Chung, J., Atlas, L., Linos, E., & Pereira, F. (2024) · JMIR Mental Health · doi:10.2196/50150 · Cited by 73
Wandelt, S., Zheng, C., Wang, S., Liu, Y., & Sun, X. (2024) · Applied Sciences · doi:10.3390/app14177455 · Cited by 105
Large Language Model based Multi-Agents: A Survey of Progress and Challenges
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X. (2024) · arXiv · doi:10.48550/arxiv.2402.01680 · Cited by 1,713
Sentiment Analysis of Twitter Data Using NLP Models: A Comprehensive Review
Albladi, A., Islam, M., & Seals, C. (2025) · IEEE Access · doi:10.1109/access.2025.3541494 · Cited by 107
Stance Detection with Collaborative Role-Infused LLM-Based Agents
Lan, X., Gao, C., Jin, D., & Li, Y. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31360 · Cited by 69
Generative agent‐based modeling: an introduction and tutorial
Ghaffarzadegan, N., Majumdar, A., Williams, R., & Hosseinichimeh, N. (2024) · System Dynamics Review · doi:10.1002/sdr.1761 · Cited by 144
Ghaffarzadegan, N., Majumdar, A., Williams, R., & Hosseinichimeh, N. (2023) · arXiv · doi:10.48550/arxiv.2309.11456 · Cited by 68
Exploring the frontiers of LLMs in psychological applications: a comprehensive review
Ke, L., Tong, S., Cheng, P., & Peng, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11297-5 · Cited by 149
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63
“…we also tested whether to provide examples of pre-labeled documents within the prompt (i.e., zero-and few-shot prompts, see Chae and Davidson 2025; Rytting et al. 2023; Ziems et al. 2024), though our work highlights the challenges of these formats when analyzing long documents speci fi cally.”
The dangers of using proprietary LLMs for research
Ollion, É., Shen, R., Macanovic, A., & Chatelain, A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00783-6 · Cited by 62
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Khondaker, M. T. I., Waheed, A., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.16 · Cited by 130
ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems
Qiu, Y., & Jin, Y. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200308 · Cited by 104
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57
“Rather than solely relying on human annotators, we can assign the jobs to workers and LLMs, then seek an agreement between human annotations and LLM predictions [106].”
BackgroundSelf-citationThe Persuasive Power of Large Language Models
Breum, S. M., Egdal, D. V., Mortensen, V. G., Møller, A. G., & Aiello, L. M. (2023) · arXiv · doi:10.48550/arxiv.2312.15523 · Cited by 57
Pang, R. Y., Schroeder, H., Smith, K. S., Barocas, S., Xiao, Z., Tseng, E., & Bragg, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713726 · Cited by 146
Xu, Q., Qiu, F., Zhou, G., Zhang, C., Ding, K., Chang, F., Lu, F., Yu, Y., Ma, D., & Liu, J. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103244 · Cited by 72
Large Language Models in Qualitative Research: Uses, Tensions, and Intentions
Schroeder, H., Quéré, M. A. L., Randazzo, C., Mimno, D., & Schoenebeck, S. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713120 · Cited by 142
Large Language Models for Text Classification: From Zero-Shot Learning to Instruction-Tuning
Chae, Y., & Davidson, T. (2025) · Sociological Methods & Research · doi:10.1177/00491241251325243 · Cited by 102
Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM
Lam, M. S., Teoh, J. E. M., Landay, J. A., Heer, J., & Bernstein, M. S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642830 · Cited by 188
Automated Annotation with Generative AI Requires Validation
Pangakis, N., Wolken, S., & Fasching, N. (2023) · arXiv · doi:10.48550/arxiv.2306.00176 · Cited by 47
“Numerous other studies present similar analyses with varying results and recommendations (e.g., He et al., 2023; Wang et al., 2021; Ziems et al., 2023; Zhu et al., 2023; Ding et al., 2022).”
Li, M., Shi, T., Ziems, C., Kan, M., Chen, N. F., Liu, Z., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.92 · Cited by 161
Self-citationPeykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 84
The Persuasive Power of Large Language Models
Breum, S. M., Egdal, D. V., Gram Mortensen, V., Møller, A. G., & Aiello, L. M. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31304 · Cited by 57
Positioning Political Texts with Large Language Models by Asking and Averaging
Le Mens, G., & Gallego, A. (2025) · Political Analysis · doi:10.1017/pan.2024.29 · Cited by 42
Survey of Cultural Awareness in Language Models: Text and Beyond
Pawar, S., Park, J., Jin, J., Arora, A., Myung, J., Yadav, S., Haznitrama, F. G., Song, I., Oh, A. H., & Augenstein, I. (2025) · Computational Linguistics · doi:10.1162/coli.a.14 · Cited by 219
Prompt Stability Scoring for Text Annotation with Large Language Models
Barrie, C., Palaiologou, E., & Törnberg, P. (2024) · arXiv · doi:10.48550/arxiv.2407.02039 · Cited by 44
Jeon, J., Kim, L., & Park, J. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.102836 · Cited by 41
Large language models and political science
Linegar, M., Kocielnik, R., & Alvarez, R. M. (2023) · Frontiers in Political Science · doi:10.3389/fpos.2023.1257092 · Cited by 42
Start Generating: Harnessing Generative Artificial Intelligence for Sociological Research
Davidson, T. (2024) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231241259651 · Cited by 72
Large language models in urban planning
Fu, X., Li, C., Quan, S. J., Yiğitcanlar, T., & Wasserman, D. (2025) · Nature Cities · doi:10.1038/s44284-025-00261-7 · Cited by 55
Mačina, J., Daheim, N., Chowdhury, S. P., Sinha, T., Kapur, M., Gurevych, I., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.372 · Cited by 211
Hashmi, E., Yayilgan, S. Y., Hameed, I. A., Yamin, M. M., Ullah, M., & Abomhara, M. (2024) · IEEE Access · doi:10.1109/access.2024.3452987 · Cited by 45
Karjus, A. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04503-w · Cited by 32
MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming
Li, Y., Sun, L., & Zhang, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711032 · Cited by 195
MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming
Li, Y., Sun, L., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.06500 · Cited by 38
Marres, N., Castelle, M., Gobbo, B., Poletti, C., & Tripp, J. (2024) · Big Data & Society · doi:10.1177/20539517241255103 · Cited by 62
Zhao, X., Sun, Y., Liu, W., & Wong, C.-W. (2024) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmae022 · Cited by 52
Wang, G., Kumar, S., Huang, Z., & Liu, R. (2024) · Desalination and Water Treatment · doi:10.1016/j.dwt.2024.100829 · Cited by 40
Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Lin, H., & Zhang, Y. (2026) · Social Science Computer Review · doi:10.1177/08944393251366243 · Cited by 33
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
Sorensen, T., Jiang, L., Hwang, J. D., Levine, S., Pyatkin, V., West, P., Dziri, N., Lu, X., Rao, K., Bhagavatula, C., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.29970 · Cited by 267
Reinforcement Learning With LLMs Interaction for Distributed Diffusion Model Services
Du, H., Zhang, R., Niyato, D. T., Kang, J., Xiong, Z., Cui, S. R., Shen, X., & Kim, D. I. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3584698 · Cited by 63
McDonald, D., Papadopoulos, R., & Benningfield, L. (2024) · doi:10.36227/techrxiv.171665607.76504195/v1 · Cited by 68
Embedding Democratic Values into Social Media AIs via Societal Objective Functions
Jia, C., Lam, M. S., Mai, M. C., Hancock, J. T., & Bernstein, M. S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641002 · Cited by 31
Davidson, T., & Karell, D. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339184 · Cited by 58
From Prompt Engineering to Prompt Science with Humans in the Loop
Shah, C. S. (2025) · Communications of the ACM · doi:10.1145/3709599 · Cited by 49
Fundamental Capabilities and Applications of Large Language Models: A Survey
Li, J., Gao, Y., Yang, Y., Bai, Y., Zhou, X., Li, Y., Sun, H., Liu, Y., Si, X., Ye, Y., et al. (2025) · ACM Computing Surveys · doi:10.1145/3735632 · Cited by 82
Is GPT a Computational Model of Emotion?
Tak, A. N., & Gratch, J. S. (2023) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii59096.2023.10388119 · Cited by 59
ChatGPT for Text Annotation? Mind the Hype!
Ollion, E., Shen, R., Macanovic, A., & Chatelain, A. (2023) · doi:10.31235/osf.io/x58kn · Cited by 26
Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts
Halterman, A., & Keith, K. A. (2025) · Political Analysis · doi:10.1017/pan.2025.10017 · Cited by 119
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · doi:10.31234/osf.io/xcwdn · Cited by 28
Wiest, I. C., Wolf, F., Leßmann, M.-E., van Treeck, M., Ferber, D., Zhu, J., Boehme, H., Bressem, K. K., Ulrich, H., Ebert, M., et al. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-01103-4 · Cited by 65
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23
Majdik, Z. P., Graham, S. S., Edward, J. C. S., Rodriguez, S. N., Karnes, M. S., Jensen, J. T., Barbour, J. B., & Rousseau, J. F. (2024) · JMIR AI · doi:10.2196/52095 · Cited by 35
Kholodna, N., Julka, S., Khodadadi, M., Gumus, M. N., & Granitzer, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70381-2_25 · Cited by 70
Leveraging prompt-based LLMs for automated scoring and feedback generation in higher education
AlGhamdi, E., Li, Y., Gašević, D., & Chen, G. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105511 · Cited by 49
Siam, K., Varela, A., Faruk, M. J. H., Cheng, J. Q., Gu, H., Maruf, A. A., & Aung, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-31010-4 · Cited by 40
Computational Politeness in Natural Language Processing: A Survey
Priya, P., Firdaus, M., & Ekbal, A. (2024) · ACM Computing Surveys · doi:10.1145/3654660 · Cited by 49
Simulacrum of Stories: Examining Large Language Models as Qualitative Research Participants
Kapania, S., Agnew, W. S., Eslami, M., Heidari, H., & Fox, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713220 · Cited by 86
Analyzing and Mitigating Cultural Hallucinations of Commercial Language Models in TurkishPreprint
Boztemir, Y., & Çalışkan, N. (2024) · doi:10.36227/techrxiv.171504678.80895811/v1 · Cited by 57
Hong, J., Seto, C., Fan, A., & Maciejewski, R. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3536358 · Cited by 71
PersonaFlow: Designing LLM-Simulated Expert Perspectives for Enhanced Research Ideation
Liu, Y., Sharma, P., Oswal, M., Xia, H., & Huang, Y. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735789 · Cited by 87
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 88
Wiest, I. C., Wolf, F., Leßmann, M.-E., van Treeck, M., Ferber, D., Zhu, J., Boehme, H., Bressem, K. K., Ulrich, H., Ebert, M. P., et al. (2024) · medRxiv · doi:10.1101/2024.09.02.24312917 · Cited by 24
Machine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 88
Wang, Z., Wang, D., Xu, Y., Zhou, L., & Zhou, Y. (2025) · Journal of Chinese Political Science · doi:10.1007/s11366-025-09917-6 · Cited by 31
Metadata
- Authors
- Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D.
- Affiliations
- Stanford University (US)
- Georgia Institute of Technology (US)
- Dartmouth College (US)
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 1
- Pages
- 237-291
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4389636360
- Semantic Scholar
- 93752cae0d4ecd2d09d6660feb3c1860af973f18
- Cited by
- 585
- References
- 311
- Citations by year
- 2026: 201 · 2025: 197 · 2024: 146
- FWCI
- 271.63
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2247357, 2308994, 2039655)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- large language models, computational social science, zero-shot learning, persuasion detection, text classification, text generation, manual annotation, evaluation pipeline
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.