Can Large Language Models Transform Computational Social Science?
Open accessThe DOI matches its Crossref record; 322 of the 322 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the computational social science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers’ gold references. We conclude that the performance of today’s LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the domain of psychology, LLM-based agents can be leveraged for conducting simulation experiments, providing mental health support and so on [102-105].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research assistant: Beyond their application in specialized domains, LLM-based agents are increasingly adopted as versatile assistants in the broad field of social science research [105,114].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Political science and economy: LLM-based agents can also be utilized to study political science and economy [29,105,106].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Researchers have also recently noted GPT’s ability to help with computational social science tasks (49–54), detect misinformation (55), infer politicians’ ideologies (56), write persuasive political arguments (57), respond to patient questions (58), simulate human research participants (59–61), and model collective behavior (56).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–324 of 324
Extracting Social Power Relationships from Natural Language
Bramsen, P., Escobar-Molano, M. L., Patel, A., & Alonso, R. D. H. (2011) · Cited by 100
Cheng, J., Adamic, L. A., Kleinberg, J. M., & Leskovec, J. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2882993 · Cited by 98
The price of debiasing automatic metrics in natural language evalaution
Chaganty, A., Mussmann, S., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1060 · Cited by 96 · cited 2×
“The validity of this paradigm depends on the expected agreement between humans and prompted models (Chaganty et al., 2018).”
BackgroundSeekers, Providers, Welcomers, and Storytellers: Modeling Social Roles in Online Health Communities
Yang, D., Kraut, R. E., Smith, T., Mayfield, E., & Jurafsky, D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300574 · Cited by 94
Self-citationWord Epoch Disambiguation: Finding How Words Change Over Time
Mihalcea, R. F., & Năstase, V. (2012) · Cited by 94
Oral Language Expectations for African American Preschoolers and Kindergartners
Craig, H. K., & Washington, J. A. (2002) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2002/007 · Cited by 93
Analytical sociology and computational social science
Keuschnigg, M., Lovsjö, N., & Hedström, P. (2018) · Journal of Computational Social Science · doi:10.1007/s42001-017-0006-5 · Cited by 90
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Solaiman, I., & Dennison, C. (2021) · arXiv · doi:10.48550/arxiv.2106.10328 · Cited by 90
WHY PERSONAS WORK: THE PSYCHOLOGICAL EVIDENCE
Grudin, J. (2006) · The Persona Lifecycle · doi:10.1016/b978-012566251-2/50013-7 · Cited by 89
Taylor, S. H., DiFranzo, D. J., Choi, Y. H., Sannon, S., & Bazarova, N. N. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359220 · Cited by 88
Reframing Human-AI Collaboration for Generating Free-Text Explanations
Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M., & Choi, Y. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.47 · Cited by 87
Characterizing Online Discussion Using Coarse Discourse Sequences
Zhang, A., Culbertson, B., & Paritosh, P. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14886 · Cited by 84
Hate begets Hate: A Temporal Study of Hate Speech
Mathew, B., Illendula, A., Saha, P., Sarkar, S., Goyal, P., & Mukherjee, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415163 · Cited by 84
Latent Hatred: A Benchmark for Understanding Implicit Hate SpeechPreprint
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., Choudhury, M. D., & Yang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05322 · Cited by 83
Self-citationMulti-VALUE: A Framework for Cross-Dialectal English NLP
Ziems, C., Held, W., Yang, J., Dhamala, J., Gupta, R., & Yang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.44 · Cited by 82
Self-citationConnotation Frames of Power and Agency in Modern Films
Sap, M., Prasettio, M. C., Holtzman, A., Rashkin, H., & Choi, Y. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1247 · Cited by 80
Predicting the Topical Stance and Political Leaning of Media using Tweets
Stefanov, P., Darwish, K., Atanasov, A., & Nakov, P. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.50 · Cited by 80
Weakly Supervised Learning of Nuanced Frames for Analyzing Polarization in News Media
Roy, S., & Goldwasser, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.620 · Cited by 79
IMPLI: Investigating NLI Models’ Performance on Figurative Language
Stowe, K., Utama, P. A., & Gurevych, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.369 · Cited by 79
Modeling Protagonist Emotions for Emotion-Aware Storytelling
Brahman, F., & Chaturvedi, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.426 · Cited by 78
ConvoKit: A Toolkit for the Analysis of Conversations
Chang, J. P., Chiam, C., Fu, L., Wang, A., Zhang, J., & Danescu-Niculescu-Mizil, C. (2020) · Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.8 · Cited by 76
Empathic Conversations: A Multi-level Dataset of Contextualized ConversationsPreprint
Omitaomu, D., Tafreshi, S., Liu, T., Buechel, S., Callison-Burch, C., Eichstaedt, J. C., Ungar, L., & Sedoc, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12698 · Cited by 76
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76
Self-citationOn Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · cited 1×
“Most notably, LLMs are known to amplify social biases and stereotypes (Sheng et al., 2021; Abid et al., 2021; Borchers et al., 2022; Lucy and Bamman, 2021; Shaikh et al., 2022).”
BackgroundSelf-citationEmotions, Oral Arguments, and Supreme Court Decision Making
Black, R. C., Treul, S. A., Johnson, T. R., & Goldman, J. (2011) · The Journal of Politics · doi:10.1017/s002238161100003x · Cited by 75
SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing
Yu, T., Zhang, R., Polozov, A., Meek, C., & Awadallah, A. H. (2021) · International Conference on Learning Representations · Cited by 75
Sims, M., Park, J. H., & Bamman, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1353 · Cited by 74
SOLIDARITY OR SCHISM: IDEOLOGICAL CONGRUENCE AND THE TWITTER NETWORKS OF EGYPTIAN ACTIVISTS*
Abul-Fottouh, D., & Fetner, T. (2018) · Mobilization: An International Quarterly · doi:10.17813/1086-671x-23-1-23 · Cited by 73
Terrified: How Anti-Muslim Fringe Organizations Became Mainstream
Bail, C. A. (2014) · Princeton University Press eBooks · doi:10.23943/princeton/9780691159423.001.0001 · Cited by 70
Yang, D., Chen, J., Yang, Z., Jurafsky, D., & Hovy, E. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1364 · Cited by 68
Self-citationMind your Ps and Qs: the impact of politeness and rudeness in online communities
Burke, M., & Kraut, R. E. (2008) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1460563.1460609 · Cited by 67
ASPECTNEWS: Aspect-Oriented Summarization of News Documents
Ahuja, O., Xu, J., Gupta, A., Horecka, K. M., & Durrett, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.449 · Cited by 67
Humor and social distance in elementary school children
SHERMAN, L. W. (1988) · Humor - International Journal of Humor Research · doi:10.1515/humr.1988.1.4.389 · Cited by 66
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yao, Y., Huang, S., Wang, W., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.40 · Cited by 66
Nelson, L. K. (2021) · American Journal of Sociology · doi:10.1086/714915 · Cited by 66
Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models
Sap, M., Horvitz, E., Choi, Y., Smith, N. A., & Pennebaker, J. W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.178 · Cited by 65
The Pragmatics behind Politics: Modelling Metaphor, Framing and Emotion in Political Discourse
Cabot, .-L. H., Dankers, V., Abadi, D., Fischer, A. H., & Shutova, E. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.402 · Cited by 64
What makes a metaphor literary? Answers from two computational studies
Jacobs, A. M., & Kinder, A. (2018) · Metaphor and Symbol · doi:10.1080/10926488.2018.1434943 · Cited by 63
Vala, H., Jurgens, D., Piper, A., & Ruths, D. A. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1088 · Cited by 62
Fazeen, M., Dantu, R., & Guturu, P. (2011) · Social Network Analysis and Mining · doi:10.1007/s13278-011-0017-9 · Cited by 60
Identifying Media Frames and Frame Dynamics Within and Across Policy Issues
Boydstun, A. E., & Gross, J. H. (2013) · Cited by 60
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
Bommasani, R., Creel, K. A., Kumar, A., Jurafsky, D., & Liang, P. (2022) · arXiv · doi:10.48550/arxiv.2211.13972 · Cited by 60 · cited 1×
“Combining LLMs with true human samples is essential to avoid an algorithmic monoculture and could lead to fragile findings covering only the limited perspectives represented (Kleinberg and Raghavan, 2021; Bommasani et al., 2022).”
BackgroundCondolence and Empathy in Online Communities
Zhou, N., & Jurgens, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.45 · Cited by 59
Quantifying the narrative flow of imagined versus autobiographical stories
Sap, M., Jafarpour, A., Choi, Y., Smith, N. A., Pennebaker, J. W., & Horvitz, E. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2211715119 · Cited by 58
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · cited 2×
“In CSS, human evaluations can be unreliable (Karpinska et al., 2021).”
BackgroundCreating a Novel Geolocation Corpus from Historical Texts
DeLozier, G., Wing, B., Baldridge, J., & Nesbit, S. (2016) · Linguistic Annotation Workshop · doi:10.18653/v1/w16-1721 · Cited by 55
Learning to Recognize Dialect Features
Demszky, D., Sharma, D., Clark, J. H., Prabhakaran, V., & Eisenstein, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.184 · Cited by 55
CDLM: Cross-Document Language Modeling
Caciularu, A., Cohan, A., Beltagy, I., Peters, M., Cattan, A., & Dagan, I. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.225 · Cited by 55
Figurative Language in Recognizing Textual Entailment
Chakrabarty, T., Ghosh, D., Poliak, A., & Muresan, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.297 · Cited by 54
Training Temporal Word Embeddings with a Compass
Carlo, V. D., Bianchi, F., & Palmonari, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016326 · Cited by 53
Pre-Trained Language Models for Interactive Decision-Making
Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen, T., Huang, D.-A., Akyürek, E., Anandkumar, A., et al. (2022) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2202.01771 · Cited by 53
Picking on the Same Person: Does Algorithmic Monoculture Lead to Outcome Homogenization?
Bommasani, R., Creel, K., Kumar, A., Jurafsky, D., & Liang, P. (2022) · neural information processing systems · doi:10.52202/068431-0265 · Cited by 53
Novel Event Detection and Classification for Historical Texts
Sprugnoli, R., & Tonelli, S. (2019) · Computational Linguistics · doi:10.1162/coli_a_00347 · Cited by 52
Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires
Nguyen, T., Yates, A., Zirikly, A., Desmet, B., & Cohan, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.578 · Cited by 52
Emotional Feedback and the Viral Spread of Social Media Messages About Autism Spectrum Disorders
Bail, C. A. (2016) · American Journal of Public Health · doi:10.2105/ajph.2016.303181 · Cited by 51
Ryskina, M., Rabinovich, E., Berg-Kirkpatrick, T., Mortensen, D. R., & Tsvetkov, Y. (2020) · arXiv (Cornell University) · doi:10.7275/1jra-8m83 · Cited by 51
King, G., & Lowe, W. (2003) · International Organization · doi:10.1017/s0020818303573064 · Cited by 50
Modeling Motivational Interviewing Strategies on an Online Peer-to-Peer Counseling Platform
Shah, R. S., Holt, F., Hayati, S. A., Agarwal, A., Wang, Y.-C., Kraut, R. E., & Yang, D. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555640 · Cited by 49
Self-citationCapturing Failures of Large Language Models Via Human Cognitive Biases
Jones, E., & Steinhardt, J. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0856 · Cited by 48
Inferring Interpersonal Relations in Narrative Summaries
Srivastava, S., Chaturvedi, S., & Mitchell, T. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10349 · Cited by 46
Learning Personas from Dialogue with Attentive Memory Networks
Chu, E., Vijayaraghavan, P., & Roy, D. K. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1284 · Cited by 46
Mapping the geographical diffusion of new wordsPreprint
Eisenstein, J., O’Connor, B., Smith, N. A., & Xing, E. P. (2012) · arXiv (Cornell University) · Cited by 43
Capturing Failures of Large Language Models via Human Cognitive Biases
Jones, E., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2202.12299 · Cited by 43
Open Extraction of Fine-Grained Political Statements
Bamman, D., & Smith, N. A. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1008 · Cited by 42
Speech Acts in Political Speeches
Safwat, S. (2015) · Journal of Modern Education Review · doi:10.15341/jmer(2155-7993)/07.05.2015/008 · Cited by 42
Towards Best Experiment Design for Evaluating Dialogue System Output
Santhanam, S., & Shaikh, S. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8610 · Cited by 42
Adaptive Testing and Debugging of NLP Models
Ribeiro, M. T., & Lundberg, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.230 · Cited by 42
Altikriti, S. (2016) · International Journal of Linguistics · doi:10.5296/ijl.v8i2.9274 · Cited by 40
To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence
Ziems, C., & Yang, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.82 · Cited by 39
Self-citationInducing Positive Perspectives with Text Reframing
Ziems, C., Li, M., Zhang, A., & Yang, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.257 · Cited by 39
Self-citationBrighter than Gold: Figurative Language in User Generated Comparisons
Niculae, V., & Danescu-Niculescu-Mizil, C. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1215 · Cited by 36
A Modular Architecture for Unsupervised Sarcasm Generation
Mishra, A., Tater, T., & Sankaranarayanan, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1636 · Cited by 35
Hierarchical Chinese Legal event extraction via Pedal Attention Mechanism
Shen, S., Qi, G., Li, Z., Bi, S., & Wang, L. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.9 · Cited by 35
Humans in Humans Out: On GPT Converging Toward Common Sense in both Success and FailurePreprint
Koralus, P., & Wang-Maścianica, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.17276 · Cited by 34
Towards Modelling Language Innovation Acceptance in Online Social Networks
Kershaw, D., Rowe, M., & Stacey, P. (2016) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2835776.2835784 · Cited by 33
Zhu, J., & Jurgens, D. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.25 · Cited by 33
Semantic Variation in Online Communities of Practice
Tredici, M. D., & Fernández, R. (2018) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1806.05847 · Cited by 30
The Health Psychology of Humor
Martin, R. A., & Ford, T. E. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812143-6.00010-2 · Cited by 30
Looking for a handsome carpenter! Debiasing GPT-3 job advertisements
Borchers, C., Gala, D., Gilburt, B., Oravkin, E., Bounsi, W., Asano, Y. M., & Kirk, H. (2022) · Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.22 · Cited by 29
Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines
Gabriel, S., Hallinan, S., Sap, M., Nguyen, P., Roesner, F., Choi, E., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.222 · Cited by 28
SLIDE - a Sentiment Lexicon of Common Idioms
Jochim, C., Bonin, F., Bar-Haim, R., & Slonim, N. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3i9iizbhe2gj · Cited by 28
Detecting Latent Ideology in Expert Text: Evidence From Academic Papers in Economics
Jelveh, Z., Kogut, B., & Naidu, S. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1191 · Cited by 27
Weakly Supervised Role Identification in Teamwork Interactions
Yang, D., Wen, M., & Rosé, C. P. (2015) · doi:10.3115/v1/p15-1161 · Cited by 26
Self-citationComparing rating scales and preference judgements in language evaluation
Belz, A., & Kow, E. (2010) · University of Brighton Repository (University of Brighton) · Cited by 26
Why Do You Feel This Way? Summarizing Triggers of Emotions in Social Media Posts
Zhan, H., Sosea, T., Caragea, C., & Li, J. J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.642 · Cited by 25
Ideological Phrase Indicators for Classification of Political Discourse Framing on Twitter
Johnson, K., Lee, I.-T., & Goldwasser, D. (2017) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/w17-2913 · Cited by 24
Language Models Trained on Media Diets Can Predict Public Opinion
Chu, E., Andreas, J., Ansolabehere, S., & Roy, D. (2023) · arXiv · doi:10.48550/arxiv.2303.16779 · Cited by 24
Living Machines: A study of atypical animacy
Ardanuy, M. C., Nanni, F., Beelen, K., Hosseini, K., Ahnert, R., Lawrence, J., McDonough, K., Tolfo, G., Wilson, D. C. S., & McGillivray, B. C. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.400 · Cited by 24
Gender and Power: How Gender and Gender Environment Affect Manifestations of Power
Prabhakaran, V., Reid, E. E., & Rambow, O. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1211 · Cited by 23
Bootstrapped Text-level Named Entity Recognition for Literature
Brooke, J., Hammond, A., & Baldwin, T. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2056 · Cited by 23
Improving Hate Speech Type and Target Detection with Hateful Metaphor Features
Lemmens, J., Markov, I., & Daelemans, W. M. P. (2021) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/2021.nlp4if-1.2 · Cited by 23
Event Extraction from Historical Texts: A New Dataset for Black Rebellions
Lai, V. D., Nguyen, M., Kaufman, H., & Nguyen, T. H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.211 · Cited by 23
Egami, N., Hinck, M., Stewart, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.04746 · Cited by 23
Figurative Usage Detection of Symptom Words to Improve Personal Health Mention Detection
Iyer, A., Joshi, A., Karimi, S., Sparks, R. S., & Paris, C. L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1108 · Cited by 21
Civil Unrest on Twitter (CUT): A Dataset of Tweets to Support Research on Civil Unrest
Sech, J., DeLucia, A., Buczak, A. L., & Dredze, M. H. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.28 · Cited by 21
The structure of online social networks modulates the rate of lexical change
Zhu, J. Y., & Jurgens, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.178 · Cited by 21
Hürriyetoğlu, A., Tanev, H., Zavarella, V., Jakub Piskorski, B., Yeniterzi, R., Yüret, D., & Villavicencio, A. (2021) · Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) · doi:10.18653/v1/2021.case-1.1 · Cited by 21
(2008) · Sage eBooks · doi:10.4135/9781412963909.n71 · Cited by 21
Clean or Annotate: How to Spend a Limited Data Collection Budget
Chen, D., & Bowman, Z. Y. A. S. R. (2022) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/2022.deeplo-1.17 · Cited by 21
Proceedings of the 6th International Natural Language Generation Conference
Kelleher, J. D., Namee, B. M., & van der Sluis, I. (2010) · Cited by 20
Saldías-Fuentes, B., & Roy, D. K. (2020) · doi:10.18653/v1/2020.nuse-1.10 · Cited by 20
A Hybrid Approach for Drug Abuse Events Extraction from Twitter
Jenhani, F., Gouider, M. S., & Saïd, L. B. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.08.121 · Cited by 19
Designing interactive systems to encourage empathy between users
Coulton, P., Huck, J., Hudson‐Smith, A., Barthel, R., Mavros, P., Roberts, J., & Powell, P. A. (2014) · ACM Conference Companion Publication on Designing Interactive Systems (DIS Companion) · doi:10.1145/2598784.2602770 · Cited by 18
Sociolinguistic Approaches to Humor
Davies, C. E. (2017) · Routledge eBooks · doi:10.4324/9781315731162-33 · Cited by 18
Metaphor Detection in a Poetry Corpus
Kesarwani, V., Inkpen, D. Z., Śzpakowicz, S., & Tănăsescu, C. (2017) · Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL) · doi:10.18653/v1/w17-2201 · Cited by 17
Living Machines: A study of atypical animacy
Ardanuy, M. C., Nanni, F., Beelen, K., Hosseini, K., Ahnert, R., Lawrence, J., McDonough, K., Tolfo, G., Wilson, D. C. S., & McGillivray, B. C. (2021) · Queen Mary Research Online (Queen Mary University of London) · doi:10.17863/cam.63006 · Cited by 17
MD3: The Multi-Dialect Dataset of Dialogues
Eisenstein, J., Prabhakaran, V., Rivera, C., Demszky, D., & Sharma, D. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2150 · Cited by 17
Tweet Stance Detection Using Multi-Kernel Convolution and Attentive LSTM Variants
Siddiqua, U. A., Chy, A. N., & Aono, M. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edp7080 · Cited by 14
Calls to Action on Social Media: Detection, Social Impact, and Censorship Potential
Rogers, A., Kovaleva, O., & Rumshisky, A. (2019) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/d19-5005 · Cited by 13
Cards Against AI: Predicting Humor in a Fill-in-the-blank Party Game
Ofer, D., & Shahaf, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.394 · Cited by 13
Critical Digital Humanities: The Search for a Methodology
DOBSON, J. E. (2019) · doi:10.5406/j.ctvfjd0mf · Cited by 13
Zong, M., & Krishnamachari, B. (2022) · arXiv · doi:10.48550/arxiv.2212.00857 · Cited by 13
Lörres, Möppes, and the Swiss. (Re)Discovering regional patterns in anonymous social media data
Purschke, C., & Hovy, D. (2019) · Journal of Linguistic Geography · doi:10.1017/jlg.2019.10 · Cited by 11
Style Over Substance: Evaluation Biases for Large Language Models
Wu, M., & Aji, A. F. (2023) · arXiv · doi:10.48550/arxiv.2307.03025 · Cited by 11
Roles and Success in Wikipedia Talk Pages: Identifying Latent Patterns of Behavior
Maki, K., Yoder, M. M., Jo, Y., & Rosé, C. P. (2017) · International Joint Conference on Natural Language Processing · Cited by 11
Gómez-Zará, D., Boon, M., & Birnbaum, L. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172993 · Cited by 10
Measure Country-Level Socio-Economic Indicators with Streaming News: An Empirical Study
Min, B., & Zhao, X. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1121 · Cited by 9
Inducing Stereotypical Character Roles from Plot Structure
Jahan, L., Mittal, R., & Finlayson, M. A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.39 · Cited by 8
Examining the Ordering of Rhetorical Strategies in Persuasive Requests
Shaikh, O., Chen, J., Saad-Falcon, J., Chau, P., & Yang, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.116 · Cited by 7
Self-citationDoctor Who? Framing Through Names and Titles in German.
van den Berg, E., Korfhage, K., Ruppenhofer, J., Wiegand, M., & Markert, K. (2020) · Publication Server of the Institute for German Language (Institute for German Language) · Cited by 3
Zhu, J. Y., & Jurgens, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03158 · Cited by 1
MD3: The Multi-Dialect Dataset of DialoguesPreprint
Eisenstein, J., Prabhakaran, V., Rivera, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11355
Algorithmic monoculture and social welfare
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603202 · cited 1×
“Combining LLMs with true human samples is essential to avoid an algorithmic monoculture and could lead to fragile findings covering only the limited perspectives represented (Kleinberg and Raghavan, 2021; Bommasani et al., 2022).”
BackgroundDiffusion of Lexical Change in Social Media
Eisenstein, J., O'Connor, B., Smith, N. A., & Xing, E. P. (2012) · arXiv · doi:10.48550/arxiv.1210.5268
Research works printed but not linked: 17
- Ahmed, 2010, Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective, 1140
- Bamman, 2013, Learning latent personas of film characters, 352
- Cialdini, 2003, Influence
- Dang, 2005, Overview of DUC 2005, 1
- Egami, 2023, Using large language model annotations for valid downstream statistical inference in social science: Design-based semi-supervised learning, ArXiv preprint
- Eisenstein, 2011, Discovering sociolinguistic associations with structured sparsity, 1365
- Garfinkel, 2016, Studies in ethnomethodology, 85
- Gross, 2013, Testing the etch-a-sketch hypothesis: A computational analysis of Mitt Romney’s ideological makeover during the 2012 primary vs. general elections
- Herring, 1994, Politeness in computer culture: Why women thank and men flame, 278
- Kuhn, 1962, The Structure of Scientific Revolutions
- Luo, 2020, Detecting stance in media on global warming, 3296, 10.18653/v1/2020.findings-emnlp.296 · DOI
- Murray, 2015, Narrative psychology, Qualitative Psychology: A Practical Guide to Research Methods, 85
- Prabhakaran, 2012, Predicting overt display of power in written dialogs, 518
- Rothbaum, 2000, Cognitive-behavioral therapy, 320
- Sacks, 1992, Lectures on conversation: Volume I
- Zhuo, 2023, Exploring AI ethics of ChatGPT: A diagnostic analysis, ArXiv preprint
- Zong, 2022, A survey on GPT-3, ArXiv preprint
Cited by
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Self-citationA survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Self-citationThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Metadata
- Authors
- Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D.
- Affiliations
- Stanford University (US)
- Georgia Institute of Technology (US)
- Dartmouth College (US)
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 1
- Pages
- 237-291
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4389636360
- Semantic Scholar
- 93752cae0d4ecd2d09d6660feb3c1860af973f18
- Cited by
- 585
- References
- 311
- Citations by year
- 2026: 201 · 2025: 197 · 2024: 146
- FWCI
- 271.63
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2247357, 2308994, 2039655)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- large language models, computational social science, zero-shot learning, persuasion detection, text classification, text generation, manual annotation, evaluation pipeline
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.