Can Large Language Models Transform Computational Social Science?
Open accessThe DOI matches its Crossref record; 322 of the 322 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the computational social science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers’ gold references. We conclude that the performance of today’s LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the domain of psychology, LLM-based agents can be leveraged for conducting simulation experiments, providing mental health support and so on [102-105].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research assistant: Beyond their application in specialized domains, LLM-based agents are increasingly adopted as versatile assistants in the broad field of social science research [105,114].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Political science and economy: LLM-based agents can also be utilized to study political science and economy [29,105,106].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [105]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Researchers have also recently noted GPT’s ability to help with computational social science tasks (49–54), detect misinformation (55), infer politicians’ ideologies (56), write persuasive political arguments (57), respond to patient questions (58), simulate human research participants (59–61), and model collective behavior (56).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Framing: toward clarification of a fractured paradigm
Entman, R. M. (1993) · Journal of Communication · doi:10.1111/j.1460-2466.1993.tb01304.x · Cited by 16,203 · cited 1×
“Political scientists study how political actors move agendas (Grimmer, 2010) by persuasively framing their discourse “to promote a particular problem definition, causal interpretation, moral evaluation, and/or treatment recommendation” (Entman, 1993).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Politeness: Some Universals in Language Usage
Brown, P., Levinson, S. C., & Gumperz, J. J. (1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813085 · Cited by 10,438
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“LLMs generally follow scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) where performance increases with the size of the model and training data.”
BackgroundThe spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“…analyses of social norms and conventions (Centola et al., 2018; Bicchieri, 2005), information diffusion (Leskovec et al., 2009; Tan et al., 2014; Vosoughi et al., 2018; Cheng et al., 2016), emotional contagion (Bail, 2016), collective behaviors (Barberá et al., 2015), and social movements…”
BackgroundSpeech Genres and Other Late Essays
Bakhtin, M. M. (1986) · University of Texas Press eBooks · doi:10.7560/720466 · Cited by 8,470
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Construal-level theory of psychological distance.
Trope, Y., & Liberman, N. (2010) · Psychological Review · doi:10.1037/a0018963 · Cited by 6,618 · cited 1×
“Humor and politeness (Brown and Levinson, 1987) are correlated with subjective impressions of psychological distance between speakers (Trope and Liberman, 2010), while persuasive techniques bind agents in social commitments, with applications in the science of management and organizations.”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206 · cited 1×
“Additional application domains for the task include emotional contagion (Bail, 2016) and human factors behind economic markets (Bollen et al., 2011; Nguyen and Shirai, 2015).”
BackgroundDynamic models of segregation†
Schelling, T. C. (1971) · The Journal of Mathematical Sociology · doi:10.1080/0022250x.1971.9989794 · Cited by 4,915
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Identity and interaction: a sociocultural linguistic approach
Bucholtz, M., & Hall, K. (2005) · Discourse Studies · doi:10.1177/1461445605054407 · Cited by 3,797 · cited 1×
“Language variation is closely related to social identity (Bucholtz and Hall, 2005), from group membership (Del Tredici and Fernández, 2017), geographical region (Purschke and Hovy, 2019), and social class (Preoţiuc-Pietro et al.”
BackgroundLazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Tit for Tat? The Spiraling Effect of Incivility in the Workplace
Andersson, L. M., & Pearson, C. M. (1999) · The Academy of Management Review · doi:10.2307/259136 · Cited by 3,256
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Networks, Crowds, and Markets: Reasoning about a Highly Connected World
Easley, D., & Kleinberg, J. (2010) · doi:10.1017/cbo9780511761942 · Cited by 2,590
Shmueli, G. (2010) · Statistical Science · doi:10.1214/10-sts330 · Cited by 2,517
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 1×
“Since social systems evolve unpredictably (Salganik et al., 2006), simulated samples inherently have limited predictive and explanatory power.”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 2×
“, 2022), community and their members’ interactions (Park et al., 2023), public opinion (Argyle et al.”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · arXiv · doi:10.48550/arxiv.1804.08771 · Cited by 2,111
The Grammar of Society: The Nature and Dynamics of Social Norms
Bicchieri, C. (2005) · doi:10.1017/cbo9780511616037 · Cited by 1,991
Linguistic styles: language use as an individual difference
Pennebaker, J. W., & King, L. A. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1296 · Cited by 1,961
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
(2009) · Cited by 1,900
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Social Media and the Decision to Participate in Political Protest: Observations From Tahrir Square
Tufekci, Z., & Wilson, C. (2012) · Journal of Communication · doi:10.1111/j.1460-2466.2012.01629.x · Cited by 1,532
Meme-tracking and the dynamics of the news cycle
Leskovec, J., Backstrom, L., & Kleinberg, J. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557077 · Cited by 1,530
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 5×
“We choose FLAN-T5 (Chung et al., 2022) as an open-source model with strong zeroshot and few-shot performance.”
BackgroundMethodInfluentialContent Analysis in Communication Research
Burns, T., & Berelson, B. (1953) · The British Journal of Sociology · doi:10.2307/587175 · Cited by 1,421
A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION
PLUTCHIK, R. (1980) · Theories of Emotion · doi:10.1016/b978-0-12-558701-3.50007-7 · Cited by 1,370
Event History Modeling: A Guide for Social Scientists
Box-Steffensmeier, J. M., & Jones, B. S. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511790874 · Cited by 1,335
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
Birch, A., & Osborne, M. (2011) · The Association for Computational Linguistics · Cited by 1,200
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 3×
“, 2017) series, and ChatGPT (Qin et al., 2023; Gilardi et al., 2023) which is the conversation-based LLM trained through RLHF (Christiano et al.”
BackgroundMethodMariani, J. (2018) · HAL · Cited by 1,151
DiMaggio, P., Nag, M., & Blei, D. (2013) · Poetics · doi:10.1016/j.poetic.2013.08.004 · Cited by 1,145
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,108
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · cited 2×
“, 2018), with applications to public opinion (Bhatia, 2017; Garg et al., 2018; Kozlowski et al., 2019; Abul-Fottouh and Fetner, 2018), voting behavior (Black et al.”
BackgroundGenerating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 9×
“Furthermore, we see significant performance benefits from both RLHF models, ChatGPT and text-davinci-003.”
BackgroundMethodInfluentialLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1×
“These insights can help us understand stereotypical biases and representational harms in media (Blodgett et al., 2020).”
BackgroundBLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005 · cited 3×
“, 2019), and BLEURT (Sellam et al., 2020) that rely on comparisons to human written groundtruth all produced largely uninterpretable scores for generation tasks (see Table 7 in the Appendix).”
BackgroundWho attains social status? Effects of personality and physical attractiveness in social groups.
Anderson, C., John, O. P., Keltner, D., & Kring, A. M. (2001) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.81.1.116 · Cited by 987
Albeck, J. H. (2001) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.2001.55.2.295 · Cited by 973
Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1141 · Cited by 943
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 1×
“LLMs generally follow scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) where performance increases with the size of the model and training data.”
BackgroundTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Choice Under Uncertainty: Problems Solved and Unsolved
Machina, M. J. (1987) · Journal of Economic Perspectives · doi:10.1257/jep.1.1.121 · Cited by 878
Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2302.06476 · Cited by 869
Self-citationEvaluation methods for topic models
Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553515 · Cited by 863
SemEval-2016 Task 6: Detecting Stance in Tweets
Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., & Cherry, C. (2016) · Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · doi:10.18653/v1/s16-1003 · Cited by 844
The Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · cited 2×
“These features can be used to study the sociolinguistics of language change (Kulkarni et al., 2015; Hamilton et al., 2016b) or the linguistic biases in educational assessments (Craig and Washington, 2002) and online moderation (Sap et al., 2019).”
BackgroundPersonality and Political Attitudes: Relationships across Issue Domains and Political Contexts
GERBER, A. S., HUBER, G. A., DOHERTY, D., DOWLING, C. M., & HA, S. E. (2010) · American Political Science Review · doi:10.1017/s0003055410000031 · Cited by 813
Framing responsibility for political issues: The case of poverty
Iyengar, S. (1990) · Political Behavior · doi:10.1007/bf00992330 · Cited by 787 · cited 1×
“, 2020), as well as issue (Iyengar, 1990) and entity framing (van den Berg et al.”
BackgroundOut of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Unsupervised Learning of Narrative Event Chains
Chambers, N., & Jurafsky, D. (2008) · Cited by 769
Experimental evidence for tipping points in social convention
Centola, D., Becker, J., Brackbill, D., & Baronchelli, A. (2018) · Science · doi:10.1126/science.aas8827 · Cited by 750
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
The General Inquirer: A Computer Approach to Content Analysis.
Hartman, J. J., Stone, P. J., Dunphy, D. C., Smith, M. S., & Ogilvia, D. M. (1967) · American Sociological Review · doi:10.2307/2092070 · Cited by 723
Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., & Thue, B. (2020) · Psychological Science · doi:10.1177/0956797620916782 · Cited by 704
Vanderwende, L., Daumé, H., & Kirchhoff, K. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 685
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 2×
“The performance of LLMs as tools for classification and parsing may vary systematically as a function of demographic variation in the target population (Zhao et al., 2018).”
BackgroundMethodBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · cited 1×
“…the striking ability to reliably classify text, summarize documents, answer questions, and generate interpretable explanations in a variety of domains, even exceeding human performance without the need for supervision (Bang et al., 2023; Qin et al., 2023; ar X iv :s ub m it/ 48 40 03 8 [ cs .”
BackgroundThe online disinhibition effect
Suler, J. (2005) · International Journal of Applied Psychoanalytic Studies · doi:10.1002/aps.42 · Cited by 640
A Computational Approach to Politeness with Application to Social FactorsPreprint
Danescu-Niculescu-Mizil, C., Sudhof, M., Jurafsky, D., Leskovec, J., & Potts, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.6078 · Cited by 636
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Harvey Sacks: Social Science and Conversation Analysis
Silverman, D. J. (1998) · Cited by 555
Grimmer, J. (2010) · Political Analysis · doi:10.1093/pan/mpp034 · Cited by 550
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.85 · Cited by 504
Self-citationPersonalized Persuasion: Tailoring Persuasive Appeals to Recipients’ Personality Traits
Hirsh, J. B., Kang, S. K., & Bodenhausen, G. V. (2012) · Psychological Science · doi:10.1177/0956797611436349 · Cited by 501
The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings
Kozlowski, A. C., Taddy, M., & Evans, J. A. (2019) · American Sociological Review · doi:10.1177/0003122419877135 · Cited by 499
Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions
Cheng, J., Bernstein, M., Danescu-Niculescu-Mizil, C., & Leskovec, J. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998213 · Cited by 480
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 3×
“It is well-known that automatic performance metrics fail to capture human preferences (Goyal et al., 2022; Liang et al., 2022).”
BackgroundMethodInfluentialPersistent Anti-Muslim Bias in Large Language Models
Abid, A., Farooqi, M., & Zou, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462624 · Cited by 476 · cited 1×
“Most notably, LLMs are known to amplify social biases and stereotypes (Sheng et al., 2021; Abid et al., 2021; Borchers et al., 2022; Lucy and Bamman, 2021; Shaikh et al., 2022).”
BackgroundComputational social science: Obstacles and opportunities
Lazer, D. M. J., Pentland, A., Watts, D. J., Aral, S., Athey, S., Contractor, N., Freelon, D., Gonzalez-Bailon, S., King, G., Margetts, H., et al. (2020) · Science · doi:10.1126/science.aaz8170 · Cited by 453 · cited 2×
“Following Lazer et al. (2020), we define Computational Social Science as the development and application of computational methods to the scientific analysis of behavioral and linguistic data.”
BackgroundMethodInnateness and culture in the evolution of language
Kirby, S., Dowman, M., & Griffiths, T. L. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608222104 · Cited by 442 · cited 2×
“Semantic change detection can serve as a proxy measure for the spread and change of culture (Kirby et al., 2007), both on the internet (Eisenstein, 2012; Eisenstein et al.”
Background(2021) · International Conference on Learning Representations · Cited by 435
Prediction and explanation in social systems
Hofman, J. M., Sharma, A., & Watts, D. J. (2017) · Science · doi:10.1126/science.aal3856 · Cited by 429
Game Theory in the Social Sciences: Concepts and Solutions.
Weiss, J. H., & Shubik, M. (1984) · The Journal of Finance · doi:10.2307/2327630 · Cited by 427
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · cited 1×
“For open-ended generation tasks in particular, embedding-similarity metrics like BERTScore are insufficient (Novikova et al., 2017) and human evaluation is strongly preferable (Santhanam and Shaikh, 2019).”
BackgroundAutomatic Sarcasm Detection: A Survey
Joshi, A., Bhattacharyya, P., & Carman, M. J. (2018) · ACM Computing Surveys · doi:10.1145/3124420 · Cited by 411
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 406
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Curry, A. C., & Rieser, V. (2017) · arXiv · doi:10.48550/arxiv.1707.06875 · Cited by 406
Will, G. K., King, G., Lowe, W., Abbott, T. T. M., Bond, D., Bond, J., Bradford, G., Cobb, C., Pevehouse, J., Quinn, K., et al. (2003) · Cited by 401
Statistically Significant Detection of Linguistic Change
Kulkarni, V., Al-Rfou, R., Perozzi, B., & Skiena, S. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741627 · Cited by 390
Digital Footprints: Opportunities and Challenges for Online Social Research
Golder, S. A., & Macy, M. W. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043145 · Cited by 378
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
The Critical Periphery in the Growth of Social Protests
Barberá, P., Wang, N., Bonneau, R., Jost, J. T., Nagler, J., Tucker, J., & González-Bailón, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0143611 · Cited by 348 · cited 2×
“…social norms and conventions (Centola et al., 2018; Bicchieri, 2005), information diffusion (Leskovec et al., 2009; Tan et al., 2014; Vosoughi et al., 2018; Cheng et al., 2016), emotional contagion (Bail, 2016), collective behaviors (Barberá et al., 2015), and social movements (Nelson, 2021, 2015).”
BackgroundAn Introduction to Sociolinguistics
Holmes, J., & Wilson, N. (2022) · Routledge eBooks · doi:10.4324/9780367821852 · Cited by 344
Pre-Trained Language Models for Interactive Decision-Making
Li, S., Puig, X., Paxton, C., et al. (2022) · Neural Information Processing Systems · doi:10.52202/068431-2262 · Cited by 341
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
Machine Translation: Mining Text for Social Theory
Evans, J. A., & Aceves, P. (2016) · Annual Review of Sociology · doi:10.1146/annurev-soc-081715-074206 · Cited by 333
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.486 · Cited by 332
Antisocial Behavior in Online Discussion Communities
Cheng, J., Danescu-Niculescu-Mizil, C., & Leskovec, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v9i1.14583 · Cited by 314
CARER: Contextualized Affect Representations for Emotion Recognition
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., & Chen, Y.-S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1404 · Cited by 308
Temporal Analysis of Language through Neural Language Models
Kim, Y., Chiu, Y.-I., Hanaki, K., Hegde, D., & Petrov, S. (2014) · Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science · doi:10.3115/v1/w14-2517 · Cited by 305 · cited 2×
“, 2014) and in historical archives (Mihalcea and Nastase, 2012; Kim et al., 2014; Kulkarni et al., 2015; Rudolph and Blei, 2018)1.”
BackgroundMaxwell, K. T., Oberlander, J., & Croft, W. B. (2013) · The Association for Computational Linguistics · Cited by 294
Beyond Binary Labels: Political Ideology Prediction of Twitter Users
Preoţiuc-Pietro, D., Liu, Y., Hopkins, D., & Ungar, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1068 · Cited by 289
Gender and Representation Bias in GPT-3 Generated Stories
Lucy, L., & Bamman, D. (2021) · Proceedings of the Third Workshop on Narrative Understanding · doi:10.18653/v1/2021.nuse-1.5 · Cited by 287
Echoes of power: language effects and power differences in social interaction
Danescu-Niculescu-Mizil, C., Lee, L., Pang, B., & Kleinberg, J. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187931 · Cited by 285
Picking on the Same Person: Does Algorithmic Monoculture Lead to Outcome Homogenization?
Bommasani, R., Creel, K., Kumar, A., Jurafsky, D., & Liang, P. (2022) · neural information processing systems · doi:10.52202/068431-0265 · Cited by 284
Cold-start Active Learning through Self-supervised Language Modeling
Yuan, M., Lin, H.-T., & Boyd-Graber, J. L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.637 · Cited by 281
Exploring Cyberbullying and Other Toxic Behavior in Team Competition Online Games
Kwak, H., Blackburn, J., & Han, S. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702529 · Cited by 275
Political Ideology Detection Using Recursive Neural Networks
Iyyer, M., Enns, P., Boyd-Graber, J., & Resnik, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1105 · Cited by 275
Gender identity and lexical variation in social media
Bamman, D., Eisenstein, J., & Schnoebelen, T. (2014) · Journal of Sociolinguistics · doi:10.1111/josl.12080 · Cited by 268
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Sharma, A., Miner, A., Atkins, D., & Althoff, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.425 · Cited by 268
A Survey on Computational Propaganda Detection
Martino, G. D. S., Cresci, S., Barrón‐Cedeño, A., Yu, S., Pietro, R. D., & Nakov, P. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/672 · Cited by 263
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Predicting Factuality of Reporting and Bias of News Media Sources
Baly, R., Karadzhov, G., Alexandrov, D., Glass, J., & Nakov, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1389 · Cited by 254
We Can Detect Your Bias: Predicting the Political Ideology of News Articles
Baly, R., Martino, G. D. S., Glass, J. R., & Nakov, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.404 · Cited by 254
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
Document-Level Event Argument Extraction by Conditional Generation
Li, S., Ji, H., & Han, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.69 · Cited by 254
Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings
Demszky, D., Garg, N., Voigt, R., Zou, J., Shapiro, J., Gentzkow, M., & Jurafsky, D. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1304 · Cited by 253 · cited 2×
“We evaluate on the Indian English dialect feature detection task of Demszky et al. (2019) because this is one of the only available datasets to be handlabeled by a domain expert.”
BackgroundMethodTowards an ISO Standard for Dialogue Act Annotation
Bunt, H., Alexandersson, J., Carletta, J., Choe, J.-W., Fang, A. C., Hasida, K., Lee, K. Y., Petukhova, V. V., Popescu-Belis, A., Romary, L., et al. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5jv9phvv5iwc · Cited by 253
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247
Topic Modeling based Sentiment Analysis on Social Media for Stock Market Prediction
Nguyen, T. H., & Shirai, K. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1131 · Cited by 246
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244
Diffusion of Lexical Change in Social Media
Eisenstein, J., O'Connor, B., Smith, N. A., & Xing, E. P. (2014) · PLoS ONE · doi:10.1371/journal.pone.0113114 · Cited by 242
Finding social roles in Wikipedia
Welser, H. T., Cosley, D., Kossinets, G., Lin, A., Dokshin, F., Gay, G., & Smith, M. (2011) · Proceedings of the 2011 iConference · doi:10.1145/1940761.1940778 · Cited by 238
Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change
Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1229 · Cited by 235
Persuasion and social movements
Stewart, C. J., Smith, C. A., & Denton, R. E. (1984) · Medical Entomology and Zoology · Cited by 232
Extraction and Analysis of Fictional Character Networks: A Survey
Labatut, V., & Bost, X. (2019) · ACM Computing Surveys · doi:10.1145/3344548 · Cited by 229
Adapting computational text analysis to social science (and vice versa)
DiMaggio, P. (2015) · Big Data & Society · doi:10.1177/2053951715602908 · Cited by 228
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · cited 1×
“However, as discussed in Perez et al. (2021), LLMs can have varied performance in response to prompts which are semantically quite similar.”
BackgroundMaking computers laugh: investigations in automatic humor recognition
Mihalcea, R., & Strapparava, C. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220642 · Cited by 224
Computational Sociolinguistics: A Survey
Nguyen, D., Doğruöz, A. S., Rosé, C. P., & de Jong, F. (2016) · Computational Linguistics · doi:10.1162/coli_a_00258 · Cited by 222
An analysis of the user occupational class through Twitter content
Preoţiuc-Pietro, D., Lampos, V., & Aletras, N. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1169 · Cited by 222
A survey on empathetic dialogue systems
Ma, Y., Nguyen, K. L., Xing, F. Z., & Cambria, E. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.011 · Cited by 220
Social Simulacra: Creating Populated Prototypes for Social Computing Systems
Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., & Bernstein, M. S. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545616 · Cited by 212
Expert Finding in a Social Network
Zhang, J., Tang, J., & Li, J. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-71703-4_106 · Cited by 207
Narrative Theory for Computational Narrative Understanding
Piper, A., So, R. J., & Bamman, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.26 · Cited by 202
Reframing Human-AI Collaboration for Generating Free-Text Explanations
Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M. O., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.47 · Cited by 202
Do Anti-Immigrant Laws Shape Public Sentiment? A Study of Arizona’s SB 1070 Using Twitter Data
Flores, R. D. (2017) · American Journal of Sociology · doi:10.1086/692983 · Cited by 196
A Survey of Event Extraction From Text
Xiang, W., & Wang, B. (2019) · IEEE Access · doi:10.1109/access.2019.2956831 · Cited by 195
Dynamic Embeddings for Language Evolution
Rudolph, M., & Blei, D. M. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3185999 · Cited by 191
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., De Choudhury, M., & Yang, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.29 · Cited by 190
Self-citationTan, C., Lee, L., & Pang, B. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1017 · Cited by 188
The narrative arc: Revealing core narrative structures through text analysis
Boyd, R. L., Blackburn, K. G., & Pennebaker, J. W. (2020) · Science Advances · doi:10.1126/sciadv.aba2196 · Cited by 185 · cited 1×
“Literary studies are closely tied to the analysis of themes (Jockers and Mimno, 2013), settings (Piper et al., 2021), and narratives (Sap et al., 2022; Saldias and Roy, 2020; Boyd et al., 2020).”
BackgroundConceptions of Time and Events in Social Science Methods: Causal and Narrative Approaches
Abbott, A. (1990) · Historical Methods: A Journal of Quantitative and Interdisciplinary History · doi:10.1080/01615440.1990.10594204 · Cited by 183
Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions
Tan, C., Niculae, V., Danescu-Niculescu-Mizil, C., & Lee, L. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883081 · Cited by 180
Conversations Gone Awry: Detecting Early Signs of Conversational Failure
Zhang, J., Chang, J., Danescu-Niculescu-Mizil, C., Dixon, L., Hua, Y., Taraborelli, D., & Thain, N. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1125 · Cited by 175 · cited 1×
“Using the Conversations Gone Awry corpus (Zhang et al., 2018), we investigate whether LLMs can predict future toxicity from early cues.”
MethodNews Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175
MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection
Zarrella, G., & Marsh, A. (2016) · Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · doi:10.18653/v1/s16-1074 · Cited by 173
SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection
Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., & Tahmasebi, N. (2020) · Proceedings of the Fourteenth Workshop on Semantic Evaluation · doi:10.18653/v1/2020.semeval-1.1 · Cited by 172
Significant themes in 19th-century literature
Jockers, M. L., & Mimno, D. (2013) · Poetics · doi:10.1016/j.poetic.2013.08.005 · Cited by 168
Cotfas, L., Delcea, C., Roxin, I., Ioanăş, C., Gherai, D. S., & Tajariol, F. (2021) · IEEE Access · doi:10.1109/access.2021.3059821 · Cited by 167
Empathic communities: reaching out across the Web
Preece, J. J. (1998) · interactions · doi:10.1145/274430.274435 · Cited by 164
Text Mining: A Guidebook for the Social Sciences
Ignatow, G., & Mihalcea, R. (2017) · Sage eBooks · doi:10.4135/9781483399782 · Cited by 164
Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.12867 · Cited by 164 · cited 1×
“C L ] 1 2 A pr 2 02 3 Zhuo et al., 2023; Goyal et al., 2022).”
Cultural Performances: Proceedings of the Third Berkeley Women and Language Conference
Frankenfield, G., Bucholtz, M., Liang, A. C., Sutton, L. A., & Hines, C. (1997) · Language · doi:10.2307/417361 · Cited by 163
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450097 · Cited by 163
Harzing, A.-W., Baldueza, J., Barner-Rasmussen, W., Barzantny, C., Canabal, A., Davila, A., Espejo, A., Ferreira, R., Giroud, A., Koester, K., et al. (2009) · International Business Review · doi:10.1016/j.ibusrev.2009.03.001 · Cited by 160
FLUTE: Figurative Language Understanding through Textual Explanations
Chakrabarty, T., Saakyan, A., Ghosh, D., & Muresan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.481 · Cited by 157
Neural Metaphor Detection in Context
Gao, G., Choi, E., Choi, Y., & Zettlemoyer, L. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1060 · Cited by 152
Humor Styles and Symbolic Boundaries
Kuipers, G. (2009) · Journal of Literary Theory · doi:10.1515/jlt.2009.013 · Cited by 151
The emotional side of software developers in JIRA
Ortu, M., Murgia, A., Destefanis, G., Tourani, P., Tonelli, R., Marchesi, M., & Adams, B. (2016) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2901739.2903505 · Cited by 151
Kim, T., Lee, D.-R., & Wong, N. Y. S. (2015) · Journal of Business and Psychology · doi:10.1007/s10869-015-9406-9 · Cited by 150
Humor Detection: A Transformer Gets the Last Laugh
Weller, O., & Seppi, K. D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1372 · Cited by 148
Algorithmic monoculture and social welfare
Kleinberg, J., & Raghavan, M. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2018340118 · Cited by 147
Patterns of empathy in online communication
Pfeil, U., & Zaphiris, P. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240763 · Cited by 145
How to Ask for a Favor: A Case Study on the Success of Altruistic Requests
Althoff, T., Danescu-Niculescu-Mizil, C., & Jurafsky, D. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14547 · Cited by 144
Hate begets Hate: A Temporal Study of Hate Speech
Mathew, B., Illendula, A., Saha, P., Sarkar, S., Goyal, P., & Mukherjee, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415163 · Cited by 144
Feuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships
Iyyer, M., Guha, A., Chaturvedi, S., Boyd-Graber, J. L., & Daumé, H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1180 · Cited by 143
Multilingual stance detection in social media political debates
Lai, M., Cignarella, A. T., Farías, D. I. H., Bosco, C., Patti, V., & Rosso, P. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101075 · Cited by 140
Cheng, J., Adamic, L. A., Kleinberg, J. M., & Leskovec, J. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2882993 · Cited by 136
Making sense of discourse analysis
Paltridge, B. (2000) · Cited by 134
Pre-Trained Language Models for Interactive Decision-MakingPreprint
Li, S., Puig, X., Paxton, C., Du, Y., Wang, C. J., Fan, L., Chen, T., Huang, D.-A., Akyürek, E., Anandkumar, A., et al. (2022) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2202.01771 · Cited by 132
Predicting the Topical Stance and Political Leaning of Media using Tweets
Stefanov, P., Darwish, K. M., Atanasov, A., & Nakov, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.50 · Cited by 131
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131 · cited 1×
“Most notably, LLMs are known to amplify social biases and stereotypes (Sheng et al., 2021; Abid et al., 2021; Borchers et al., 2022; Lucy and Bamman, 2021; Shaikh et al., 2022).”
BackgroundDetecting Stance in Media On Global Warming
Luo, Y., Card, D., & Jurafsky, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.296 · Cited by 127
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127 · cited 1×
“As zero-shot models approach or outperform the quality of the gold-reference generations, reference-based scoring becomes an invalid construct for measuring models’ true utility (Raji et al., 2021), even if we assume the semantic similarity metrics are ideal.”
BackgroundHumor Recognition Using Deep Learning
Chen, P., & Soo, V. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2018 · Cited by 123
Effects of Humor on Persuasion
Markiewicz, D. (1974) · Sociometry · doi:10.2307/2786391 · Cited by 121
Field, A., Kliger, D., Wintner, S., Pan, J., Jurafsky, D., & Tsvetkov, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1393 · Cited by 120
Characterizing Online Discussion Using Coarse Discourse Sequences
Zhang, A. X., Culbertson, B., & Paritosh, P. K. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14886 · Cited by 119
Associative judgment and vector space semantics.
Bhatia, S. (2016) · Psychological Review · doi:10.1037/rev0000047 · Cited by 117
Research works printed but not linked: 17
- Ahmed, 2010, Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective, 1140
- Bamman, 2013, Learning latent personas of film characters, 352
- Cialdini, 2003, Influence
- Dang, 2005, Overview of DUC 2005, 1
- Egami, 2023, Using large language model annotations for valid downstream statistical inference in social science: Design-based semi-supervised learning, ArXiv preprint
- Eisenstein, 2011, Discovering sociolinguistic associations with structured sparsity, 1365
- Garfinkel, 2016, Studies in ethnomethodology, 85
- Gross, 2013, Testing the etch-a-sketch hypothesis: A computational analysis of Mitt Romney’s ideological makeover during the 2012 primary vs. general elections
- Herring, 1994, Politeness in computer culture: Why women thank and men flame, 278
- Kuhn, 1962, The Structure of Scientific Revolutions
- Luo, 2020, Detecting stance in media on global warming, 3296, 10.18653/v1/2020.findings-emnlp.296 · DOI
- Murray, 2015, Narrative psychology, Qualitative Psychology: A Practical Guide to Research Methods, 85
- Prabhakaran, 2012, Predicting overt display of power in written dialogs, 518
- Rothbaum, 2000, Cognitive-behavioral therapy, 320
- Sacks, 1992, Lectures on conversation: Volume I
- Zhuo, 2023, Exploring AI ethics of ChatGPT: A diagnostic analysis, ArXiv preprint
- Zong, 2022, A survey on GPT-3, ArXiv preprint
Cited by
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Self-citationA survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Self-citationThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Metadata
- Authors
- Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D.
- Affiliations
- Stanford University (US)
- Georgia Institute of Technology (US)
- Dartmouth College (US)
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 1
- Pages
- 237-291
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4389636360
- Semantic Scholar
- 93752cae0d4ecd2d09d6660feb3c1860af973f18
- Cited by
- 585
- References
- 311
- Citations by year
- 2026: 201 · 2025: 197 · 2024: 146
- FWCI
- 271.63
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2247357, 2308994, 2039655)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- large language models, computational social science, zero-shot learning, persuasion detection, text classification, text generation, manual annotation, evaluation pipeline
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.