CoChat
Sign Up
PreprintCited by 16

Can Large Language Models Transform Computational Social Science?

Ziems, Caleb, Held, William, Shaikh, Omar, Chen, Jiaao, Zhang, Zhehao, Yang, Diyi·2023·arXiv
DOI 10.48550/arxiv.2305.03514arXiv 2305.03514
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the Computational Social Science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that the performance of today's LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.03514

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In CSS generation tasks, LLMs often produce explanations that surpass the quality of gold references provided by crowd workers.

    “When it comes to generation tasks, LLMs often produce explanations that surpass the quality of gold references provided by crowd workers.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [269]

  • LLMs achieve the best performance on misinformation, stance, and emotion classification.

    “Conversely, LLMs achieve the best performance on misinformation, stance, and emotion classification.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [269]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 279

  1. Abbott, Andrew. 1990. Conceptions of time and events in social science methods: Causal and narrative approaches. Historical Methods: A Journal of Quantitative and Interdisciplinary History, 23(4):140-150.
  2. Abid, Abubakar, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298-306.
  3. Abul-Fottouh, Deena and Tina Fetner. 2018. Solidarity or schism: ideological congruence and the twitter networks of egyptian activists. Mobilization: An International Quarterly, 23(1):23-44.
  4. Ahmed, Amr and Eric Xing. 2010. Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1140-1150, Association for Computational Linguistics, Cambridge, MA.
  5. Ahuja, Ojas, Jiacheng Xu, Akshay Gupta, Kevin Horecka, and Greg Durrett. 2022. Aspectnews: Aspect-oriented summarization of news documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6494-6506.
  6. Althoff, Tim, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2014. How to ask for a favor: A case study on the success of altruistic requests. In Proceedings of the International AAAI Conference on Web and Social Media, volume 8, pages 12-21.
  7. Altikriti, Sahar. 2016. Persuasive speech acts in barack obama's inaugural speeches (2009, 2013) and the last state of the union address (2016). International Journal of Linguistics, 8(2):47-66.
  8. Amber, Boydstun, Gross Justin, Philip Resnik, and Noah Smith. 2013. Identifying media frames and frame dynamics within and across policy issues. In New directions in analyzing text as Data workshop.
  9. Anderson, Cameron, Oliver P John, Dacher Keltner, and Ann M Kring. 2001. Who attains social status? effects of personality and physical attractiveness in social groups. Journal of personality and social psychology, 81(1):116.
  10. Andersson, Lynne M and Christine M Pearson. 1999. Tit for tat? the spiraling effect of incivility in the workplace. Academy of management review, 24(3):452-471.
  11. Argyle, Lisa P, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2022. Out of one, many: Using language models to simulate human samples. Political Analysis, pages 1-15.
  12. Bail, Chris. 2014. Terrified: How anti-Muslim fringe organizations became mainstream. Princeton University Press.
  13. Bail, Christopher A. 2016. Emotional feedback and the viral spread of social media messages about autism spectrum disorders. American journal of public health, 106(7):1173-1180.
  14. Bakhtin, Mikhail Mikhaȋlovich. 2010. Speech genres and other late essays. University of Texas press.
  15. Baly, Ramy, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, Association for Computational Linguistics, Online.
  16. Baly, Ramy, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3528-3539, Association for Computational Linguistics, Brussels, Belgium.
  17. Bamman, David, Jacob Eisenstein, and Tyler Schnoebelen. 2014. Gender identity and lexical variation in social media. Journal of Sociolinguistics, 18(2):135-160.
  18. Bamman, David, Brendan O'Connor, and Noah A. Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361, Association for Computational Linguistics, Sofia, Bulgaria.
  19. Bamman, David and Noah A. Smith. 2015. Open extraction of fine-grained political statements. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 76-85, Association for Computational Linguistics, Lisbon, Portugal.
  20. Bang, Yejin, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ArXiv preprint, abs/2302.04023. · arXiv
  21. Barberá, Pablo, Ning Wang, Richard Bonneau, John T Jost, Jonathan Nagler, Joshua Tucker, and Sandra González-Bailón. 2015. The critical periphery in the growth of social protests. PloS one, 10(11):e0143611.
  22. Beltagy, Iz, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv preprint, abs/2004.05150. · arXiv
  23. Belz, Anja and Eric Kow. 2010. Comparing rating scales and preference judgements in language evaluation. In Proceedings of the 6th International Natural Language Generation Conference.
  24. Bender, Emily M, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610-623.
  25. van den Berg, Esther, Katharina Korfhage, Josef Ruppenhofer, Michael Wiegand, and Katja Markert. 2020. Doctor who? framing through names and titles in German. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4924-4932, European Language Resources Association, Marseille, France.
  26. Bhatia, Sudeep. 2017. Associative judgment and vector space semantics. Psychological review, 124(1):1.
  27. Bicchieri, Cristina. 2005. The grammar of society: The nature and dynamics of social norms. Cambridge University Press.
  28. Black, Ryan C, Sarah A Treul, Timothy R Johnson, and Jerry Goldman. 2011. Emotions, oral arguments, and supreme court decision making. The Journal of Politics, 73(2):572-581.
  29. Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Association for Computational Linguistics, Online.
  30. Bollen, Johan, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of computational science, 2(1):1-8.
  31. Bommasani, Rishi, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. 2022. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35:3663-3678.
  32. Bommasani, Rishi, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. ArXiv preprint, abs/2108.07258. · arXiv
  33. Borchers, Conrad, Dalia Gala, Benjamin Gilburt, Eduard Oravkin, Wilfried Bounsi, Yuki M Asano, and Hannah Kirk. 2022. Looking for a handsome carpenter! debiasing GPT-3 job advertisements. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 212-224, Association for Computational Linguistics, Seattle, Washington.
  34. Box-Steffensmeier, Janet M and Bradford S Jones. 2004. Event history modeling: A guide for social scientists. Cambridge University Press.
  35. Brahman, Faeze and Snigdha Chaturvedi. 2020. Modeling protagonist emotions for emotion-aware storytelling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5277-5294, Association for Computational Linguistics, Online.
  36. Bramsen, Philip, Martha Escobar-Molano, Ami Patel, and Rafael Alonso. 2011. Extracting social power relationships from natural language. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 773-782, Association for Computational Linguistics, Portland, Oregon, USA.
  37. Brooke, Julian, Adam Hammond, and Timothy Baldwin. 2016. Bootstrapped text-level named entity recognition for literature. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 344-350, Association for Computational Linguistics, Berlin, Germany.
  38. Brown, Penelope and Stephen C Levinson. 1987. Politeness: Some universals in language usage, volume 4. Cambridge university press.
  39. Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  40. Bucholtz, Mary and Kira Hall. 2005. Identity and interaction: A sociocultural linguistic approach. Discourse studies, 7(4-5):585-614.
  41. Buechel, Sven, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. 2018. Modeling empathy and distress in reaction to news stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4758-4765, Association for Computational Linguistics, Brussels, Belgium.
  42. Bunt, Harry, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David Traum. 2010. Towards an ISO standard for dialogue act annotation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), European Language Resources Association (ELRA), Valletta, Malta.
  43. Burke, Moira and Robert Kraut. 2008. Mind your ps and qs: the impact of politeness and rudeness in online communities. In Proceedings of the 2008 ACM conference on Computer supported cooperative work, pages 281-284.
  44. Caciularu, Avi, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. 2021. Cdlm: Cross-document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648-2662.
  45. Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
  46. Carlo, Valerio Di, Federico Bianchi, and Matteo Palmonari. 2019. Training temporal word embeddings with a compass. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 -February 1, 2019, pages 6326-6334, AAAI Press.
  47. Centola, Damon, Joshua Becker, Devon Brackbill, and Andrea Baronchelli. 2018. Experimental evidence for tipping points in social convention. Science, 360(6393):1116-1119.
  48. Chaganty, Arun, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Association for Computational Linguistics, Melbourne, Australia.
  49. Chakrabarty, Tuhin, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. 2021. Figurative language in recognizing textual entailment. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3354-3361, Association for Computational Linguistics, Online.
  50. Chakrabarty, Tuhin, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. Flute: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139-7159.
  51. Chambers, Nathanael and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789-797, Association for Computational Linguistics, Columbus, Ohio.
  52. Chang, Jonathan, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada, pages 288-296, Curran Associates, Inc.
  53. Chang, Jonathan P., Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 57-60, Association for Computational Linguistics, 1st virtual meeting.
  54. Chen, Derek, Zhou Yu, and Samuel Bowman. 2022. Clean or annotate: How to spend a limited data collection budget. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 152-168.
  55. Chen, Peng-Yu and Von-Wun Soo. 2018. Humor recognition using deep learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 113-117, Association for Computational Linguistics, New Orleans, Louisiana.
  56. Cheng, Justin, Lada A. Adamic, Jon M. Kleinberg, and Jure Leskovec. 2016. Do cascades recur? In Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 -15, 2016, pages 671-681, ACM.
  57. Cheng, Justin, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pages 1217-1230.
  58. Cheng, Justin, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2015. Antisocial behavior in online discussion communities. In Proceedings of the international aaai conference on web and social media, volume 9, pages 61-70.
  59. Child, Rewon, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. ArXiv preprint, abs/1904.10509. · arXiv
  60. Christiano, Paul F., Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299-4307.
  61. Chu, Eric, Jacob Andreas, Stephen Ansolabehere, and Deb Roy. 2023. Language models trained on media diets can predict public opinion. ArXiv preprint, abs/2303.16779. · arXiv
  62. Chu, Eric, Prashanth Vijayaraghavan, and Deb Roy. 2018. Learning personas from dialogue with attentive memory networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2638-2646, Association for Computational Linguistics, Brussels, Belgium.
  63. Chung, Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. ArXiv preprint, abs/2210.11416. · arXiv
  64. Cialdini, Robert B. 2003. Influence. Influence At Work. Coll Ardanuy, Mariona, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Katherine McDonough, Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. 2020. Living machines: A study of atypical animacy. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4534-4545, International Committee on Computational Linguistics, Barcelona, Spain (Online).
  65. Coulton, Paul, Jonny Huck, Andrew Hudson-Smith, Ralph Barthel, Panagiotis Mavros, Jennifer Roberts, and Philip Powell. 2014. Designing interactive systems to encourage empathy between users. In Proceedings of the 2014 companion publication on Designing interactive systems. pages 13-16.
  66. Danescu-Niculescu-Mizil, Cristian, Lillian Lee, Bo Pang, and Jon M. Kleinberg. 2012. Echoes of power: language effects and power differences in social interaction. In Proceedings of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012, pages 699-708, ACM.
  67. Danescu-Niculescu-Mizil, Cristian, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 250-259, Association for Computational Linguistics, Sofia, Bulgaria.
  68. Dang, Hoa Trang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1-12, Citeseer.
  69. Davies, Catherine E. 2017. Sociolinguistic approaches to humor. In The Routledge handbook of language and humor. Routledge, pages 472-488.
  70. Del Tredici, Marco and Raquel Fernández. 2017. Semantic variation in online communities of practice. In IWCS 2017 - 12th International Conference on Computational Semantics -Long papers.
  71. DeLozier, Grant, Ben Wing, Jason Baldridge, and Scott Nesbit. 2016. Creating a novel geolocation corpus from historical texts. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 188-198, Association for Computational Linguistics, Berlin, Germany.
  72. Demszky, Dorottya, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Jurafsky. 2019. Analyzing polarization in social media: Method and application to tweets on 21 mass shootings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2970-3005, Association for Computational Linguistics, Minneapolis, Minnesota.
  73. Demszky, Dorottya, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315-2338, Association for Computational Linguistics, Online.
  74. Dhamala, Jwala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872.
  75. DiMaggio, Paul. 2015. Adapting computational text analysis to social science (and vice versa). Big Data & Society, 2(2):2053951715602908.
  76. DiMaggio, Paul, Manish Nag, and David Blei. 2013. Exploiting affinities between topic modeling and the sociological perspective on culture: Application to newspaper coverage of us government arts funding. Poetics, 41(6):570-606.
  77. Dobson, James E. 2019. Critical digital humanities: the search for a methodology. University of Illinois Press.
  78. Easley, David and Jon Kleinberg. 2010. Networks, crowds, and markets: Reasoning about a highly connected world. Cambridge university press.
  79. Egami, Naoki, Musashi Jacobs-Harukawa, Brandon M Stewart, and Hanying Wei. 2023. Using large language model annotations for valid downstream statistical inference in social science: Design-based semi-supervised learning. ArXiv preprint, abs/2306.04746. · arXiv
  80. Eisenstein, Jacob. 2012. Mapping the geographical diffusion of new words. ArXiv preprint, abs/1210.5268. · arXiv
  81. Eisenstein, Jacob, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. PloS one, 9(11):e113114.
  82. Eisenstein, Jacob, Vinodkumar Prabhakaran, Clara Rivera, Dorottya Demszky, and Devyani Sharma. 2023. Md3: The multi-dialect dataset of dialogues. ArXiv preprint, abs/2305.11355. · arXiv
  83. Eisenstein, Jacob, Noah A. Smith, and Eric P. Xing. 2011. Discovering sociolinguistic associations with structured sparsity. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1365-1374, Association for Computational Linguistics, Portland, Oregon, USA.
  84. Ellsberg, Daniel. 1961. Risk, ambiguity, and the savage axioms. The quarterly journal of economics, 75(4):643-669.
  85. ElSherief, Mai, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. ArXiv preprint, abs/2109.05322. · arXiv
  86. Entman, Robert M. 1993. Framing: Toward clarification of a fractured paradigm. Journal of communication, 43(4):51-58.
  87. Evans, James A and Pedro Aceves. 2016. Machine translation: Mining text for social theory. Annual review of sociology, 42:21-50.
  88. Fabbri, Alexander R, Wojciech Kryści ński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
  89. Fazeen, Mohamed, Ram Dantu, and Parthasarathy Guturu. 2011. Identification of leaders, lurkers, associates and spammers in a social network: context-dependent and context-independent approaches. Social Network Analysis and Mining, 1(3):241-254.
  90. Field, Anjalie, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in Russian news: a computational analysis of intricate political strategies. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3570-3580, Association for Computational Linguistics, Brussels, Belgium.
  91. Flores, René D. 2017. Do anti-immigrant laws shape public sentiment? a study of arizona's sb 1070 using twitter data. American Journal of Sociology, 123(2):333-384.
  92. Gabriel, Saadia, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. Misinfo reaction frames: Reasoning about readers' reactions to news headlines. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3108-3127.
  93. Gao, Ge, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 607-613, Association for Computational Linguistics, Brussels, Belgium.
  94. Garfinkel, Harold. 2016. Studies in ethnomethodology. In Social Theory Re-Wired. Routledge, pages 85-95.
  95. Garg, Nikhil, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635-E3644.
  96. Gerber, Alan S, Gregory A Huber, David Doherty, Conor M Dowling, and Shang E Ha. 2010. Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review, 104(1):111-133.
  97. Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. ArXiv preprint, abs/2303.15056. · arXiv
  98. Golder, Scott A and Michael W Macy. 2014. Digital footprints: Opportunities and challenges for online social research. Annual Review of Sociology, 40:129-152.
  99. Gomez-Zara, Diego, Miriam Boon, and Larry Birnbaum. 2018. Who is the hero, the villain, and the victim? detection of roles in news articles using natural language techniques. In 23rd International Conference on Intelligent User Interfaces, pages 311-315.
  100. Goyal, Tanya, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. ArXiv preprint, abs/2209.12356. · arXiv
  101. Graham, Elizabeth E. 1995. The involvement of sense of humor in the development of social relationships. Communication Reports, 8(2):158-169.
  102. Grimmer, Justin. 2010. A bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases. Political Analysis, 18(1):1-35.
  103. Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016a. Cultural shift or linguistic drift? comparing two computational measures of semantic change. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116-2121, Association for Computational Linguistics, Austin, Texas.
  104. Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Association for Computational Linguistics, Berlin, Germany.
  105. Harzing, Anne-Wil, Joyce Baldueza, Wilhelm Barner-Rasmussen, Cordula Barzantny, Anne Canabal, Anabella Davila, Alvaro Espejo, Rita Ferreira, Axele Giroud, Kathrin Koester, et al. 2009. Rating versus ranking: What is the best way to reduce response and language bias in cross-national research? International business review, 18(4):417-432.
  106. Hashim, Safair Safwat Mohammed and Suhair Safwat. 2015. Speech acts in political speeches. Journal of Modern Education Review, 5(7):699-706.
  107. Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  108. Herring, Susan C. 1994. Politeness in computer culture: Why women thank and men flame. In Cultural performances: Proceedings of the third Berkeley women and language conference, pages 278-294.
  109. Hirsh, Jacob B, Sonia K Kang, and Galen V Bodenhausen. 2012. Personalized persuasion: Tailoring persuasive appeals to recipients' personality traits. Psychological science, 23(6):578-581.
  110. Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. ArXiv preprint, abs/2203.15556. · arXiv
  111. Hofman, Jake M, Amit Sharma, and Duncan J Watts. 2017. Prediction and explanation in social systems. Science, 355(6324):486-488.
  112. Huguet Cabot, Pere-Lluís, Verna Dankers, David Abadi, Agneta Fischer, and Ekaterina Shutova. 2020. The Pragmatics behind Politics: Modelling Metaphor, Framing and Emotion in Political Discourse. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4479-4488, Association for Computational Linguistics, Online.
  113. Hürriyeto glu, Ali, Hristo Tanev, Vanni Zavarella, Jakub Piskorski, Reyyan Yeniterzi, Deniz Yuret, and Aline Villavicencio. 2021. Challenges and applications of automated extraction of socio-political events from text (CASE 2021): Workshop and shared task report. In Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), pages 1-9, Association for Computational Linguistics, Online.
  114. Ignatow, Gabe and Rada Mihalcea. 2016. Text mining: A guidebook for the social sciences. Sage Publications.
  115. Iyengar, Shanto. 1990. Framing responsibility for political issues: The case of poverty. Political behavior, 12(1):19-40.
  116. Iyer, Adith, Aditya Joshi, Sarvnaz Karimi, Ross Sparks, and Cecile Paris. 2019. Figurative usage detection of symptom words to improve personal health mention detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1142-1147, Association for Computational Linguistics, Florence, Italy.
  117. Iyyer, Mohit, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1113-1122, Association for Computational Linguistics, Baltimore, Maryland.
  118. Iyyer, Mohit, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former Friends: Unsupervised learning for dynamic fictional relationships. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1534-1544, Association for Computational Linguistics, San Diego, California.
  119. Jacobs, Arthur M and Annette Kinder. 2018. What makes a metaphor literary? answers from two computational studies. Metaphor and Symbol, 33(2):85-100.
  120. Jahan, Labiba, Rahul Mittal, and Mark Finlayson. 2021. Inducing stereotypical character roles from plot structure. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 492-497.
  121. Jelveh, Zubin, Bruce Kogut, and Suresh Naidu. 2014. Detecting latent ideology in expert text: Evidence from academic papers in economics. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1804-1809, Association for Computational Linguistics, Doha, Qatar.
  122. Jochim, Charles, Francesca Bonin, Roy Bar-Haim, and Noam Slonim. 2018. SLIDE -a sentiment lexicon of common idioms. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan.
  123. Jockers, Matthew L and David Mimno. 2013. Significant themes in 19th-century literature. Poetics, 41(6):750-769.
  124. Johnson, Kristen, I-Ta Lee, and Dan Goldwasser. 2017. Ideological phrase indicators for classification of political discourse framing on Twitter. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 90-99, Association for Computational Linguistics, Vancouver, Canada.
  125. Jones, Erik and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. ArXiv preprint, abs/2202.12299. · arXiv
  126. Joshi, Aditya, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):1-22.
  127. Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv preprint, abs/2001.08361. · arXiv
  128. Karpinska, Marzena, Nader Akoury, and Mohit Iyyer. 2021. The perils of using Mechanical Turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic.
  129. Kershaw, Daniel, Matthew Rowe, and Patrick Stacey. 2016. Towards modelling language innovation acceptance in online social networks. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, San Francisco, CA, USA, February 22-25, 2016, pages 553-562, ACM.
  130. Kesarwani, Vaibhav, Diana Inkpen, Stan Szpakowicz, and Chris Tanasescu. 2017. Metaphor detection in a poetry corpus. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 1-9, Association for Computational Linguistics, Vancouver, Canada.
  131. Keuschnigg, Marc, Niclas Lovsjö, and Peter Hedström. 2018. Analytical sociology and computational social science. Journal of Computational Social Science, 1(1):3-14.
  132. Kiela, Douwe, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110-4124, Association for Computational Linguistics, Online.
  133. Kim, Tae-Yeol, Deog-Ro Lee, and Noel Yuen Shan Wong. 2016. Supervisor humor and employee outcomes: The role of social distance and affective trust in supervisor. Journal of Business and Psychology, 31:125-139.
  134. Kim, Yoon, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61-65, Association for Computational Linguistics, Baltimore, MD, USA.
  135. King, Gary and Will Lowe. 2003. An automated information extraction tool for international conflict data with performance as good as human coders: A rare events evaluation design. International Organization, 57(3):617-642.
  136. Kirby, Simon, Mike Dowman, and Thomas L Griffiths. 2007. Innateness and culture in the evolution of language. Proceedings of the National Academy of Sciences, 104(12):5241-5245.
  137. Kleinberg, Jon, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237-293.
  138. Kleinberg, Jon and Manish Raghavan. 2021. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22):e2018340118.
  139. Koralus, Philipp and Vincent Wang-Maścianica. 2023. Humans in humans out: On gpt converging toward common sense in both success and failure. ArXiv preprint, abs/2303.17276. · arXiv
  140. Kosinski, Michal, David Stillwell, and Thore Graepel. 2013. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the national academy of sciences, 110(15):5802-5805.
  141. Kozlowski, Austin C, Matt Taddy, and James A Evans. 2019. The geometry of culture: Analyzing the meanings of class through word embeddings. American Sociological Review, 84(5):905-949.
  142. Kuhn, Thomas S. 1962. The structure of Scientific Revolutions. The University of Chicago Press.
  143. Kulkarni, Vivek, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18-22, 2015, pages 625-635, ACM.
  144. Kwak, Haewoon, Jeremy Blackburn, and Seungyeop Han. 2015. Exploring cyberbullying and other toxic behavior in team competition online games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, Seoul, Republic of Korea, April 18-23, 2015, pages 3739-3748, ACM.
  145. Labatut, Vincent and Xavier Bost. 2019. Extraction and analysis of fictional character networks: A survey. ACM Computing Surveys (CSUR), 52(5):1-40.
  146. Lai, Viet, Minh Van Nguyen, Heidi Kaufman, and Thien Huu Nguyen. 2021. Event extraction from historical texts: A new dataset for black rebellions. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2390-2400, Association for Computational Linguistics, Online.
  147. Lazer, David, Alex Pentland, Lada Adamic, Sinan Aral, Albert-László Barabási, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, et al. 2009. Computational social science. Science, 323(5915):721-723.
  148. Lazer, David MJ, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al. 2018. The science of fake news. Science, 359(6380):1094-1096.
  149. Lazer, David MJ, Alex Pentland, Duncan J Watts, Sinan Aral, Susan Athey, Noshir Contractor, Deen Freelon, Sandra Gonzalez-Bailon, Gary King, Helen Margetts, et al. 2020. Computational social science: Obstacles and opportunities. Science, 369(6507):1060-1062.
  150. Lemmens, Jens, Ilia Markov, and Walter Daelemans. 2021. Improving hate speech type and target detection with hateful metaphor features. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 7-16, Association for Computational Linguistics, Online.
  151. Leskovec, Jure, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 497-506.
  152. Li, Sha, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 894-908, Association for Computational Linguistics, Online.
  153. Li, Shuang, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. 2022. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199-31212.
  154. Liang, Percy, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. ArXiv preprint, abs/2211.09110. · arXiv
  155. Liu, Chia-Wei, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, Association for Computational Linguistics, Austin, Texas.
  156. Lucy, Li and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48-55, Association for Computational Linguistics, Virtual.
  157. Luo, Yiwei, Dallas Card, and Dan Jurafsky. 2020a. Desmog: Detecting stance in media on global warming. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3296-3315.
  158. Luo, Yiwei, Dallas Card, and Dan Jurafsky. 2020b. Detecting stance in media on global warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296-3315, Association for Computational Linguistics, Online.
  159. Machina, Mark J. 1987. Choice under uncertainty: Problems solved and unsolved. Journal of Economic perspectives, 1(1):121-154.
  160. Maki, Keith, Michael Yoder, Yohan Jo, and Carolyn Rosé. 2017. Roles and success in Wikipedia talk pages: Identifying latent patterns of behavior. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1026-1035, Asian Federation of Natural Language Processing, Taipei, Taiwan.
  161. Markiewicz, Dorothy. 1974. Effects of humor on persuasion. Sociometry, pages 407-422.
  162. Martin, Rod A and Thomas Ford. 2018. The psychology of humor: An integrative approach. Academic press.
  163. Martino, Giovanni Da San, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020. A survey on computational propaganda detection. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4826-4832, ijcai.org.
  164. Mathew, Binny, Anurag Illendula, Punyajoy Saha, Soumya Sarkar, Pawan Goyal, and Animesh Mukherjee. 2020. Hate begets hate: A temporal study of hate speech. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2):1-24.
  165. Mendelsohn, Julia, Ceren Budak, and David Jurgens. 2021. Modeling framing in immigration discourse on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2219-2263, Association for Computational Linguistics, Online.
  166. Mihalcea, Rada and Vivi Nastase. 2012. Word epoch disambiguation: Finding how words change over time. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 259-263, Association for Computational Linguistics, Jeju Island, Korea.
  167. Mihalcea, Rada and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 531-538, Association for Computational Linguistics, Vancouver, British Columbia, Canada.
  168. Min, Bonan and Xiaoxi Zhao. 2019. Measure country-level socio-economic indicators with streaming news: An empirical study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1249-1254, Association for Computational Linguistics, Hong Kong, China.
  169. Mishra, Abhijit, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6144-6154, Association for Computational Linguistics, Hong Kong, China.
  170. Mohammad, Saif, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31-41, Association for Computational Linguistics, San Diego, California.
  171. Murphy, Kevin M and Andrei Shleifer. 2004. Persuasion in politics. American Economic Review, 94(2):435-439.
  172. Murray, Michael et al. 2015. Narrative psychology. Qualitative psychology: A practical guide to research methods, pages 85-107.
  173. Muthukrishna, Michael, Adrian V Bell, Joseph Henrich, Cameron M Curtin, Alexander Gedranovich, Jason McInerney, and Braden Thue. 2020. Beyond western, educated, industrial, rich, and democratic (weird) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological science, 31(6):678-701.
  174. Nelson, Laura K. 2021. Cycles of conflict, a century of continuity: the impact of persistent place-based political logics on social movement strategy. American Journal of Sociology, 127(1):1-59.
  175. Nguyen, Dong, A Seza Do gruöz, Carolyn P Rosé, and Franciska de Jong. 2016. Computational sociolinguistics: A survey. Computational linguistics, 42(3):537-593.
  176. Nguyen, Thien Hai and Kiyoaki Shirai. 2015. Topic modeling based sentiment analysis on social media for stock market prediction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1354-1364, Association for Computational Linguistics, Beijing, China.
  177. Nguyen, Thong, Andrew Yates, Ayah Zirikly, Bart Desmet, and Arman Cohan. 2022. Improving the generalizability of depression detection by leveraging clinical questionnaires. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8446-8459.
  178. Niculae, Vlad and Cristian Danescu-Niculescu-Mizil. 2014. Brighter than gold: Figurative language in user generated comparisons. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2008-2018, Association for Computational Linguistics, Doha, Qatar.
  179. Novikova, Jekaterina, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Association for Computational Linguistics, Copenhagen, Denmark.
  180. Ofer, Dan and Dafna Shahaf. 2022. Cards against AI: Predicting humor in a fill-in-the-blank party game. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5397-5403, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates.
  181. Omitaomu, Damilola, Shabnam Tafreshi, Tingting Liu, Sven Buechel, Chris Callison-Burch, Johannes Eichstaedt, Lyle Ungar, and João Sedoc. 2022. Empathic conversations: A multi-level dataset of contextualized conversations. ArXiv preprint, abs/2205.12698. OpenAI. 2023. Gpt-4 technical report. · arXiv
  182. Ortu, Marco, Alessandro Murgia, Giuseppe Destefanis, Parastou Tourani, Roberto Tonelli, Michele Marchesi, and Bram Adams. 2016. The emotional side of software developers in jira. In Proceedings of the 13th international conference on mining software repositories, pages 480-483.
  183. Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
  184. Park, Joon Sung, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1-18.
  185. Pennebaker, James W and Laura A King. 1999. Linguistic styles: language use as an individual difference. Journal of personality and social psychology, 77(6):1296.
  186. Perez, Ethan, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Advances in neural information processing systems, 34:11054-11070.
  187. Pfeil, Ulrike and Panayiotis Zaphiris. 2007. Patterns of empathy in online communication. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 919-928.
  188. Picard, Rosalind W. 2000. Affective computing. MIT press.
  189. Pilehvar, Mohammad Taher and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Association for Computational Linguistics, Minneapolis, Minnesota.
  190. Piper, Andrew, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 298-311.
  191. Plutchik, Robert. 1980. A general psychoevolutionary theory of emotion. In Theories of emotion. Elsevier, pages 3-33.
  192. Post, Matt. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Association for Computational Linguistics, Brussels, Belgium.
  193. Prabhakaran, Vinodkumar, Owen Rambow, and Mona Diab. 2012. Predicting overt display of power in written dialogs. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 518-522, Association for Computational Linguistics, Montréal, Canada.
  194. Prabhakaran, Vinodkumar, Emily E. Reid, and Owen Rambow. 2014. Gender and power: How gender and gender environment affect manifestations of power. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1965-1976, Association for Computational Linguistics, Doha, Qatar.
  195. Preece, Jenny. 1998. Empathic communities: Reaching out across the web. interactions, 5(2):32-43.
  196. Preoţiuc-Pietro, Daniel, Vasileios Lampos, and Nikolaos Aletras. 2015. An analysis of the user occupational class through Twitter content. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1754-1764, Association for Computational Linguistics, Beijing, China.
  197. Preoţiuc-Pietro, Daniel, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction of Twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729-740, Association for Computational Linguistics, Vancouver, Canada.
  198. Purschke, Christoph and Dirk Hovy. 2019. Lörres, möppes, and the swiss.(re) discovering regional patterns in anonymous social media data. Journal of Linguistic Geography, 7(2):113-134.
  199. Qin, Chengwei, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? ArXiv preprint, abs/2302.06476. · arXiv
  200. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551.

200 of 279 shown

Cited by

Sorted by citations

Authors
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2305.03514
arXiv
2305.03514
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review