A fine-grained comparison of pragmatic language understanding in humans and language models
Open accessAbstract
Pragmatics and non-literal language understanding are essential to human communication, and present a long-standing challenge for artificial language models.We perform a finegrained comparison of language models and humans on seven pragmatic phenomena, using zero-shot prompting on an expert-curated set of English materials.We ask whether models (1) select pragmatic interpretations of speaker utterances, (2) make similar error patterns as humans, and (3) use similar linguistic cues as humans to solve the tasks.We find that the largest models achieve high accuracy and match human error patterns: within incorrect responses, models favor literal interpretations over heuristic-based distractors.We also find preliminary evidence that models and humans are sensitive to similar linguistic cues.Our results suggest that pragmatic behaviors can emerge in models without explicitly constructed representations of mental states.However, models tend to struggle with phenomena relying on social expectation violations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221 · see citing works
Politeness : Some Universals in Language Usage
Brown, P. (1987) · Cited by 11,901 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Does the autistic child have a “theory of mind” ?
Baron-Cohen, S., Leslie, A. M., & Frith, U. (1985) · Cognition · doi:10.1016/0010-0277(85)90022-8 · Cited by 8,599 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
The psychology of humor: an integrative approach
Martin, R. A., & Ford, T. E. (2008) · Choice Reviews Online · doi:10.5860/choice.45-2902 · Cited by 1,680 · see citing works
On the semantic properties of logical operators in English
Horn, L. R. (1972) · University Microfilms International eBooks · Cited by 1,504 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Communicative competence and theory of mind in autism: A test of relevance theory
Happe, F. G. E. (1993) · Cognition · doi:10.1016/0010-0277(93)90026-r · Cited by 1,161 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · see citing works
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · see citing works
Predicting pragmatic reasoning in language games
Frank, M. C., & Goodman, N. D. (2012) · Science · doi:10.1126/science.1218633 · Cited by 868 · see citing works
Searle, J. R. (1975) · BRILL eBooks · doi:10.1163/9789004368811_004 · Cited by 804 · see citing works
Core mechanisms in ‘theory of mind’
Leslie, A. M., Friedman, O., & German, T. P. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.10.001 · Cited by 800 · see citing works
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 776 · see citing works
How to Construct a Minimal Theory of Mind
Butterfill, S., & Apperly, I. A. (2013) · Mind & Language · doi:10.1111/mila.12036 · Cited by 643 · see citing works
Irony as relevant inappropriateness
Attardo, S. (2000) · Journal of Pragmatics · doi:10.1016/s0378-2166(99)00070-3 · Cited by 630 · see citing works
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605 · see citing works
Social IQa: Commonsense Reasoning about Social Interactions
Sap, M., Rashkin, H., Chen, D., LeBras, R., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1454 · Cited by 588 · see citing works
Wilson, D., & Sperber, D. (1992) · Lingua · doi:10.1016/0024-3841(92)90025-e · Cited by 579 · see citing works
Cultural Foundations of Learning
Li, J. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139028400 · Cited by 490 · see citing works
Submentalizing: I Am Not Really Reading Your Mind
Heyes, C. M. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613518076 · Cited by 438 · see citing works
Over-reliance on English hinders cognitive science
Blasi, D. E., Henrich, J., Adamou, E., Kemmerer, D., & Majid, A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.09.015 · Cited by 396 · see citing works
VEATCH, T. C. (1998) · humr · doi:10.1515/humr.1998.11.2.161 · Cited by 341 · see citing works
Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J. Y., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.703 · Cited by 325 · see citing works
Neural evidence that utterance-processing entails mentalizing: The case of irony
Spotorno, N., Koun, É., Prado, J., der Henst, J. V., & Noveck, I. A. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.06.046 · Cited by 256 · see citing works
(2011) · doi:10.4324/9780203833926 · Cited by 249 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · see citing works
Revisiting the Evaluation of Theory of Mind through Question Answering
Le, M., Boureau, Y.-L., & Nickel, M. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1598 · Cited by 220 · see citing works
Brennan, S. E., Galati, A., & Kuhlen, A. K. (2010) · The Psychology of learning and motivation/The psychology of learning and motivation · doi:10.1016/s0079-7421(10)53008-1 · Cited by 205 · see citing works
Camden, C. T., Motley, M. T., & Wilson, A. (1984) · Western Journal of Speech Communication · doi:10.1080/10570318409374167 · Cited by 196 · see citing works
Probabilistic pragmatics, or why Bayes’ rule is probably important for pragmatics
Franke, M., & Jäger, G. (2016) · Zeitschrift für Sprachwissenschaft · doi:10.1515/zfs-2016-0002 · Cited by 190 · see citing works
Contextual effects in understanding indirect requests∗
Gibbs, R. W. (1979) · Discourse Processes · doi:10.1080/01638537909544450 · Cited by 166 · see citing works
Pragmatic reasoning through semantic inference
Bergen, L., Lévy, R., & Goodman, N. D. (2016) · Semantics and Pragmatics · doi:10.3765/sp.9.20 · Cited by 159 · see citing works
Neural language models as psycholinguistic subjects: Representations of syntactic state
Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., & Lévy, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1004 · Cited by 154 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Self-citationAndrés-Roqueta, C., & Katsos, N. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00996 · Cited by 128 · see citing works
Embedded Implicatures as Pragmatic Inferences under Compositional Lexical Uncertainty
Potts, C., Lassiter, D., Lévy, R., & Frank, M. C. (2015) · Journal of Semantics · doi:10.1093/jos/ffv012 · Cited by 127 · see citing works
Shah, C. S., & Bender, E. M. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505816 · Cited by 125 · see citing works
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Sap, M., Le Bras, R., Fried, D., & Choi, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.248 · Cited by 118 · see citing works
Evaluating Compositionality in Sentence EmbeddingsPreprint
Dasgupta, I., Guo, D., Stuhlmüller, A., Gershman, S. J., & Goodman, N. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04302 · Cited by 110 · see citing works
Pragmatics across Languages and Cultures
Trosborg, A. (2010) · Mouton eBooks · doi:10.1515/9783110214444 · Cited by 109 · see citing works
The Rational Speech Act Framework
Degen, J. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031220-010811 · Cited by 99 · see citing works
Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00901 · Cited by 98 · see citing works
Language Models as Agent Models
Andreas, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.423 · Cited by 92 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
Jeretič, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.768 · Cited by 83 · see citing works
What if…? Imagining non-Western perspectives on pragmatic theory and practice
Ameka, F. K., & Terkourafi, M. (2019) · Journal of Pragmatics · doi:10.1016/j.pragma.2019.04.001 · Cited by 82 · see citing works
Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
Ullman, T. (2023) · arXiv · doi:10.48550/arxiv.2302.08399 · Cited by 81 · see citing works
Analyzing Compositionality-Sensitivity of NLI Models
Nie, Y., Wang, Y., & Bansal, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016867 · Cited by 72 · see citing works
Pragmatics and theory of mind in older adults’ humor comprehension
Bischetti, L., Ceccato, I., Lecce, S., Cavallini, E., & Bambini, V. (2019) · Current Psychology · doi:10.1007/s12144-019-00295-w · Cited by 67 · see citing works
Pragmatics in human-computer conversations
Saygın, A. P., & Cicekli, I. (2002) · Journal of Pragmatics · doi:10.1016/s0378-2166(02)80001-7 · Cited by 65 · see citing works
Polite Speech Emerges From Competing Social Goals
Yoon, E. J., Tessler, M. H., Goodman, N. D., & Frank, M. C. (2020) · Open Mind · doi:10.1162/opmi_a_00035 · Cited by 64 · see citing works
Pragmatic markers: the missing link between language and Theory of Mind
Rubio‐Fernández, P. (2020) · Synthese · doi:10.1007/s11229-020-02768-z · Cited by 55 · see citing works
Formalizing the Pragmatics of Metaphor Understanding
Kao, J., Bergen, L., & Goodman, N. D. (2014) · eScholarship (California Digital Library) · Cited by 54 · see citing works
Evaluating Theory of Mind in Question Answering
Nematzadeh, A., Burns, K., Grant, E., Gopnik, A., & Griffiths, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1261 · Cited by 53 · see citing works
Jacoby, N., & Fedorenko, E. G. (2018) · Language Cognition and Neuroscience · doi:10.1080/23273798.2018.1525494 · Cited by 53 · see citing works
Self-citationHumor and the Incongruity Hypothesis
Deckers, L. H., & Kizer, P. (1975) · The Journal of Psychology · doi:10.1080/00223980.1975.9915778 · Cited by 47 · see citing works
Two mentalizing capacities and the understanding of two types of lie telling in children.
Hsu, Y. K., & Cheung, H. (2012) · Developmental Psychology · doi:10.1037/a0031128 · Cited by 43 · see citing works
Testing the Ability of Language Models to Interpret Figurative Language
Liu, E., Cui, C., Zheng, K., & Neubig, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.330 · Cited by 42 · see citing works
Common ground, cooperation, and recipient design in human-computer interactions
Dombi, J., Sydorenko, T., & Timpe‐Laughlin, V. (2022) · Journal of Pragmatics · doi:10.1016/j.pragma.2022.03.001 · Cited by 40 · see citing works
Assessing Theory of Mind by Humor: The Humor Comprehension and Appreciation Test (ToM-HCAT)
Aykan, S., & Nalçacı, E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01470 · Cited by 36 · see citing works
Theory of Mind, pragmatics and the brain
Enrici, I., Bara, B. G., & Adenzato, M. (2019) · Pragmatics & Cognition · doi:10.1075/pc.19010.enr · Cited by 32 · see citing works
Incrementality and efficiency shape pragmatics across languages
Rubio‐Fernández, P., & Jara‐Ettinger, J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1922067117 · Cited by 29 · see citing works
Demedardi, M.-J., Brechet, C., Gentaz, É., & Monnier, C. (2020) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2020.105045 · Cited by 29 · see citing works
Harnessing the linguistic signal to predict scalar inferences
Schuster, S., Chen, Y., & Degen, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.479 · Cited by 25 · see citing works
IMPLI: Investigating NLI Models’ Performance on Figurative Language
Stowe, K., Utama, P. A., & Gurevych, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.369 · Cited by 25 · see citing works
Let's talk (ironically) about the weather: Modeling verbal irony.
Kao, J., & Goodman, N. D. (2015) · eScholarship (California Digital Library) · Cited by 23 · see citing works
Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective
Tong, X., Shutova, E., & Lewis, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.372 · Cited by 23 · see citing works
Moss, J., & Schunn, C. D. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00562 · Cited by 21 · see citing works
Rational speech act models of pragmatic reasoning in reference gamesPreprint
Frank, M. C. (2016) · doi:10.31234/osf.io/f9y6b · Cited by 21 · see citing works
Concadia: Towards Image-Based Text Generation with a Purpose
Kreiss, E., Fang, F., Goodman, N. D., & Potts, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.308 · Cited by 21 · see citing works
Theory of mind and preschoolers’ understanding of misdeed and politeness lies.
Vendetti, C., Kamawar, D., & Andrews, K. (2018) · Developmental Psychology · doi:10.1037/dev0000666 · Cited by 21 · see citing works
Cross-speaker repetition and epistemic stance in Tzeltal, Yucatec, and Zapotec conversations
Brown, P., Sicoli, M. A., & Guen, O. L. (2021) · Journal of Pragmatics · doi:10.1016/j.pragma.2021.07.005 · Cited by 20 · see citing works
GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning
Zheng, Z., Qiu, S., Fan, L., Zhu, Y., & Zhu, S. C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.182 · Cited by 15 · see citing works
Floyd, S. (2021) · Annual Review of Anthropology · doi:10.1146/annurev-anthro-101819-110158 · Cited by 13 · see citing works
Self-citationLanguage specific constraints on conversation: Evidence from Danish and NorwegianPreprint
Dideriksen, C., Christiansen, M. H., Dingemanse, M., Højmark‐Bertelsen, M., Johansson, C., Tylén, K., & Fusaroli, R. (2022) · doi:10.31234/osf.io/t3s6c · Cited by 10 · see citing works
Not unreasonable: Carving vague dimensions with contraries and contradictions.
Tessler, M. H., & Franke, M. (2018) · eScholarship (California Digital Library) · Cited by 10 · see citing works
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models
Beyer, A., Loáiciga, S., & Schlangen, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.328 · Cited by 8 · see citing works
Predicting Scalar Inferences From "Or" to "Not Both" Using Neural Sentence Encoders
Li, E., Schuster, S., & Degen, J. (2021) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/xr01-a852 · Cited by 6 · see citing works
Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE
Gu, Y., Fu, Y., Pyatkin, V., Magnusson, I., Mishra, B. D., & Clark, P. A. (2022) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2022.flp-1.12 · Cited by 6 · see citing works
Understanding jokes draws most heavily on the Theory of Mind brain networkPreprint
Struhl, M. K., Gallée, J., Balewski, Z. Z., & Fedorenko, E. G. (2016) · doi:10.31234/osf.io/h2nyx · Cited by 6 · see citing works
Self-citationTalking with tact: Polite language as a balance between informativity and kindness.
Yoon, E. J., Tessler, M. H., Goodman, N. D., & Frank, M. C. (2016) · Cognitive Science · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- W.C. Booth. 1974. A Rhetoric of Irony. Litera- ture/Criticism -The University of Chicago Press. University of Chicago Press.
- G. Lakoff and M. Johnson. 1980. Metaphors We Live By. University of Chicago Press.
- Stephen Levinson. 2000. Presumptive meaning: The theory of generalized conversational implicature. MIT Press.
- Eleonore Lumer and Hendrik Buschmeier. 2022. Mod- eling Social Influences on Indirectness in a Rational Speech Act Approach to Politeness. In Proceedings of the 44th Annual Conference of the Cognitive Sci- ence Society.
- Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: A cognitive perspective. arXiv preprint.
- Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022. Large language models are not zero-shot communi- cators. arXiv preprint.
- D. Wilson and D. Sperber. 2012. Meaning and Rele- vance. Cambridge University Press.
- George Yule. 1996. Pragmatics, 1 edition. Oxford Introduction to Language Study. Oxford University Press.
Cited by
Sorted by citations
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Self-citationLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Yiu, E., Kosoy, E., & Alison Gopnik, C. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231201401 · Cited by 61 · see citing works
PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities
Sravanthi, S., Doshi, M., Tankala, P., Murthy, R., Dabre, R., & Bhattacharyya, P. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.719 · Cited by 46 · see citing works
Can large language models help augment English psycholinguistic datasets?
Trott, S. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02337-z · Cited by 45 · see citing works
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Shapira, N., Levy, M., Alavi, S. H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M., & Shwartz, V. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.138 · Cited by 44 · see citing works
Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.699 · Cited by 38 · see citing works
Andersson, M., & McIntyre, D. (2025) · Journal of Pragmatics · doi:10.1016/j.pragma.2025.02.001 · Cited by 31 · see citing works
“Personhood and AI: Why large language models don’t understand us”
Browning, J. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01724-y · Cited by 27 · see citing works
Manner implicatures in large language models
Cong, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-80571-3 · Cited by 27 · see citing works
Rubio‐Fernández, P. (2024) · Lingua · doi:10.1016/j.lingua.2024.103851 · Cited by 22 · see citing works
Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
Fried, D., Tomlin, N., Hu, J., Patel, R., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.findings-emnlp.840 · Cited by 18 · see citing works
Self-citationTowards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Ma, Z., Sansom, J., Peng, R., & Chai, J. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.72 · Cited by 17 · see citing works
Sievers, B., & Thornton, M. A. (2024) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsae014 · Cited by 16 · see citing works
Humans Learn Language from Situated Communicative Interactions. What about Machines?
Beuls, K., & Eecke, P. V. (2024) · Computational Linguistics · doi:10.1162/coli_a_00534 · Cited by 15 · see citing works
Dealing with Semantic Underspecification in Multimodal NLP
Pezzelle, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.675 · Cited by 14 · see citing works
The pragmatic profile of ChatGPT: assessing the communicative skills of a conversational agentPreprint
Pietro, C. B. D. S., Frau, F., Mangiaterra, V., & Bambini, V. (2023) · doi:10.31234/osf.io/ckghw · Cited by 13 · see citing works
Do Large Language Models Understand Conversational Implicature – A Case Study with a Chinese Sitcom
Yue, S., Song, S., Cheng, X., & Hu, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_24 · Cited by 11 · see citing works
Re-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10 · see citing works
Self-citationDo Multimodal Large Language Models and Humans Ground Language Similarly?
Jones, C. R., Bergen, B. K., & Trott, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00531 · Cited by 9 · see citing works
Jones, C. R., Trott, S., & Bergen, B. K. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00674 · Cited by 9 · see citing works
GPT VS. HUMAN FOR SCIENTIFIC REVIEWS: A DUAL SOURCE REVIEW ON APPLICATIONS OF CHATGPT IN SCIENCE
Wu, C., Varghese, A. J., Oommen, V., & Karniadakis, G. E. (2024) · Journal of Machine Learning for Modeling and Computing · doi:10.1615/jmachlearnmodelcomput.2024052432 · Cited by 8 · see citing works
Evaluating convergence between two data visualization literacy assessments
Brockbank, E., Verma, A., Lloyd, H. E. D., Huey, H., Padilla, L., & Fan, J. E. (2025) · Cognitive Research Principles and Implications · doi:10.1186/s41235-025-00622-9 · Cited by 7 · see citing works
Enhancing Interpretability Using Human Similarity Judgements to Prune Word Embeddings
Manrique, N. F., Bao, W., Herbelot, A., & Hasson, U. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.13 · Cited by 6 · see citing works
Neural Generative Models and the Parallel Architecture of Language: A Critical Review and Outlook
Rambelli, G., Chersoni, E., Testa, D., Blache, P., & Lenci, A. (2024) · Topics in Cognitive Science · doi:10.1111/tops.12733 · Cited by 6 · see citing works
Yerukola, A., Vaduguru, S., Fried, D., & Sap, M. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-short.26 · Cited by 6 · see citing works
Backus, A., Cohen, M., Cohn, N., Faber, M., Krahmer, E., Laparle, S., Maier, E., van Miltenburg, E., Roelofsen, F., Sciubba, E., et al. (2023) · Linguistics in the Netherlands · doi:10.1075/avt.00094.bac · Cited by 6 · see citing works
Opinion paper: generative AI and the future of scientometrics
Lepori, B., Andersen, J. P., & Donnay, K. (2026) · Scientometrics · doi:10.1007/s11192-026-05667-1 · Cited by 5 · see citing works
Iida, A., Okuoka, K., Fukuda, S., Omori, T., Nakashima, R., & Osawa, M. (2024) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3687272.3688303 · Cited by 5 · see citing works
Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning
Wu, S., Yang, S., Chen, Z., & Su, Q. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1258 · Cited by 5 · see citing works
Comparing human and LLM politeness strategies in free production
Zhao, H., & Hawkins, R. D. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.820 · Cited by 5 · see citing works
Collaborative Growth: When Large Language Models Meet Sociolinguistics
Nguyen, D. (2025) · Language and Linguistics Compass · doi:10.1111/lnc3.70010 · Cited by 5 · see citing works
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4 · see citing works
AI, be less ‘stereotypical’: ChatGPT’s speech is conventional but never unique
Tantucci, V., & Sparvoli, C. (2025) · Intercultural Pragmatics · doi:10.1515/ip-2025-2003 · Cited by 4 · see citing works
Ungrounding intelligence: From biological minds to artificial cognition
Abbate, F., & Gaudenzi, R. (2025) · Philosophical Psychology · doi:10.1080/09515089.2025.2592680 · Cited by 3 · see citing works
Solidjonov, D. (2026) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-026-00200-5 · Cited by 3 · see citing works
Qu, J., Arguello, J., & Wang, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658547 · Cited by 3 · see citing works
Floyd, S., Jouravlev, O., Mineroff, Z., Poliak, M., Gibson, E. A. F., & Fedorenko, E. G. (2023) · doi:10.31234/osf.io/e2xta · Cited by 3 · see citing works
Self-citationChatGPT for President! Presupposed content in politicians versus GPT-generated texts
Garassino, D., Brocca, N., & Masia, V. (2025) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2025.100156 · Cited by 2 · see citing works
Pragmatic inference and mapping for conversational implicature
Hu, J., Liu, Y., Ma, L., Zhang, W.-N., & Liu, T. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02741-7 · Cited by 2 · see citing works
Self-citationBoldi, A., Gabbatore, I., & Bosco, F. M. (2025) · Electronics · doi:10.3390/electronics14224411 · Cited by 2 · see citing works
How Important Is Language for Human-Like Intelligence?
Lupyan, G., Gentry, H., & Zettersten, M. (2026) · Perspectives on Psychological Science · doi:10.1177/17456916251398539 · Cited by 2 · see citing works
Large Language Models are biased to overestimate profoundness
Herrera-Berg, E., Browne, T. V., León-Villagrá, P., Vives, M., & Calderon, C. B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.599 · Cited by 2 · see citing works
Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models
Zhao, L., Nguyen, K., & Daumé, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.227 · Cited by 2 · see citing works
Sieker, J., & Zarrieß, S. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.14 · Cited by 2 · see citing works
Turing Jest: Distributional Semantics and One‐Line Jokes
Trott, S., Walker, D. E., Taylor, S. M., & Seana Coulson, A. (2025) · Cognitive Science · doi:10.1111/cogs.70066 · Cited by 2 · see citing works
LLMs replicate metaphor norms based on word co-occurrence but struggle with topic-vehicle mappings
Pissani, L., Jobanputra, M., & Demberg, V. (2026) · Frontiers in Language Sciences · doi:10.3389/flang.2026.1756514 · Cited by 2 · see citing works
Prompt Engineering 101 Prompt Engineering Guidelines from a Linguistic Perspective
Han, W., Wei, X., Cui, X., Cheng, N., Jiang, G., Qian, W., & Zhang, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_34 · Cited by 1 · see citing works
Language Understanding in the Human Machine Era 2024: Proceedings of 1st LUHME Workshop
(2024) · Faculdade de Letras da Universidade do Porto eBooks · doi:10.21747/978-989-9193-12-3/lan · Cited by 1 · see citing works
Relevant answers to polar questions
Hawkins, R. D., Tsvilodub, P., Bergey, C. A., Goodman, N. D., & Franke, M. (2025) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2023.0505 · Cited by 1 · see citing works
Not a Bug, a Feature? How Repetition Enhances Perceived Naturalness in Synthesized Conversation
Park, Y., Cho, H., & Song, J. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2582771 · Cited by 1 · see citing works
Floyd, S., Jouravlev, O., Poliak, M., Mineroff, Z., Gibson, E. A. F., & Fedorenko, E. G. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2424400122 · Cited by 1 · see citing works
Self-citationLLM-Based Evaluation of Utterances with Implicature Understanding
Iida, A., Okuoka, K., Omori, T., Nakashima, R., & Osawa, M. (2025) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3765766.3765841 · Cited by 1 · see citing works
Pragmatics in Human–Computer Interaction: A Systematic Literature Review
Wang, W., & Liu, X. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2635678 · Cited by 1 · see citing works
The Anthropomimetic Turn in Contemporary AI
Shevlin, H. (2026) · Springer eBooks · doi:10.1007/978-3-032-11814-1_26 · Cited by 1 · see citing works
Economidou‐Kogetsidis, M., & Lai, S. (2026) · Journal of Pragmatics · doi:10.1016/j.pragma.2026.05.009 · Cited by 1 · see citing works
Linguistics and human brain: a perspective of computational neuroscience
Zhang, F., Chai, B., Wu, Y., Siok, W. T., & Wang, N. (2026) · Cognitive Neurodynamics · doi:10.1007/s11571-026-10496-2 · Cited by 1 · see citing works
Large language models fail to derive atypicality inferences in a human-like manner
Kurch, C., Ryzhova, M., & Demberg, V. (2024) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2024.cmcl-1.8 · Cited by 1 · see citing works
Context, conflict, and the time-course of interpreting ironyPreprint
Adler, R., Salig, L. K., Novick, J. M., & Huang, Y. T. (2025) · doi:10.31234/osf.io/kv7c5_v1 · Cited by 1 · see citing works
Comparing the Evaluation and Production of Loophole Behavior in Humans and Large Language Models
Murthy, S. K., Parece, K., Bridgers, S., Qian, P., & Ullman, T. D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.264
Evaluating convergence between two data visualization literacy assessmentsPreprint
Brockbank, E., Verma, A., Lloyd, H. E. D., Huey, H., Padilla, L., & Fan, J. E. (2025) · doi:10.31234/osf.io/kac4z_v2
Word predictability in Portuguese: Cloze norming study vs. LLMsPreprint
Aristia, J. (2025) · doi:10.31234/osf.io/e8fzj_v1
Jafari, M., Hua, Y., Xue, H., & Salim, F. D. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.287
Dasar Neurobiologis Akuisisi Bahasa Dini: Mengapa Anak masih Mengungguli Kecerdasan Buatan
Tjandrawinata, R. R. (2025) · MEDICINUS · doi:10.56951/qk5jw768
Machine Versus Human: A Comparative Study on Satire Annotation
Corley, T. R., & Govender, A. (2026) · Oxford Journal of Student Scholarship · doi:10.65161/rec2fx7fn7b22gsyj
Feng, S., Chen, J., Zhou, M., Guowei, W. U., Liu, X., Li, Y., Zhou, T., & Yang, Y. (2026) · Current Psychology · doi:10.1007/s12144-026-09285-1
Sripada, C., McInnerney, A., & Lewis, R. L. (2026) · Behavioral and Brain Sciences · doi:10.1017/s0140525x26104725
Human and AI Refusals in Egyptian Arabic: A Comparative Pragmatic AnalysisPreprint
Morkus, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7073045
The difference between artificial intelligence implicatures and human implicatures in conversation
Hawarin, R., & Eshreteh, M. K. M. (2026) · Journal of Language, Literature, Social and Cultural Studies · doi:10.58881/jllscs.v4i2.475
Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilitie
Kankowski, F., Solstad, T., Zarrieß, S., & Bott, O. (2026) · Dialogue & Discourse · doi:10.5210/dad.2026.201
Large Language Models as Distributional Baselines for Language Tasks
Trott, S., Michaelov, J. A., Jones, C., Chang, T. A., & Bergen, B. K. (2026) · Open Mind · doi:10.1162/opmi.a.373
Boldi, A., Gabbatore, I., & Bosco, F. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7366891
Benchmarking Pragmatic Reasoning of Large Language Models in the Czech Language
Karpizenkov, Y., Horák, A., & Lacina, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_10
Green, D. (2026) · International Journal for the Semiotics of Law - Revue internationale de Sémiotique juridique · doi:10.1007/s11196-026-10572-0
Speech as Inference, Signal, and Social Action: A Sequence-Sensitive Integrative ReviewPreprint
Svet, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7435323
What Do You Mean? Exploring the Alleged Theory of Mind Abilities of Large Language Models
Lombardi, A., & Lenci, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.807
Les grands modèles de langue : bilan et perspectives pour la linguistique
Poibeau, T. (2026) · Langages · doi:10.3917/lang.243.0073
Towards Complexity-Aware Evaluation of Large Language Models in text Classification
Kyritsis, K., Perikos, I., Liapis, C. M., Kapoulas, V., & Paraskevas, M. (2026) · Pan-Hellenic Conference on Progress in Computing and Informatics (PCI) · doi:10.1145/3829216.3829287
Badara, A., Rifa’ie, M., Yusoh, R., & Salama, I. A.-K. (2026) · Rainbow Journal of Literature Linguistics and Cultural Studies · doi:10.15294/rainbow.v15i3.65667
(2026) · V&R unipress eBooks · doi:10.14220/9783737020428.209
- Authors
- Hu, J., Floyd, S., Jouravlev, O., Fedorenko, E., & Gibson, E.
- Affiliations
- Massachusetts Institute of Technology (US)
- Sarah Lawrence College (US)
- Carleton University (CA)
- McGovern Institute for Brain Research (US)
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 4194-4213
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385571325
- Cited by
- 78
- References
- 86
- Citations by year
- 2026: 23 · 2025: 21 · 2024: 19 · 2023: 15
- FWCI
- 12.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2116918, 1745302)
- Simons Center for the Social Brain, Massachusetts Institute of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- figurative language, zero-shot prompting, error patterns, linguistic cues, mental state representation, English pragmatics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
