PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities
Open accessAbstract
LLMs have demonstrated remarkable capability for understanding semantics, but their understanding of pragmatics is not well studied.To this end, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely, Implicature, Presupposition, Reference, and Deixis.We curate high-quality test sets for each task, consisting of Multiple Choice Question Answers (MCQA).PUB includes a total of 28k data points, 6.1k are newly annotated.We evaluate nine models varying in the number of parameters and type of training.Our study reveals several key observations about the pragmatic capabilities of LLMs: 1. chat-fine-tuning strongly benefits smaller models, 2. large base models are competitive with their chat-fine-tuned counterparts, 3. there is a huge variance in performance across different pragmatics phenomena, and 4. a noticeable performance gap between human capabilities and model capabilities.We hope that PUB will enable comprehensive evaluation of LLM's pragmatic reasoning capabilities.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These findings, together with those by Kabbara & Cheung (2022), Sieker and Zarrieß (2023) and Sravanthi et al. (2024), suggest that LLMs tend to rely on superficial cues when processing inferential phenomena.
“these findings, together with those by Kabbara & Cheung (2022), Sieker and Zarrie ß (2023) and Sravanthi et al. (2024), suggest that LLMs tend to rely on superficial cues when processing inferential phenomena.”
From ChatGPT for President! Presupposed content in politicians versus GPT-generated texts (2025) · cites this paper as Sravanthi et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,998 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · see citing works
<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · see citing works
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · see citing works
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
Jeretič, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.768 · Cited by 83 · see citing works
A fine-grained comparison of pragmatic language understanding in humans and language models
Hu, J., Floyd, S., Jouravlev, O., Fedorenko, E., & Gibson, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.230 · Cited by 78 · see citing works
Testing the Ability of Language Models to Interpret Figurative Language
Liu, E., Cui, C., Zheng, K., & Neubig, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.330 · Cited by 42 · see citing works
“I’d rather just go to bed”: Understanding Indirect Answers
Louis, A. P., Roth, D., & Radlinski, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.601 · Cited by 37 · see citing works
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
Kim, N., Pavlick, E., Ayan, B. K., & Ramachandran, D. (2021) · doi:10.18653/v1/2021.acl-long.304 · Cited by 30 · see citing works
NOPE: A Corpus of Naturally-Occurring Presuppositions in English
Parrish, A., Schuster, S., Warstadt, A., et al. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.28 · Cited by 11 · see citing works
Let’s do it “again”: A First Computational Approach to Detecting Adverbial Presupposition Triggers
Cianflone, A., Feng, Y., Kabbara, J., & Cheung, J. C. K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1256 · Cited by 8 · see citing works
(QA)$^2$: Question Answering with Questionable AssumptionsPreprint
Kim, N., Htut, P. M., Bowman, S. R., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.10003 · Cited by 4 · see citing works
PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations
Qi, P., Du, N., Manning, C. D., & Huang, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.385 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Jens Allwood. 1981. On the distinctions between se- mantics and pragmatics. In Crossing the Boundaries in Linguistics: Studies Presented to Manfred Bier- wisch, pages 177-189. Springer.
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. CoRR, abs/2110.14168. · arXiv
- Lingjia Deng, Janyce Wiebe, and Yoonjung Choi. 2014. Joint inference and disambiguation of implicit senti- ments via implicature constraints. In COLING 2014, 25th International Conference on Computational Lin- guistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland, pages 79-88. ACL.
- Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41-58. Brill.
- Laurence R Horn and Gregory L Ward. 2004. The handbook of pragmatics. Wiley Online Library.
- Jad Kabbara and Jackie Chi Kit Cheung. 2022. Inves- tigating the performance of transformer-based NLI models on presuppositional inferences. In Proceed- ings of the 29th International Conference on Compu- tational Linguistics, COLING 2022, Gyeongju, Re- public of Korea, October 12-17, 2022, pages 779- 785. International Committee on Computational Lin- guistics.
- Jad Kabbara and Jackie Chi Kit Cheung. 2023. Investi- gating the effect of pre-finetuning bert models on nli involving presuppositions. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 10482-10494.
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:453-466.
- Shibamouli Lahiri. 2015. Squinky! A corpus of sentence-level formality, informativeness, and im- plicature. CoRR, abs/1506.02306. · arXiv
- George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press.
- Elissa Li, Sebastian Schuster, and Judith Degen. 2021. Predicting scalar inferences from "or" to "not both" using neural sentence encoders. In Proceedings of the Society for Computation in Linguistics 2021, pages 446-450.
- Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natu- ral Language Processing, IJCNLP 2017, Taipei, Tai- wan, November 27 -December 1, 2017 -Volume 1: Long Papers, pages 986-995. Asian Federation of Natural Language Processing.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. ArXiv, abs/1609.07843. · arXiv
- Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fer- nández. 2016. The lambada dataset: Word predic- tion requiring a broad discourse context. ArXiv, abs/1606.06031. · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Cited by
Sorted by citations
Large language models forecast patient health trajectories enabling digital twins
Makarov, N., Bordukova, M., Quengdaeng, P., Garger, D., Rodriguez‐Esteban, R., Schmich, F., & Menden, M. P. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02004-3 · Cited by 65 · see citing works
Technological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51 · see citing works
Manner implicatures in large language models
Cong, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-80571-3 · Cited by 27 · see citing works
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2025) · arXiv · doi:10.48550/arxiv.2507.19218 · Cited by 26 · see citing works
Haroon, R., & Dogar, F. R. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675633 · Cited by 19 · see citing works
MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models
Park, D., Lee, J.-W., Park, S., Jeong, H., Koo, Y., Hwang, S., Park, S., & Lee, S. (2024) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2024.genbench-1.7 · Cited by 11 · see citing works
Re-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10 · see citing works
Conversational Alignment With Artificial Intelligence in Context
Sterken, R., & Kirkpatrick, J. R. (2024) · Philosophical Perspectives · doi:10.1111/phpe.12205 · Cited by 10 · see citing works
Kostikova, A., Beese, D., Paaßen, B., Pütz, O., Wiedemann, G., & Eger, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.337 · Cited by 9 · see citing works
Yakura, H. (2024) · Journal of Intelligence · doi:10.3390/jintelligence12070070 · Cited by 6 · see citing works
The Gricean Maxims in NLP - A Survey
Krause, L., & Vossen, P. T. J. M. (2024) · International Conference on Natural Language Generation · doi:10.18653/v1/2024.inlg-main.39 · Cited by 6 · see citing works
Opinion paper: generative AI and the future of scientometrics
Lepori, B., Andersen, J. P., & Donnay, K. (2026) · Scientometrics · doi:10.1007/s11192-026-05667-1 · Cited by 5 · see citing works
Assessing the Potential of LLM-assisted Annotation for Corpus Pragmatics: The Case of Humor
Bianco, A., Brocca, N., & Garassino, D. (2026) · Corpus Pragmatics · doi:10.1007/s41701-026-00235-7 · Cited by 5 · see citing works
Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning
Wu, S., Yang, S., Chen, Z., & Su, Q. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1258 · Cited by 5 · see citing works
Александрович, Ш. В., Şimşek, H. B., Kudriashov, S., Randautsova, R., & Shevela, S. A. (2025) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2025-23-345-360 · Cited by 5 · see citing works
Pragmatic Inference Chain (PIC) Improving LLMs’ Reasoning of Authentic Implicit Toxic Language
Chen, X., & Wang, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.296 · Cited by 5 · see citing works
VAGUE: Visual Contexts Clarify Ambiguous Expressions
Nam, H., Ahn, J., Ka, K., Chung, J., & Yu, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00151 · Cited by 5 · see citing works
How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction
Sadlier-Brown, E., Lou, M., Silfverberg, M., & Kam, C. L. H. (2024) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2024.cmcl-1.20 · Cited by 4 · see citing works
Szoke, D. R., Hutzler, I., Liu, J., Addante, S., Akhtar, Z., Smith, D. L., Dickens, K., Small, C., Pridgen, S., & Held, P. (2026) · JMIR Mental Health · doi:10.2196/91367 · Cited by 4 · see citing works
ChatGPT for President! Presupposed content in politicians versus GPT-generated texts
Garassino, D., Brocca, N., & Masia, V. (2025) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2025.100156 · Cited by 2 · see citing works
Solidjonov, D. (2026) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-026-00200-5 · Cited by 3 · see citing works
Pragmatic inference and mapping for conversational implicature
Hu, J., Liu, Y., Ma, L., Zhang, W.-N., & Liu, T. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02741-7 · Cited by 2 · see citing works
VIBE: Can a VLM Read the Room?
Chakraborty, T., Caplan, E., & Goldwasser, D. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1252 · Cited by 2 · see citing works
ExpressivityBench: Can LLMs Communicate Implicitly?
Tint, J., Sagar, S., Taparia, A., Raines, K., Pathiraja, B., Liu, C., & Senanayake, R. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.235 · Cited by 2 · see citing works
Relevant answers to polar questions
Hawkins, R. D., Tsvilodub, P., Bergey, C. A., Goodman, N. D., & Franke, M. (2025) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2023.0505 · Cited by 1 · see citing works
Comparing generative AI with native speakers in terms of request expressions in Japanese
Chen, Y., Yue, P., & Davidge, H. (2025) · Discover Education · doi:10.1007/s44217-025-00920-w · Cited by 1 · see citing works
Pragmatics in Human–Computer Interaction: A Systematic Literature Review
Wang, W., & Liu, X. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2635678 · Cited by 1 · see citing works
Krykoniuk, K., Hopkin-King, C., & Roberts, S. G. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-07900-x · Cited by 1 · see citing works
AI as a humanized subject (mitigation strategies of artificial intelligence)
Николаевич, С. Л. (2025) · Communication Studies · doi:10.24147/2413-6182.2025.12(4).789-804 · Cited by 1 · see citing works
Humaira, S. N., Fatimah, Z., Situmorang, C. N., Aila, S., Idham, R. A. S., & Putri, D. M. (2026) · Sintaks Jurnal Bahasa & Sastra Indonesia · doi:10.57251/sin.v6i2.2131 · Cited by 1 · see citing works
Annotation of Greeting and Introduction in Russian Spoken Dialogues Using ISO 24617-2:2020
Klokova, K. S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-97019-1_3
Understand the Implication: Learning to Think for Pragmatic Understanding
Sravanthi, S. L., Maharaj, K., Gunnu, S., Mishra, A., & Bhattacharyya, P. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1218
Self-citationDoes ChatGPT get it? LLM-driven annotation of HumorPreprint
Bianco, A., Brocca, N., & Garassino, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5500673
Pravica biti slišan: izzivi in projekti na področju sodnega in skupnostnega tolmačenja in prevajanja
Grčar, A. S., Vintar, Š., Južnič, T. M., Pokorn, N. K., Orlić, N., Burjan, L., Kulovec, M., Božič, T. D., Greif, M., Samobor, A., et al. (2025) · Založba Univerze v Ljubljani (University of Ljubljana Press) eBooks · doi:10.4312/9789612977221
Linguistic Theories Coincide with Misformalization in Temporal Logic
Gordon, C. S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00329
Towards a Usage-Based Pedagogy for Second Language Learning with Robots
Verhelst, E., & Belpaeme, T. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3776734.3794484
Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
Yusoff, M. A. G. B., Tan, J., Chen, B., Liu, G. G., & Chen, X. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1448
Navigating Developers' Quagmire: LLM-Enabled Privacy Compliance Analysis for SDK Integrations
Hu, Z., & Wang, X. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00126
Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilitie
Kankowski, F., Solstad, T., Zarrieß, S., & Bott, O. (2026) · Dialogue & Discourse · doi:10.5210/dad.2026.201
Shin, G., Kwon, N., & Mun, S. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-08299-1
Benchmarking Pragmatic Reasoning of Large Language Models in the Czech Language
Karpizenkov, Y., Horák, A., & Lacina, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_10
Всеволодович, Н. Н., & Александрович, Ш. В. (2025) · Communication Studies · doi:10.24147/2413-6182.2025.12(4).881-896
Implicature in Interaction: Implicature-Aware Prompting Improves User Evaluations of LLM Responses
Hota, A., & Jokinen, J. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2720982
Les grands modèles de langue : bilan et perspectives pour la linguistique
Poibeau, T. (2026) · Langages · doi:10.3917/lang.243.0073
Firt, E., & Nir, B. (2026) · Pragmatics and Society · doi:10.1075/ps.25161.fir
Sadeq, S. A. R. (2026) · مجلة الباحث · doi:10.63797/6qfx3685
Pragmatic competence of large language models: detection and interpretation of implicatures
Александрович, Ш. В. (2026) · Communication Studies · doi:10.24147/2413-6182.2026.13(2).425-438
Badara, A., Rifa’ie, M., Yusoh, R., & Salama, I. A.-K. (2026) · Rainbow Journal of Literature Linguistics and Cultural Studies · doi:10.15294/rainbow.v15i3.65667
- Authors
- Sravanthi, S., Doshi, M., Tankala, P., Murthy, R., Dabre, R., & Bhattacharyya, P.
- Affiliations
- Indian Institute of Technology Bombay (IN)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 12075-12097
- OpenAlex
- W4402670074
- Cited by
- 46
- References
- 4
- Citations by year
- 2026: 21 · 2025: 16 · 2024: 9
- FWCI
- 11.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- pragmatics, large language models, pragmatic reasoning, implicature, presupposition, reference, deixis, benchmark dataset, multiple-choice question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
