When Your Language Model Cannot Even Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle
Open accessAbstract
The increasing interest in probing the linguistic capabilities of large language models (LLMs) has long reached the area of semantics and pragmatics, including the phenomenon of presuppositions.In this study, we investigate a phenomenon that, however, has not yet been investigated, i.e., the phenomenon of antipresupposition and the principle that accounts for it, the Maximize Presupposition!principle (MP!).Through an experimental investigation using psycholinguistic data and four open-source BERT model variants, we explore how language models handle different antipresuppositions and whether they apply the MP! principle in their predictions.Further, we examine whether fine-tuning with Natural Language Inference data impacts adherence to the MP! principle.Our findings reveal that LLMs tend to replicate context-based n-grams rather than follow the MP! principle, with fine-tuning not enhancing their adherence.Notably, our results further indicate a striking difficulty of LLMs to correctly predict determiners, in relatively simple linguistic contexts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These findings, together with those by Kabbara & Cheung (2022), Sieker and Zarrieß (2023) and Sravanthi et al. (2024), suggest that LLMs tend to rely on superficial cues when processing inferential phenomena.
“these findings, together with those by Kabbara & Cheung (2022), Sieker and Zarrie ß (2023) and Sravanthi et al. (2024), suggest that LLMs tend to rely on superficial cues when processing inferential phenomena.”
From ChatGPT for President! Presupposed content in politicians versus GPT-generated texts (2025) · cites this paper as Sieker and Zarrie ß (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“Therefore, while we do not foresee any ethical concerns specific to this paper, the broader ethical concerns pertaining to LLMs remain of relevance to our research (cf., e.g., Bender et al. (2021).”
BackgroundChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
A fine-grained comparison of pragmatic language understanding in humans and language models
Hu, J., Floyd, S., Jouravlev, O., Fedorenko, E., & Gibson, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.230 · Cited by 78 · see citing works
Flagged first, then by citations
Research works printed but not linked: 28
- Nadine Bade and Florian Schwarz. 2021. New data on the competition between definites and indefinites. Experiments in Linguistic Meaning, 1(0):15-26.
- David I. Beaver, Bart Geurts, and Kristie Denlinger. 2021. Presupposition. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Spring 2021 edition. Metaphysics Research Lab, Stanford University.
- Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72. · DOI
- David Blunier. 2022. Antipresuppositions, logophors and shifted indexicality. In Proceedings of the 23rd Amsterdam Colloquium., pages 45-54.
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics. · DOI
- Yan Cong. 2022. Psycholinguistic diagnosis of lan- guage models' commonsense reasoning. In Proceed- ings of the First Workshop on Commonsense Repre- sentation and Reasoning (CSRR 2022), pages 17-22, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
- Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The CommitmentBank: Inves- tigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23.
- Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48. · DOI · Paper page
- Thibault Févry, Livio Baldini Soares, Nicholas FitzGer- ald, Eunsol Choi, and Tom Kwiatkowski. 2020. En- tities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4937-4951, Online. Association for Computational Linguistics. · DOI
- H. P. Grice. 1975. Logic and Conversation, pages 41 - 58. Brill, Leiden, Niederlande. · DOI · Paper page
- Irene Heim. 1991. Artikel und Definitheit, pages 487- 535. De Gruyter Mouton, Berlin • New York. · DOI
- Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language infer- ence models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 8690-8705, Online. Association for Computational Linguistics. · DOI
- Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Do you know that florence is packed with visitors? evaluating state-of-the-art models of speaker commit- ment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4208-4213, Florence, Italy. Association for Compu- tational Linguistics. · DOI
- Jad Kabbara and Jackie Chi Kit Cheung. 2022. Inves- tigating the performance of transformer-based NLI models on presuppositional inferences. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 779-785, Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 252-262, New Orleans, Louisiana. As- sociation for Computational Linguistics. · DOI
- Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bow- man, and Ellie Pavlick. 2019. Probing what differ- ent NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 235-249, Minneapolis, Min- nesota. Association for Computational Linguistics. · DOI
- Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist in- vented the lightbulb? presupposition verification for question-answering. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3932-3945, Online. Association for Computational Linguistics. · DOI
- Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bow- man, and Tal Linzen. 2021. NOPE: A corpus of naturally-occurring presuppositions in English. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 349-366, Online. Association for Computational Linguistics. · DOI
- Orin Percus. 2006. Antipresuppositions. Theoretical and Empirical Studies of Reference and Anaphora: Toward the establishment of generative grammar as an empirical science,, pages 52-73.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of Plausible Alter- natives: An Evaluation of Commonsense Causal Rea- soning. In AAAI Spring Symposium on Logical For- malizations of Commonsense Reasoning, Stanford University.
- Sebastian Schuster and Tal Linzen. 2022. When a sentence does not introduce a discourse entity, transformer-based models still sometimes refer to it. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 969-982, Seattle, United States. Association for Computational Linguistics. · DOI
- Judith Sieker, Oliver Bott, Torgrim Solstad, and Sina Zarrieß. 2023. Beyond the bias: Unveiling the qual- ity of implicit causality prompt continuations in lan- guage models. In Proceedings of the 16th Inter- national Natural Language Generation Conference, pages 206-220, Prague, Czechia. Association for Computational Linguistics.
- Robert Stalnaker. 1973. Presuppositions. Journal of Philosophical Logic, 2(4):447-457. · DOI
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32.
- Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguis- tic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377- 392. · DOI · Paper page
- Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442-6454, On- line. Association for Computational Linguistics. · DOI
- Dongjie Yang, Zhuosheng Zhang, and Hai Zhao. 2023. Learning better masking for better language model pre-training. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7255-7267, Toronto, Canada. Association for Computational Linguistics. · DOI
- Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2023. CREPE: Open-domain ques- tion answering with false presuppositions. In Pro- ceedings of the 61st Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 10457-10480, Toronto, Canada. As- sociation for Computational Linguistics. · DOI
Cited by
Sorted by citations
ChatGPT for President! Presupposed content in politicians versus GPT-generated texts
Garassino, D., Brocca, N., & Masia, V. (2025) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2025.100156 · Cited by 2 · see citing works
- Authors
- Sieker, J., & Zarrieß, S.
- Affiliations
- Bielefeld University (DE)
- Venue
- Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
- Published
- 2023
- Pages
- 180-198
- OpenAlex
- W4389518362
- Cited by
- 2
- References
- 7
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 4.57
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) (publishedVersion, cc-by)
- Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion, cc-by-nc)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- presupposition, determiners, large language models, BERT, semantics-pragmatics interface
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
