Language models are not naysayers: an analysis of language models on negation benchmarks
Open accessAbstract
Negation has been shown to be a major bottleneck for masked language models, such as BERT. However, whether this finding still holds for larger-sized auto-regressive language models (``LLMs'') has not been studied comprehensively. With the ever-increasing volume of research and applications of LLMs, we take a step back to evaluate the ability of current-generation LLMs to handle negation, a fundamental linguistic phenomenon that is central to language understanding. We evaluate different LLMs -- including the open-source GPT-neo, GPT-3, and InstructGPT -- against a wide range of negation benchmarks. Through systematic experimentation with varying model sizes and prompts, we show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs often struggle with constraints involving numbers, negations, or long instructions with large number of constraints.
“A qualitative examination of responses with unsatisfied constraints suggests that LLMs often struggle with constraints involving numbers, negations, or long instructions with large number of constraints , which is consistent with findings from previous works (Sui et al., 2024; García-Ferrero et al., 2023; Truong et al., 2023; Jiang et al., 2024).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Truong et al., 2023
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,595 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W. B., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,121 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002 · see citing works
Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7347 · Cited by 881 · see citing works
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Ettinger, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00298 · Cited by 511 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Kassner, N., & Schütze, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.698 · Cited by 259 · see citing works
PromptBERT: Improving BERT Sentence Embeddings with Prompts
Jiang, T., Jiao, J., Huang, S., Zhang, Z., Wang, D., Zhuang, F., Wei, F., Huang, H., Deng, D., & Zhang, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.603 · Cited by 173 · see citing works
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
On the Predictive Power of Neural Language Models for Human Real-Time Comprehension BehaviorPreprint
Wilcox, E., Gauthier, J., Hu, J., Qian, P., & Lévy, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01912 · Cited by 125 · see citing works
Demystifying Prompts in Language Models via Perplexity Estimation
Gonen, H., Iyer, S., Blevins, T., Smith, N., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.679 · Cited by 109 · see citing works
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs
Warstadt, A., Cao, Y. H., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1286 · Cited by 100 · see citing works
Consistency Analysis of ChatGPT
Jang, M., & Lukasiewicz, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.991 · Cited by 86 · see citing works
An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
Hossain, M. M., Kovatchev, V., Dutta, P., Kao, T. H., Wei, E., & Blanco, E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.732 · Cited by 66 · see citing works
SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples
Wang, H., & Dou, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-4752-2_35 · Cited by 59 · see citing works
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
Geiger, A., Richardson, K., & Potts, C. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.16 · Cited by 52 · see citing works
Understanding by Understanding Not: Modeling Negation in Language Models
Hosseini, A., Reddy, S., Bahdanau, D., Hjelm, R. D., Sordoni, A., & Courville, A. C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.102 · Cited by 47 · see citing works
Leveraging Large Language Models for Multiple Choice Question AnsweringPreprint
Robinson, J. A., Rytting, C. M., & Wingate, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.12353 · Cited by 40 · see citing works
Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA
Staliūnaitė, I., & Iacobacci, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.573 · Cited by 36 · see citing works
An Analysis of Negation in Natural Language Understanding Corpora
Hossain, M. M., Chinnappa, D., & Blanco, E. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.81 · Cited by 34 · see citing works
Can Large Language Models Truly Understand Prompts? A Case Study with Negated PromptsPreprint
Jang, J., Ye, S., & Seo, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.12711 · Cited by 30 · see citing works
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
Ravichander, A., Gardner, M., & Marasović, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.598 · Cited by 26 · see citing works
Inverse Scaling Can Become U-Shaped
Wei, J., Kim, N., Tay, Y., & Le, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.963 · Cited by 25 · see citing works
Context Matters: A Pragmatic Study of PLMs’ Negation Understanding
Gubelmann, R., & Handschuh, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.315 · Cited by 20 · see citing works
Improving negation detection with negation-focused pre-training
Truong, T. H., Baldwin, T., Cohn, T., & Verspoor, K. M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.309 · Cited by 19 · see citing works
Self-citationLanguage Models Use Monotonicity to Assess NPI Licensing
Jumelet, J., Denić, M., Szymanik, J., Hupkes, D., & Steinert‐Threlkeld, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.439 · Cited by 17 · see citing works
Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
Jang, M., Mtumbuka, F., & Lukasiewicz, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.156 · Cited by 6 · see citing works
Leveraging Affirmative Interpretations from Negation Improves Natural Language Understanding
Hossain, M. M., & Blanco, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.393 · Cited by 5 · see citing works
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
Truong, T. H., Otmakhova, Y., Baldwin, T., Cohn, T., Lau, J. H., & Verspoor, K. M. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.65 · Cited by 5 · see citing works
Self-citationLife after BERT: What do Other Muppets Understand about Language?
Lialin, V., Zhao, K., Shivagunde, N., & Rumshisky, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.227 · Cited by 4 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Cited by
Sorted by citations
Can LLMs Replace Manual Annotation of Software Engineering Artifacts?
Ahmed, T., Dévanbu, P., Treude, C., & Pradel, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00086 · Cited by 41 · see citing works
Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models
Reddig, J. M., Arora, A., & MacLellan, C. J. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00505-6 · Cited by 38 · see citing works
Meaning in Hyperspace: Word Embeddings as Tools for Cultural Measurement
Boutyline, A., & Arseniev‐Koehler, A. (2025) · Annual Review of Sociology · doi:10.1146/annurev-soc-090324-024027 · Cited by 23 · see citing works
Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics
Zhang, Y., Gibson, E. A. F., & Davis, F. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.1 · Cited by 5 · see citing works
A Large Language Model to Detect Negated Expressions in Radiology Reports
Su, Y., Babore, Y., & Kahn, C. E. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01274-9 · Cited by 4 · see citing works
Trustworthy Formal Natural Language Specifications
Gordon, C. S., & Matskevich, S. (2023) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3622758.3622890 · Cited by 4 · see citing works
Challenges and Opportunities in Text Generation Explainability
Amara, K., Sevastjanova, R., & El‐Assady, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63787-2_13 · Cited by 4 · see citing works
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3 · see citing works
“…measure under the guidelines from Kamoi et al. (2024), since decomposition is relatively straightforward and could be handled by a dedicated LLM; and (3) Unrealistic Ablations , which rely on ideal Critic and Decomposer models, useful for upper bound estimation but with limited generalization.”
Rosa, A. L., Verdoux, M., Riebler, P., Lolli, I., Daniel, C., Tannier, X., Atallah, S., BAUJAT, B., & Kempf, E. (2025) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2025.100151 · Cited by 2 · see citing works
Bennett, C. L., Kane, S. K., & Harrington, C. N. (2025) · Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3757887.3763012 · Cited by 3 · see citing works
Shin, C., Eom, D., Lee, S. M., Park, J. E., Kim, K., & Lee, K. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-16213-z · Cited by 3 · see citing works
NeIn: Telling What You Don't Want
Bui, N.-T., Hoang, D.-H., Trinh, Q.-H., Tran, M., Nguyen, T. Q., & Gauch, S. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00293 · Cited by 2 · see citing works
Cosine capital: Large language models and the embedding of all things
Brunila, M. (2025) · Big Data & Society · doi:10.1177/20539517251386055 · Cited by 2 · see citing works
Norel, R., Gewandter, J. S., Zhang, Z., Tahsin, A., Abdallah, C. G., Markman, J. D., Duan, Z., Cecchi, G., & Geha, P. (2025) · JMIR Human Factors · doi:10.2196/80269 · Cited by 2 · see citing works
Improving negative rejection ability in language models: A review of fine-tuned LLMs, RAG, and RAFT
Bowen, L., Zhuoqing, Z., Hengyu, Z., Xubin, S., & Ihnaini, B. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00334-6 · Cited by 2 · see citing works
Benchmarking Gaslighting Negation Attacks Against Reasoning Models
Zhu, B., Yin, H., Chen, J., & Jiang, Y.-G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6960-1_14 · Cited by 3 · see citing works
Modelling language using large language models
Grindrod, J. (2026) · Philosophical Studies · doi:10.1007/s11098-026-02479-0 · Cited by 3 · see citing works
Truong, T. H., Foster, E., Fazio, T. N., Holper, S., & Verspoor, K. M. (2025) · medRxiv · doi:10.1101/2025.08.02.25332570 · Cited by 2 · see citing works
Self-citationAutomatic analysis of negation cues and scopes for medical texts in French using language models
Sadoune, S., Richard, A., Talbot, F., Guyet, T., Boussel, L., & Berry, H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110795 · Cited by 1 · see citing works
Atılan, A. U., & Çetin, N. (2026) · Cutaneous and Ocular Toxicology · doi:10.1080/15569527.2026.2733364 · Cited by 1 · see citing works
AI cannot replace human participants in behavioral research
Hoover, J., Cooke, A. D. J., Tomaino, G., & Lee, S. H. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2611363123 · Cited by 1 · see citing works
Hazard-aware adaptations bridge the generalization gap in large language models: a nationwide studyPreprint
Wu, J. T.-Y., Conover, S., Su, C. C., Corrigan, J., Culnan, J. M., Liu, Y., Kelley, M., Do, N., Arya, S., Harris, A. H. S., et al. (2025) · medRxiv · doi:10.1101/2025.02.14.25322312 · Cited by 1 · see citing works
Not Wacky vs. Definitely Wacky: A Study of Scalar Adverbs in Pretrained Language Models
Lorge, I., & Pierrehumbert, J. B. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.23
Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning
Nakamura, H., Okada, M., & Taniguchi, T. (2024) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2024.3399663
Negation Scope Conversion for Unifying Negation-Annotated Datasets
Yoshida, A., Kato, Y., & Matsubara, S. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.300
Reasoning Capabilities and Invariability of Large Language Models
Raganato, A., Peñaloza, R., Viviani, M., & Pasi, G. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00025
Animate, or Inanimate, That Is the Question for Large Language Models
Pucci, G., Zanzotto, F. M., & Ranaldi, L. (2025) · Information · doi:10.3390/info16060493
On the compatibility of generative AI and generative linguistics
Portelance, E., & Jasbi, M. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00861-2 · Cited by 5 · see citing works
Interpreting Negation in GPT-2: Layer-and Head-Level Causal Analysis
Mofael, A. A., Kuhn, L. M., Alkadi, G., & Yang, K.-P. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393646
Linguistic Polarity and Decision Architecture in LLM-Based Abstract Screening for Systematic Reviews
Behrouzian, A. M., Meleti, M., Colangelo, M. T., Calciolari, E., & Galli, C. (2026) · Information · doi:10.3390/info17050449
Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Models
Zhu, B., Gui, Y., Qi, H., Chen, J., Ngo, C., & Lim, E. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810624 · Cited by 1 · see citing works
Owers, J., Simpson, E., & Lewis, M. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.781
Logical Misconceptions, Pragmatic Insufficiencies in LLMs and How to Fix Them
Asher, N. M., & Bhar, S. (2026) · Topoi · doi:10.1007/s11245-026-10459-6
Vázquez, M. Y. L., Pérez, A. M., & Gutiérrez, B. (2026) · Research Square · doi:10.21203/rs.3.rs-11167683/v1
- Authors
- Truong, T. H., Baldwin, T., Verspoor, K., & Cohn, T.
- Affiliations
- The University of Melbourne (AU)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 101-114
- arXiv
- 2306.08189
- OpenAlex
- W4385718075
- Cited by
- 34
- References
- 47
- Citations by year
- 2026: 10 · 2025: 16 · 2024: 4 · 2023: 4
- FWCI
- 6.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Joint Conference on Lexical and Computational Semantics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Australian Government (LE170100200)
- University of Melbourne (LE170100200)
- Australian Research Council (ICI70200030, LE170100200)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- negation, large language models, word meaning, prompt sensitivity, model size, GPT-3, masked language models, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
