The DOI matches its Crossref record; 90 of the 93 references it cites were found in a registry or an index.
Measuring and Improving Consistency in Pretrained Language Models
Open accessAbstract
Consistency of a model—that is, the invariance of its behavior under meaning-preserving alternations in its input—is a highly desirable property in natural language processing. In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To this end, we create ParaRel🤘, a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. Using ParaRel🤘, we show that the consistency of all PLMs we experiment with is poor— though with high variance between relations. Our analysis of the representational spaces of PLMs suggests that they have a poor structure and are currently not suitable for representing knowledge robustly. Finally, we propose a method for improving model consistency and experimentally demonstrate its effectiveness.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, LLMs can struggle on commonsense knowledge benchmarks [158], especially once low-level statistical cues are controlled for [9].”
From Dissociating language and thought in large language models (2024) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2008) · Journal of Machine Learning Research · Cited by 33,051
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Flagged first, then by citations
3 of the 93 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Elsahar, 2018, T-rex: A large scale alignment of natural language with knowledge base triples
- Hewitt, 2019, A structural probe for finding syntax in word representations
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
Cited by
Sorted by citations
Items 101–200 of 234
Metamorphic Testing of Large Language Models for Natural Language Processing
Cho, S. B., Ruberto, S., & Terragni, V. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00025 · Cited by 22
The Effect of Normalization for Bi-directional Amharic-English Neural Machine Translation
Belay, T. D., Tonja, A. L., Kolesnikova, O., Yimam, S. M., Ayele, A. A., Haile, S. B., Sidorov, G., & Gelbukh, A. F. (2022) · International Conference on Information and Communication Technology for Development for Africa (ICT4DA) · doi:10.1109/ict4da56482.2022.9971385 · Cited by 9
Dutta, S., & Chakraborty, T. (2023) · Communications of the ACM · doi:10.1145/3616863 · Cited by 12
Accurate, yet inconsistent? Consistency Analysis on Language Understanding ModelsPreprint
Jang, M., Kwon, D. S., & Lukasiewicz, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06665 · Cited by 14
Understanding the Impact of Using Generative AI Tools in a Database Course
Osorio, V. R., Bernuy, A. Z., Simion, B., & Liut, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701785 · Cited by 8
Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency
Yang, Y., Wang, Z., Xia, Y., Wei, Z., Ding, H., Piskac, R., Chen, H., & Li, J. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725271 · Cited by 8
Learning with Instance Bundles for Reading Comprehension
Dua, D., Dasigi, P., Singh, S., & Gardner, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.584 · Cited by 9
Synonyms, Antonyms and Factual Knowledge in BERT Heads
Serina, L., Putelli, L., Gerevini, A., & Serina, I. (2023) · Future Internet · doi:10.3390/fi15070230 · Cited by 9
Self-Consistent Narrative Prompts on Abductive Natural Language Inference
Chan, C., Liu, X., Chan, T. H., Cheng, J., Song, Y., Wong, G. Y., & See, S. C. W. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.67 · Cited by 15
BERTnesia: Investigating the capture and forgetting of knowledge in BERTPreprint
Wallat, J., Singh, J., & Anand, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09313 · Cited by 18
S2AF : An action framework to self-check the Understanding Self-Consistency of Large Language Models
Shao, H., Wang, F., & Xie, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107365 · Cited by 7
Energy-Aware Multilingual Evaluation of Large Language Models
de Zarzà, I., Liz, M., de Curtò, J., & Calafate, C. T. (2026) · Electronics · doi:10.3390/electronics15071395 · Cited by 7
Lost in Context? On the Sense-Wise Variance of Contextualized Word Embeddings
Wang, Y., & Zhang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3337643 · Cited by 9
Translating Natural Language Queries to SQL Using the T5 Model
Wong, A., Pham, L., Lee, Y., Chan, S. T., Sadaya, R., Khmelevsky, Y., Clement, M., Cheng, F. W. Y., Mahony, J., & Ferri, M. (2024) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon61195.2024.10553509 · Cited by 8
Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering
Chaturvedi, A., Bhar, S., Saha, S., Garain, U., & Asher, N. M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00493 · Cited by 8
Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views
Margatina, K., Wang, S., Vyas, Y., John, N. A., Benajiba, Y., & Ballesteros, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.211 · Cited by 9
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models
Hagström, L., Saynova, D., Norlund, T., Johansson, M., & Johansson, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.332 · Cited by 6
Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions
Chen, M., Iyer, A., & Pavel, A. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746393 · Cited by 6
ICR: A framework for resolving knowledge conflicts in retrieval-augmented generation
Xiong, Y., Chen, Y., & Zhang, H. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132139 · Cited by 6
Metamorphic Testing for Semantic Invariance in Large Language Models
de Curtò, J., & de Zarzà, I. (2025) · IEEE Access · doi:10.1109/access.2025.3646270 · Cited by 11
Cao, P., Chen, Y., Jin, Z., Chen, Y., Liu, K., & Zhao, J. (2026) · Artificial Intelligence · doi:10.1016/j.artint.2026.104490 · Cited by 16
Can AI Rely on the Systematicity of Truth? The Challenge of Modelling Normative Domains
Queloz, M. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00864-x · Cited by 7
Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
Jang, M., Mtumbuka, F., & Lukasiewicz, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.156 · Cited by 6
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
Monea, G., Peyrard, M., Josifoski, M., Chaudhary, V., Eisner, J. M., Kıcıman, E., Palangi, H., Patra, B., & West, R. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.369 · Cited by 19
Political Biases of ChatGPT in Different LanguagesPreprint
Retzlaff, N. (2024) · Preprints.org · doi:10.20944/preprints202406.1224.v1 · Cited by 8
Large Language Models for in Situ Knowledge Documentation and Access With Augmented Reality
Izquierdo-Domenech, J., Linares-Pellicer, J., & Ferri-Molla, I. (2023) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2023.09.002 · Cited by 8
Leveraging Large Language Models for Structure Learning in Prompted Weak Supervision
Su, J., Yu, P., Zhang, J., & Bach, S. H. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386190 · Cited by 5
How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?
Srikanth, N. P., Carpuat, M. J., & Rudinger, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00692 · Cited by 5
Visually Grounded Commonsense Knowledge Acquisition
Yao, Y., Yu, T., Zhang, A., Li, M., Xie, R., Weber, C., Liu, Z., Zheng, H.-T., Wermter, S., Chua, T., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25809 · Cited by 5
Evaluating the Knowledge Base Completion Potential of GPT
Veseli, B., Razniewski, S., Kalo, J.-C., & Weikum, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.426 · Cited by 9
TempLM: Distilling Language Models into Template-Based Generators
Zhang, T., Lee, M., Li, X. L., Shen, E., & Hashimoto, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.124 · Cited by 6
SQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentations
Leung, A., Cheng, R., Wu, J., Nichols, J., & Barik, T. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747672 · Cited by 10
Leveraging generative AI to transform statistical analysis plan authoring in clinical trials
Landman, R., Emir, B., Zhang, R., Manousaridis, G., Lee, Y. C., Jiang, L., Siu, G., Dai, F., Bell, G., Thiry, A., et al. (2026) · Clinical Trials · doi:10.1177/17407745261422365 · Cited by 4
Sakarvadia, M., Ajith, A., Khan, A., Grzenda, D., Hudson, N., Bauer, A., Chard, K., & Foster, I. T. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.26 · Cited by 9
A comprehensive evaluation of semantic relation knowledge of pretrained language models and humans
Cao, Z., Yamada, H. S., Teufel, S., & Tokunaga, T. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09858-9 · Cited by 4
Jang, M., Majumder, B. P., McAuley, J., Lukasiewicz, T., & Camburu, O.-M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.47 · Cited by 4
Byun, H., Lee, D., Jung, M., & Jang, B. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02440-y · Cited by 3
Can BERT eat RuCoLA? Topological Data Analysis to Explain
Proskurina, I., Artemova, E., & Piontkovskaya, I. (2023) · Workshop on Slavic Natural Language Processing (SlavicNLP) · doi:10.18653/v1/2023.bsnlp-1.15 · Cited by 3
Consistent Text Categorization using Data Augmentation in e-Commerce
Avigdor-Elgrabli, N., Horowitz, G., Raviv, A., & Daye, S. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.30 · Cited by 3
Seeking Clozure: Robust Hypernym extraction from BERT with Anchored Prompts
Liu, C., Cohn, T., & Frermann, L. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.18 · Cited by 3
Knowledge Base Completion for Long-Tail Entities
Chen, L., Razniewski, S., & Weikum, G. (2023) · doi:10.18653/v1/2023.matching-1.8 · Cited by 6
Generative AI for Research Data Processing: Lessons Learnt From Three Use Cases
Mitra, M., de Vos, M. G., Cortinovis, N., & Ometto, D. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678704 · Cited by 4
Knowledge Acquired by Foundation Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_4 · Cited by 3
Measuring Consistency in Text-based Financial Forecasting Models
Yang, L., Ma, Y., & Zhang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.769 · Cited by 5
Lee, J., Sakaguchi, K., & Bak, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.528 · Cited by 2
Enhancing the RAG Pipeline Through Advanced Optimization Techniques
Ilyas, Q. M., & Aziz, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-6255-6.ch003 · Cited by 2
Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence
Queloz, M. (2025) · Minds and Machines · doi:10.1007/s11023-025-09738-9 · Cited by 3
Fu, Z., Song, L., Ding, S. H. H., Alaca, F., & Acharya, S. (2025) · ACM Transactions on Privacy and Security · doi:10.1145/3771542 · Cited by 2
VeriFastScore: Speeding up long-form factuality evaluation
Rajendhran, R., Zadeh, A., Sarte, M., Li, C., & Iyyer, M. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.491 · Cited by 7
Dai, Y., Zhang, H., Wang, J., & Liao, J. (2026) · Computer Communications · doi:10.1016/j.comcom.2026.108428 · Cited by 2
Shao, H., Wang, F., Zhang, S., & Xie, Z. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104701 · Cited by 2
Kerimoğlu, C. (2026) · Dil Araştırmaları · doi:10.54316/dilarastirmalari.1879242 · Cited by 2
Reusens, M., Goethals, S., Calders, T., & Martens, D. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133279 · Cited by 2
Explainable AI for Deep Visual Recognition: Evaluation, Methods, and Open Challenges
Alharbi, K. N. (2026) · Electronics · doi:10.3390/electronics15143222 · Cited by 2
Made of Steel? Learning Plausible Materials for Components in the Vehicle Repair Domain
Eichel, A., Schlipf, H., & Walde, S. S. I. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.104 · Cited by 3
Who Are All The Stochastic Parrots Imitating? They Should Tell Us!
Shaier, S., Hunter, L., & Kann, K. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.13 · Cited by 2
Continuous Entailment Patterns for Lexical Inference in Context
Schmitt, M., & Schütze, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.556 · Cited by 2
Self-citationBetz, G., & Richardson, K. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.900943 · Cited by 2
Learning to Improve Out-of-Distribution Generalization via Self-Adaptive Language Masking
Jiang, S., Pan, Y., Chen, Q., Xiang, Y., & Wu, X. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3394774 · Cited by 2
Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency
Shen, L., Tan, W., Zheng, B., & Khashabi, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.523 · Cited by 3
Do as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1
“Furthermore, the parameterized knowledge can become inaccurate or outdated [66, 67], potentially leading to errors and hallucinations [68–70].”
ConvoMap: Interactive Visualizations for Exploring Complex Conversations in Multi-Agent Systems
Zhang, G., Bursztyn, V. S., Chan, G. Y.-Y., Guo, S., Koh, E., Oney, S., & Hoffswell, J. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00043 · Cited by 2
Wasilewski, A., & Kołaczek, G. (2026) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.70562 · Cited by 1
On the Underspecification of Situations in Open-domain Conversational Datasets
Otani, N., Araki, J., Kim, H., & Hovy, E. H. (2023) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2023.nlp4convai-1.2 · Cited by 2
Self-citationUnveiling the Evolution and Impact of Pretrained Language Models in Natural Language Processing
Bian, H. (2024) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3672758.3672854 · Cited by 1
Vo, T., Luu, S. T., & Nguyen, L.-M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02345-1 · Cited by 1
Robots That Perform Norm-Based Reference Resolution
Abrams, M., Thierauf, C., & Scheutz, M. J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-3519-1_11 · Cited by 1
Chen, Y.-C., & Huang, H. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34534 · Cited by 1
AN APPROACH TO ASSESSING THE QUALITY OF PENTEST SCENARIO GENERATION USING LARGE LANGUAGE MODELS
Konev, A., & Игоревна, П. Т. (2025) · Voprosy kiberbezopasnosti · doi:10.21681/2311-3456-2025-6-147-157 · Cited by 1
Applications of multi-agent systems in computational materials science
Ying, W., Zhong-Yi, L., & Zefeng, G. A. O. (2026) · Acta Physica Sinica · doi:10.7498/aps.75.20251687 · Cited by 1
A multi-scenario approach to continuously learn and understand norm violations
dos Santos, T. F., Osman, N., & Schorlemmer, M. (2023) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-023-09619-4 · Cited by 1
Nayak, I., Kanagaraj, K., Shrimali, M., Kumar, D., Ghamande, M. V., & Kumar, C. S. (2023) · International Conference on Automation, Computing and Renewable Systems (ICACRS) · doi:10.1109/icacrs58579.2023.10404259 · Cited by 1
On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation
Patel, G., Choshen, L., & Abend, O. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.14 · Cited by 1
Unnatural language processing: How do language models handle machine-generated prompts?
Kervadec, C., Franzon, F., & Baroni, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.959 · Cited by 1
Cao, B., Tang, Q., Lin, H., Han, X., & Sun, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.143 · Cited by 2
Continuous Entailment Patterns for Lexical Inference in ContextPreprint
Schmitt, M., & Schütze, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03695 · Cited by 2
Self-citationAI Behavioral SciencePreprint
Jackson, M. O., Mei, Q., Wang, S., Xie, Y., Yuan, W., Benzell, S. G., Brynjolfsson, E., Evans, J. A., Jabarian, B., Kleinberg, J. M., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5395006 · Cited by 2
Interviewer-Candidate Role Play: Towards Developing Real-World NLP SystemsPreprint
Varshney, N., Mishra, S. R., & Baral, C. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00315
Accelerating Multi-Exit BERT Inference via Curriculum Learning and Knowledge Distillation
Gu, S., Luo, X., Wang, X., & Guo, Y. (2023) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194023500018
Probing Power by Prompting: Harnessing Pre-trained Language Models for Power Connotation Framing
Khanehzar, S., Cohn, T., Mikołajczak, G., & Frermann, L. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.61
How Much Consistency Is Your Accuracy Worth?
Johnson, J. K., & Marasović, A. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.19
Harnessing Black-Box Control to Boost Commonsense in LM’s Generation
Tian, Y., Zhang, F., & Peng, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.329 · Cited by 1
Liang, Q., Chen, Y., Li, W., Lai, M., Ni, W., & Qiu, H. X. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-3948-6_2
Reasoning in Transformers – Mitigating Spurious Correlations and Reasoning Shortcuts
Enström, D., Kjellberg, V., & Johansson, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_18
KLoB: a Benchmark for Assessing Knowledge Localization Methods in Language Models
Ju, Y., Ma, H., Xing, X., & Zeng, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0119-6_36
Predicting ChatGPT’s Ability to Solve Complex Programming Challenges
Ho, N., May, J., Ngo, B., Formato, J., Ngo, L., Ho, V. L., & Bui, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825139
A Conceptual Framework for Efficient and Sustainable Pruning Techniques in Deep Learning Models
Smarts, N., Rajalakshmi, S., & Kuthadi, V. M. (2025) · Journal of Computer Science · doi:10.3844/jcssp.2025.1113.1128
Digital Banking Transformation: LLMs in Customer Service and Fraud Detection
Manoj, R. J., Boobesh, K., Rajan, A. P., & Saravanan, T. P. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-78940-3_20
Zonjić, R., Alfirević, L., Bašić, K., Džanija, I., & Vranić, M. (2025) · International Colloquium on Smart Grid Metrology (SMAGRIMET) · doi:10.1109/smagrimet66590.2025.11030048
Probing Fine-Grained Hierarchical Concept Comprehension and Generation in Large Language Models
Sun, K., Bai, Y., Tu, S., Li, J., & Hou, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3592339
Initializing and Retrofitting Key-Value Adaptors for Traceable Model Editing
Zhu, H., Lan, Y., Li, X., & Qian, W. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.152
Generative AI for Mental Health Education: A Comparison of ChatGPT and DeepSeek
Mu, Y., & Li, Q. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0880-8_34
Li, J., & Liu, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-6702743/v1 · Cited by 2
Liu, H., Dai, L., & Jiang, H. (2025) · Applied Sciences · doi:10.3390/app151910696
EXAMINING THE ROLE OF TEXT ANALYSIS IN SOYBEAN OIL FUTURES PRICE PREDICTION
Gao, Q., & Yu, H. (2025) · Financial and credit activity problems of theory and practice · doi:10.55643/fcaptp.5.64.2025.4823
Three Tiers of Computation in Transformers and in Brain Architectures
Graham, E., & Granger, R. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3844612
Gao, H., Chen, Z., Wang, J., & Peng, T. (2025) · International Symposium on Artificial Intelligence for Education (ISAIE) · doi:10.1145/3775073.3775234
Metadata
- Authors
- Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Ludwig-Maximilians-Universität München (DE)
- Carnegie Mellon University (US)
- Corresponding author
- Elazar, Yanai
- Kassner, Nora
- Ravfogel, Shauli
- Ravichander, Abhilasha
- Hovy, Eduard
- Goldberg, Yoav
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2021
- Volume
- 9
- Pages
- 1012-1031
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3202712981
- Cited by
- 220
- References
- 93
- Citations by year
- 2026: 37 · 2025: 40 · 2024: 30 · 2023: 74 · 2022: 20 · 2021: 17
- FWCI
- 12.35
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-12-06)
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1801316, 1914486, 1330596)
- Brandeis University
- European Commission (HORIZON2020)
- Bundesministerium für Bildung und Forschung (01IS18036A)
- Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
- H2020 European Research Council (802774, 740516)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-trained language models, consistency, factual knowledge, paraphrase, knowledge representation, representational space
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review