The DOI matches its Crossref record; 90 of the 93 references it cites were found in a registry or an index.
Measuring and Improving Consistency in Pretrained Language Models
Open accessAbstract
Consistency of a model—that is, the invariance of its behavior under meaning-preserving alternations in its input—is a highly desirable property in natural language processing. In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To this end, we create ParaRel🤘, a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. Using ParaRel🤘, we show that the consistency of all PLMs we experiment with is poor— though with high variance between relations. Our analysis of the representational spaces of PLMs suggests that they have a poor structure and are currently not suitable for representing knowledge robustly. Finally, we propose a method for improving model consistency and experimentally demonstrate its effectiveness.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, LLMs can struggle on commonsense knowledge benchmarks [158], especially once low-level statistical cues are controlled for [9].”
From Dissociating language and thought in large language models (2024) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2008) · Journal of Machine Learning Research · Cited by 33,051
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Research works printed but not linked: 3
- Elsahar, 2018, T-rex: A large scale alignment of natural language with knowledge base triples
- Hewitt, 2019, A structural probe for finding syntax in word representations
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
Cited by
Sorted by citations
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
“, 2020), and factual knowledge extraction (Elazar et al., 2021).”
BackgroundDissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214
Knowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study
Cao, Y., Zhou, L., Lee, S., Cabello, L., Chen, M., & Hershcovich, D. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.7 · Cited by 163
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. J. L., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.82 · Cited by 176
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
Few-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107
Self-citationLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Unsolved Problems in ML Safety
Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2109.13916 · Cited by 95
Is GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings
Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1272229 · Cited by 115
Few-Shot Text Generation with Pattern-Exploiting TrainingPreprint
Schick, T., & Schütze, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11926 · Cited by 113
Self-citationShorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., & Majumdar, A. (2025) · ACM Computing Surveys · doi:10.1145/3744238 · Cited by 163
Knowledge Editing for Large Language Models: A Survey
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., & Li, J. (2025) · ACM Computing Surveys · doi:10.1145/3698590 · Cited by 79
True Few-Shot Learning with Prompts—A Real-World Perspective
Schick, T., & Schütze, H. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00485 · Cited by 81
Self-citationKim, Y., Lee, J. K. W., Kim, S., Park, J., & Kim, J. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645148 · Cited by 87
Kim, J., Suh, S., Chilton, L. B., & Xia, H. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595996 · Cited by 61
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
The levers of political persuasion with conversational artificial intelligence
Hackenburg, K., Tappin, B. M., Hewitt, L., Saunders, E., Black, S., Lin, H., Fist, C., Margetts, H. Z., Rand, D. G., & Summerfield, C. (2025) · Science · doi:10.1126/science.aea3884 · Cited by 128
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“Inconsistencies can also occur between responses; many studies have demonstrated that LLMs often change their answer to a question when challenged, asked the question in a slightly different way, or re-asked the exact same question [32, 66, 71].”
Mei, K., Fereidooni, S., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594109 · Cited by 56
Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely
Kauf, C., Ivanova, A. A., Rambelli, G., Chersoni, E., She, J. S., Chowdhury, Z., Fedorenko, E. G., & Lenci, A. (2023) · Cognitive Science · doi:10.1111/cogs.13386 · Cited by 70
Zhang, Y., Sui, X., Pan, F., Yu, K., Li, K., Tian, S., Erdengasileng, A., Han, Q., Wang, W., Wang, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01014-w · Cited by 71
Scarecrows in Oz: The Use of Large Language Models in HRI
Williams, T., Matuszek, C., Mead, R., & DePalma, N. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3606261 · Cited by 63
Can We Edit Factual Knowledge by In-Context Learning?
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., & Chang, B. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.296 · Cited by 54
Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., & Wei, H. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736569 · Cited by 53
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023) · arXiv · doi:10.48550/arxiv.2310.11324 · Cited by 52
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 56
BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models
Hao, S., Tan, B., Tang, K., Ni, B., Shao, X., Zhang, H., Xing, E. P., & Hu, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.309 · Cited by 71
Language models and psychological sciences
Sartori, G., & Orrù, G. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1279317 · Cited by 58
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models
Wang, H., & Shu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.416 · Cited by 83
Large language model for knowledge synthesis and AI-enhanced biomanufacturing
Li, W., Mao, Z., Xiao, Z., Liao, X., Koffas, M. A. G., Chen, Y., Ma, H., & Tang, Y. (2025) · Trends in biotechnology · doi:10.1016/j.tibtech.2025.02.008 · Cited by 48
Prompt Consistency for Zero-Shot Task Generalization
Zhou, C., He, J., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.192 · Cited by 68
Scaling language model size yields diminishing returns for single-message political persuasion
Hackenburg, K., Tappin, B. M., Röttger, P., Hale, S. A., Bright, J., & Margetts, H. Z. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2413443122 · Cited by 72
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 40
Calibrating Factual Knowledge in Pretrained Language Models
Dong, Q., Dai, D., Song, Y., Xu, J., Sui, Z., & Li, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.438 · Cited by 39
Language Models in the Loop: Incorporating Prompting into Weak Supervision
Smith, R., Fries, J., Hancock, B. J., & Bach, S. H. (2024) · ACM / IMS Journal of Data Science · doi:10.1145/3617130 · Cited by 66
Thanasi-Boçe, M., & Hoxha, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12775-z · Cited by 44
Relational World Knowledge Representation in Contextual Language Models: A Review
Safavi, T., & Koutra, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.81 · Cited by 52
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., & Evans, O. (2023) · arXiv · doi:10.48550/arxiv.2309.12288 · Cited by 36
“Previous research has found similar inconsistencies in LLMs (Fluri et al., 2023; Elazar et al., 2021; Press et al., 2023; Hosseini et al., 2021; Lin et al., 2022; Shi et al., 2023) Forward vs backward recall in humans Does the Reversal Curse apply to humans?”
MethodLee, Y., Son, K., Kim, T. S., Kim, J., Chung, J. J. Y., Adar, E., & Kim, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3662681 · Cited by 36
Elazar, Y., Zhang, H., Goldberg, Y., & Roth, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.819 · Cited by 42
Self-citationHow Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
Li, S., Li, X., Shang, L., Dong, Z., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.136 · Cited by 45
Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
Yu, Q., Li, J., Wu, Y., Tang, S., Ji, W., & Zhuang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01971 · Cited by 42
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
Kassner, N., Tafjord, O., Schütze, H., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.697 · Cited by 47
Self-citationEntailer: Answering Questions with Faithful and Truthful Chains of Reasoning
Tafjord, O., Mishra, B. D., & Clark, P. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.134 · Cited by 35
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model BeliefsPreprint
Hase, P., Diab, M., Çelikyılmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., & Iyer, S. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13654 · Cited by 31
Shyr, C., Ren, B., Hsu, C.-Y., Nguyen, M.-H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A. T., et al. (2026) · Journal of the American Medical Informatics Association · doi:10.1101/2025.08.06.25333170 · Cited by 32
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Kim, H., Sclar, M., Zhou, X., Bras, R., Kim, G., Choi, Y., & Sap, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.890 · Cited by 27
Pandia, L., & Ettinger, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 35
Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark
Hoelscher-Obermaier, J., Persson, J., Kran, E., Konstas, I., & Barez, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.733 · Cited by 24
Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models
Hase, P., Diab, M., Çelikyılmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., & Iyer, S. V. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.199 · Cited by 32
A Survey on Knowledge Editing of Neural Networks
Mazzia, V., Pedrani, A., Caciolai, A., Rottmann, K., & Bernardi, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3498935 · Cited by 40
Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification
Li, S., Zhao, R., Li, M., Ji, H., Callison-Burch, C., & Han, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.312 · Cited by 29
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
Zhang, H., Huang, J., Li, Z., Naik, M., & Xing, E. P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.191 · Cited by 31
Misra, K., Rayz, J. T., & Ettinger, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 32
Chen, L., Darko, A., Zhang, F., Chan, A. P. C., & Yang, Q. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.112891 · Cited by 24
Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View
Cao, B., Lin, H., Han, X., Liu, F., & Sun, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.398 · Cited by 28
Zhang, J., Li, Z., Das, K., Malin, B., & Kumar, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1032 · Cited by 29
Mitchell, E., Noh, J. J., Li, S., Armstrong, W., Agarwal, A., Liu, P., Finn, C., & Manning, C. D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.115 · Cited by 34
Alrayzah, A., Alsolami, F., & Saleh, M. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1633 · Cited by 22
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 22
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 35
Self-citationThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
Geng, T., Xu, Z., Qu, Y., & Wong, W. E. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.074081 · Cited by 25
PERSONALITY CLASSIFICATION OF TEXT THROUGH MACHINE LEARNING AND DEEP LEARNING: A REVIEW (2023)
Perera, H., & Costa, L. (2023) · International Journal For Research In Advanced Computer Science And Engineering · doi:10.53555/cse.v9i4.2266 · Cited by 22
Impact of Co-occurrence on Factual Knowledge of Large Language Models
Kang, C., & Choi, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.518 · Cited by 38
Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 24
Fan, A., & Gardent, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.586 · Cited by 21
DREAM: Improving Situational QA by First Elaborating the Situation
Gu, Y., Dalvi, B. B., & Clark, P. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.82 · Cited by 20
Factual Consistency of Multilingual Pretrained Language Models
Fierro, C., & Søgaard, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.240 · Cited by 23
Lent, H., Ortner, V. K., Karmisholt, K. E., Wiegell, S. R., Nissen, C., Omland, S. H., Kamstrup, M. R., Togsverd‐Bo, K., & Hædersdal, M. (2023) · JEADV Clinical Practice · doi:10.1002/jvc2.263 · Cited by 17
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models
Saeed, M., Ahmadi, N., Nakov, P., & Papotti, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.110 · Cited by 16
Evaluating Language Models for Knowledge Base Completion
Veseli, B., Singhania, S., Razniewski, S., & Weikum, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33455-9_14 · Cited by 23
Keleg, A., & Magdy, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.389 · Cited by 27
On the Unexpected Abilities of Large Language Models
Nolfi, S. (2024) · Adaptive Behavior · doi:10.1177/10597123241256754 · Cited by 19
Rewthamrongsris, P., Burapacheep, J., Phattarataratip, E., Kulthanaamondhita, P., Tichý, A., Schwendicke, F., Osathanon, T., & Sappayatosok, K. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.100848 · Cited by 16
Zeng, F., & Gao, W. G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.278 · Cited by 17
Pre-training Language Models with Deterministic Factual Knowledge
Li, S., Li, X., Shang, L., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.764 · Cited by 16
Evaluating Superhuman Models with Consistency Checks
Fluri, L., Paleka, D., & Tramèr, F. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00017 · Cited by 39
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Youssef, P., Koraş, O. A., Li, M., Schlötterer, J., & Seifert, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1043 · Cited by 18
Static Embeddings as Efficient Knowledge Bases?
Dufter, P., Kassner, N., & Schütze, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.186 · Cited by 15
Self-citationLederman, H., & Mahowald, K. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00690 · Cited by 22
BERTopic Konu Modelleme Tekniği Kullanılarak Müşteri Şikayetlerinin Sınıflandırılması
Koruyan, K. (2022) · İzmir Sosyal Bilimler Dergisi · doi:10.47899/ijss.1167719 · Cited by 12
Self-Consistency of Large Language Models under Ambiguity
Bartsch, H., Jørgensen, O. J., Rosati, D., Hoelscher-Obermaier, J., & Pfau, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.7 · Cited by 20
Solving Proof Block Problems Using Large Language Models
Poulsen, S., Sarsa, S., Prather, J. E., Leinonen, J., Becker, B. A., Hellas, A., Denny, P. C., & Reeves, B. N. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630928 · Cited by 14
ChatGPT Incorrectness Detection in Software Reviews
Tanzil, M. H., Khan, J. Y., & Uddin, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639194 · Cited by 23
Reliability Check: An Analysis of GPT-3’s Response to Sensitive Topics and Prompt Wording
Khatun, A., & Brown, D. G. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.8 · Cited by 18
P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse PromptsPreprint
Newman, B., Choubey, P. K., & Rajani, N. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07280 · Cited by 17
The Factuality of Large Language Models in the Legal Domain
Hamdani, R. E., Bonald, T., Malliaros, F. D., Holzenberger, N., & Suchanek, F. M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679961 · Cited by 12
Xue, B., Wang, W., Wang, H., Mi, F., Wang, R., Wang, Y., Shang, L., Jiang, X., Liu, Q., & Wong, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.525 · Cited by 17
From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
Ohmer, X. I., Bruni, E., & Hupkes, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00529 · Cited by 18
Neural Knowledge Bank for Pretrained Transformers
Dai, D., Jiang, W., Dong, Q., Lyu, Y., & Sui, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_60 · Cited by 16
DuCL: Dual-stage contrastive learning framework for Chinese semantic textual matching
Zuo, Y., Lü, W., Peng, X., Wang, S., Zhang, W., & Qiao, X. (2023) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108574 · Cited by 10
Emergent Modularity in Pre-trained Transformers
Zhang, Z., Zeng, Z., Lin, Y., Xiao, C., Wang, X., Han, X., Liu, Z., Xie, R., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.250 · Cited by 14
Language Models with Rationality
Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schuetze, H., & Clark, P. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.877 · Cited by 23
Self-citationBack to Square One: Bias Detection, Training and Commonsense Disentanglement in the Winograd SchemaPreprint
Elazar, Y., Zhang, H., Goldberg, Y., & Roth, D. (2021) · arXiv (Cornell University) · Cited by 10
Self-citationYin, Q., Zhong, L. L. D., Song, Y., Bai, L., Wang, Z., Chen, L., Xu, R. Y. D., & Yang, X. (2023) · Annals of Operations Research · doi:10.1007/s10479-023-05545-6 · Cited by 10
Metadata
- Authors
- Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Ludwig-Maximilians-Universität München (DE)
- Carnegie Mellon University (US)
- Corresponding author
- Elazar, Yanai
- Kassner, Nora
- Ravfogel, Shauli
- Ravichander, Abhilasha
- Hovy, Eduard
- Goldberg, Yoav
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2021
- Volume
- 9
- Pages
- 1012-1031
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3202712981
- Cited by
- 220
- References
- 93
- Citations by year
- 2026: 37 · 2025: 40 · 2024: 30 · 2023: 74 · 2022: 20 · 2021: 17
- FWCI
- 12.35
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-12-06)
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1801316, 1914486, 1330596)
- Brandeis University
- European Commission (HORIZON2020)
- Bundesministerium für Bildung und Forschung (01IS18036A)
- Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
- H2020 European Research Council (802774, 740516)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-trained language models, consistency, factual knowledge, paraphrase, knowledge representation, representational space
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
