The DOI matches its Crossref record; 90 of the 93 references it cites were found in a registry or an index.
Measuring and Improving Consistency in Pretrained Language Models
Open accessAbstract
Consistency of a model—that is, the invariance of its behavior under meaning-preserving alternations in its input—is a highly desirable property in natural language processing. In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To this end, we create ParaRel🤘, a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. Using ParaRel🤘, we show that the consistency of all PLMs we experiment with is poor— though with high variance between relations. Our analysis of the representational spaces of PLMs suggests that they have a poor structure and are currently not suitable for representing knowledge robustly. Finally, we propose a method for improving model consistency and experimentally demonstrate its effectiveness.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, LLMs can struggle on commonsense knowledge benchmarks [158], especially once low-level statistical cues are controlled for [9].”
From Dissociating language and thought in large language models (2024) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2008) · Journal of Machine Learning Research · Cited by 33,051
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
Mariani, J. (2018) · HAL · Cited by 1,151
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1×
“Indeed, there is an ongoing discussion about the ability to acquire understanding of “meaning" from raw text signal alone (Bender and Koller, 2020).”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,057
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 820
Self-citationEvaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669
Knowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638
(2019) · doi:10.18653/v1/n19-1419 · Cited by 601
The pascal recognising textual entailment challenge
Dagan, I., & Glickman, O. (2005) · Cited by 587
Honnibal, M., Montani, I., Honnibal, M., Peters, H., Samsonov, M., Geovedi, J., Regan, J., Orosz, G., Kristiansen, S. L., R., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1212303 · Cited by 543
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 522
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Ettinger, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00298 · Cited by 511
Semantically Equivalent Adversarial Rules for Debugging NLP models
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1079 · Cited by 457
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385
Self-citationThe Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing
Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1128 · Cited by 384
e-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367 · cited 1×
“Li et al. (2019) measure and mitigate inconsistency in natural language inference (NLI), and finally, Camburu et al. (2020) propose a method for measuring inconsistencies in NLI explanations (Camburu et al., 2018).”
MethodTargeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334
What do Neural Machine Translation Models Learn about Morphology?
Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., & Glass, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1080 · Cited by 307
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Kassner, N., & Schütze, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.698 · Cited by 303
Self-citationoLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 288
Self-citationCommonsense Knowledge Mining from Pretrained Models
Davison, J., Feldman, J., & Rush, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1109 · Cited by 264
Synthetic QA Corpora Generation with Roundtrip Consistency
Alberti, C., Andor, D., Pitler, E., Devlin, J., & Collins, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1620 · Cited by 221
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 197
Georgakopoulos, G., Kavvadias, D. J., & Papadimitriou, C. H. (1988) · Journal of Complexity · doi:10.1016/0885-064x(88)90006-4 · Cited by 197
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language ModelPreprint
Xiong, W., Du, J., Wang, W. Y., & Stoyanov, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09637 · Cited by 190
(2018) · doi:10.63317/2yu5nu76n7bu · Cited by 189
How Context Affects Language Models' Factual Predictions
Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., & Riedel, S. (2020) · arXiv · doi:10.48550/arxiv.2005.04611 · Cited by 154
How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
Linzen, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.465 · Cited by 148
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.71 · Cited by 146
Self-citationFinding Universal Grammatical Relations in Multilingual BERT
Chi, E. A., Hewitt, J., & Manning, C. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.493 · Cited by 138
Bhagat, R., & Hovy, E. (2013) · International Conference on Computational Logic · doi:10.1162/coli_a_00166 · Cited by 132
Self-citationAmnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126
Self-citationImproving the Robustness of Question Answering Systems to Question Paraphrasing
Gan, W. C., & Ng, H. T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1610 · Cited by 117
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Kassner, N., Dufter, P., & Schütze, H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.284 · Cited by 116
Self-citationWarstadt, A., Zhang, Y., Li, H.-S., Liu, H., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 115
Integrating Distributional Lexical Contrast into Word Embeddings for Antonym-Synonym Distinction
Nguyen, K. A., Walde, S. S. I., & Vu, N. T. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2074 · Cited by 107
Are Red Roses Red? Evaluating Consistency of Question-Answering Models
Ribeiro, M. T., Guestrin, C., & Singh, S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1621 · Cited by 100
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering
Asai, A., & Hajishirzi, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.499 · Cited by 81
Inconsistency measures for probabilistic logics
Thimm, M. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2013.02.001 · Cited by 65
A Logic-Driven Framework for Consistency of Neural Models
Li, T., Gupta, V., Mehta, M., & Srikumar, V. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1405 · Cited by 64
Statistical significance testing––a panacea for software technology experiments?
Miller, J. (2004) · Journal of Systems and Software · doi:10.1016/j.jss.2003.12.019 · Cited by 62
Global joint models for coreference resolution and named entity classification
Denis, P., & Baldridge, J. (2009) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 61
Do Neural Language Representations Learn Physical Commonsense?
Forbes, M., Holtzman, A., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1908.02899 · Cited by 60
Commonsense Knowledge Mining from Pretrained ModelsPreprint
Feldman, J., Davison, J., & Rush, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00505 · Cited by 58
Are Pretrained Language Models Symbolic Reasoners over Knowledge?
Kassner, N., Krojer, B., & Schütze, H. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.45 · Cited by 57
Self-citationMeasuring and repairing inconsistency in probabilistic knowledge bases
Muiño, D. P. (2011) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2011.02.003 · Cited by 56
Statistical Significance Testing for Natural Language Processing
Dror, R., Peled-Cohen, L., Shlomov, S., & Reichart, R. (2020) · Synthesis lectures on human language technologies · doi:10.2200/s00994ed1v01y202002hlt045 · Cited by 55
AI Explainability 360: An Extensible Toolkit for Understanding Data and Machine Learning Models
Arya, V., Bellamy, R., Chen, P., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2020) · Journal of Machine Learning Research · Cited by 54
Do Neural Language Models Overcome Reporting Bias?
Shwartz, V., & Choi, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.605 · Cited by 48
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 41
Self-citationMeasuring Inconsistency in Probabilistic Knowledge BasesPreprint
Thimm, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2613 · Cited by 41
Do Language Embeddings capture Scales?
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.439 · Cited by 39
Self-citationProceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task
Pradhan, S. (2011) · Cited by 38
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.382 · Cited by 36
Inference Protocols for Coreference Resolution
Chang, K.-W., Samdani, R., Rozovskaya, A., Rizzolo, N., Sammons, M., & Roth, D. (2011) · Cited by 31
Be Consistent! Improving Procedural Text Comprehension using Label Consistency
Du, X., Dalvi, B. B., Tandon, N., Bosselut, A., Yih, W.-T., Clark, P. E., & Cardie, C. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1244 · Cited by 30
Shlain, M., Taub-Tabib, H., Sadde, S., & Goldberg, Y. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.3 · Cited by 29
Self-citationAutomated knowledge Base Construction
de Rijke, M., & van Harmelen, F. (2019) · Cited by 29
Hansen, P., & Jaumard, B. (2000) · Springer eBooks · doi:10.1007/978-94-017-1737-3_8 · Cited by 28
Easy Cases of Probabilistic Satisfiability
Andersen, K. A., & Pretolani, D. (2001) · Annals of Mathematics and Artificial Intelligence · doi:10.1023/a:1012332915908 · Cited by 25
Understanding Commonsense Inference Aptitude of Deep Contextual Representations
Da, J., & Kasai, J. (2019) · Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing · doi:10.18653/v1/d19-6001 · Cited by 22
Neural DatabasesPreprint
Thorne, J., Yazdani, M., Saeidi, M., Silvestri, F., Riedel, S., & Halevy, A. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06973 · Cited by 20
Deep Learning Approaches to Text Production
Narayan, S., & Gardent, C. (2020) · Synthesis lectures on human language technologies · doi:10.2200/s00979ed1v01y201912hlt044 · Cited by 13
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations
Ravfogel, S., Elazar, Y., Goldberger, J., & Goldberg, Y. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.9 · Cited by 12
Self-citationJi, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Enriching a Model's Notion of Belief using a Persistent MemoryPreprint
Kassner, N., Tafjord, O., Schütze, H., & Clark, P. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08401 · Cited by 5
Self-citationSchulze, E., Hessenmoeller, D., Knohl, A., et al. (2012) · Data Archiving and Networked Services (DANS) · doi:10.1007/978
Research works printed but not linked: 3
- Elsahar, 2018, T-rex: A large scale alignment of natural language with knowledge base triples
- Hewitt, 2019, A structural probe for finding syntax in word representations
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
Cited by
Sorted by citations
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
“, 2020), and factual knowledge extraction (Elazar et al., 2021).”
BackgroundDissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214
Knowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study
Cao, Y., Zhou, L., Lee, S., Cabello, L., Chen, M., & Hershcovich, D. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.7 · Cited by 163
Metadata
- Authors
- Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Ludwig-Maximilians-Universität München (DE)
- Carnegie Mellon University (US)
- Corresponding author
- Elazar, Yanai
- Kassner, Nora
- Ravfogel, Shauli
- Ravichander, Abhilasha
- Hovy, Eduard
- Goldberg, Yoav
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2021
- Volume
- 9
- Pages
- 1012-1031
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3202712981
- Cited by
- 220
- References
- 93
- Citations by year
- 2026: 37 · 2025: 40 · 2024: 30 · 2023: 74 · 2022: 20 · 2021: 17
- FWCI
- 12.35
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-12-06)
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1801316, 1914486, 1330596)
- Brandeis University
- European Commission (HORIZON2020)
- Bundesministerium für Bildung und Forschung (01IS18036A)
- Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
- H2020 European Research Council (802774, 740516)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-trained language models, consistency, factual knowledge, paraphrase, knowledge representation, representational space
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
