Understanding Commonsense Inference Aptitude of Deep Contextual Representations
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models.Thus, we investigate and challenge several aspects of BERT's commonsense representation abilities.First, we probe BERT's ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas.Next, we show that, by augmenting BERT's pretraining data with additional data related to the deficient attributes, we are able to improve performance on a downstream commonsense reasoning task while using a minimal amount of data.Finally, we develop a method of fine-tuning knowledge graphs embeddings alongside BERT and show the continued importance of explicit knowledge graphs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These studies have tended to be more positive than those which are based on question answering tasks, often reporting results that suggest the LLMs perform well above baseline, though still below humans.”
From Symbols and grounding in large language models (2023) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 8×
“We do this since we need to build a classification layer for the specific task, as noted in Devlin et al. (2019).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 8×
“We do this since we need to build a classification layer for the specific task, as noted in Devlin et al. (2019).”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We propose a method to evaluate BERT’s representations and compare to previous non-contextual GloVe (Pennington et al., 2014) baselines, using simple logistic classifiers.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 4×
“Several works have improved on BERT’s language objectives (Yang et al., 2019; Liu et al., 2019b), but we keep the language objectives in BERT intact for comparison.”
BackgroundResultInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“On the other hand, NLP systems have recently improved dramatically with contextualized word representations in a wide range of tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019).”
BackgroundBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Other work extends BERT to domains where its original training data does not suffice (Beltagy et al., 2019; Lee et al., 2019).”
BackgroundXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 2×
“Several works have improved on BERT’s language objectives (Yang et al., 2019; Liu et al., 2019b), but we keep the language objectives in BERT intact for comparison.”
BackgroundHow to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 1×
“Explicitly, ERNIE, Zhang et al. (2019) introduces information to contextual representations during pretraining.”
BackgroundRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 6×
“(2019) explores fine-tuning on indomain data, similarly to our fine-tuning on the RACE dataset (Lai et al., 2017).”
BackgroundMethodInfluential
Research works printed but not linked: 5
- Elia Bruni, Gemma Boleda, Marco Baroni, and Nam- Khanh Tran. 2012. Distributional semantics in tech- nicolor. In Proc. of ACL.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proc. of NAACL-HLT.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI 2011 Spring Symposium.
- Lenhart Schubert. 2015. What kinds of knowledge are needed for genuine understanding? In Proc of Cog- nitum.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In Proc. of ACL, Florence, Italy. Association for Computational Linguistics.
Cited by
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“This definition is in the spirit of the RTE definition (Dagan et al., 2005), which similarly supports a more flexible use of the notion of entailment.”
BackgroundSymbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
“[95] perform similar analyses.”
Semantic Structure in Deep Learning
Pavlick, E. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 66
Combining pre-trained language models and structured knowledgePreprint
Colon-Hernandez, P., Havasi, C., Alonso, J. B., Huggins, M., & Breazeal, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12294 · Cited by 65
Relational World Knowledge Representation in Contextual Language Models: A Review
Safavi, T., & Koutra, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.81 · Cited by 68
“modeling corpora like Wikipedia are known to contain KB-like assertions about the world (Da and Kasai, 2019).”
BackgroundGoldstein, S., & Levinstein, B. A. (2026) · Philosophy of AI · doi:10.18716/ojs/phai/2026.11767 · Cited by 76
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 97
Probing Neural Language Models for Human Tacit AssumptionsPreprint
Weir, N., Poliak, A., & Durme, B. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04877 · Cited by 36
“Previous works (Rubinstein et al., 2015; Sommerauer & Fokkens, 2018; Da & Kasai, 2019) have tested for similar types of stereotypic beliefs; they use supervised training of probing classifiers (Conneau et al., 2018) to identify concept/attribute pairs.”
MethodMisra, K., Rayz, J., & Ettinger, A. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 15
“Contemporary work has used this dataset by taking as positive instances all concepts for which a property was generated, while taking the rest as negative instances (Lucy and Gauthier, 2017; Da and Kasai, 2019, etc.) for each property.”
BackgroundMethodThe World of an Octopus: How Reporting Bias Influences a Language Model’s Perception of Color
Paik, C., Aroca-Ouellette, S., Roncone, A., & Kann, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.63 · Cited by 47
“A different approach is to analyze the representation quality of LMs for linguistic tasks by training a simple MLP on pretrained model representations (Da and Kasai, 2019; Lin et al., 2019).”
MethodAcquiring and modeling abstract commonsense knowledge via conceptualization
He, M., Fang, T., Wang, W., & Song, Y. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104149 · Cited by 41
“However, commonsense knowledge from LMs still suffer from issues including deficient capability in certain important aspects [40, 39, 35, 41], generalization in different situations [38, 12], and reporting bias [43].”
BackgroundSemGloVe: Semantic Co-Occurrences for GloVe From BERT
Gan, L., Teng, Z., Zhang, Y., Zhu, L., Wu, F., & Yang, Y. (2022) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3197316 · Cited by 22
“Previous work has shown that the contextualized representations produced by BERT capture morphological [12], [13], lexical [12], syntactic [14], [15] and semantic knowledge [16], [17].”
BackgroundProbing for Bridging Inference in Transformer Language Models
Pandit, O., & Hou, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.327 · Cited by 22
Dove, G. O. (2024) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2023.0149 · Cited by 21
WordCraft: An Environment for Benchmarking Commonsense AgentsPreprint
Jiang, M., Luketina, J., Nardelli, N., Minervini, P., Torr, P. H. S., Whiteson, S., & Rocktäschel, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.09185 · Cited by 20
“Recent works highlight that such pre-trained models capture many aspects of commonsense (Da & Kasai, 2019) and relational knowledge (Petroni et al., 2019), in addition to linguistic knowledge (Jawahar et al., 2019; Hewitt & Manning, 2019; Reif et al., 2019).”
BackgroundIdentify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks
Bauer, L., & Bansal, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.192 · Cited by 16
Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations
Bihani, G., & Rayz, J. T. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.9 · Cited by 12
Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms
Kennington, C. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.11 · Cited by 10
Modeling Task Effects on Meaning Representation in the Brain via Zero-Shot MEG PredictionPreprint
Toneva, M., Stretcu, O., Póczos, B., Wehbe, L., & Mitchell, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08424 · Cited by 21
Commonsense Knowledge in Word Associations and ConceptNet
Liu, C., Cohn, T., & Frermann, L. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.38 · Cited by 13
What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured
Henlein, A., & Mehler, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.425 · Cited by 8
Towards Semantics-Enhanced Pre-Training: Can Lexicon Definitions Help Learning Sentence Meanings?
Ren, X., Sun, X., Wang, H., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17619 · Cited by 6
“…in whatever data scale; and more importantly (c) certain semantics are not naturally expressed in sentences due to reporting bias, which is a problem that cannot be addressed by empirical approaches (Lucy and Gauthier 2017; Da and Kasai 2019; Forbes, Holtzman, and Choi 2019; Kwon et al. 2019).”
BackgroundIdentifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
Wu, Z., Liu, N. F., & Potts, C. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.11 · Cited by 8
Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings
Lee, J., Park, D., Lee, J., Choi, H., & Lee, S.-E. (2025) · IEEE Access · doi:10.1109/access.2025.3618700 · Cited by 7
“(4) Which individual words or sensory modalities reveal significant differences between human and models, and what do these differences reveal about the limitations or strengths of these models in simulating embodied experiences?”
Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models
Derby, S., Devereux, B. J., & Miller, P. C. (2021) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2021.cmcl-1.25 · Cited by 7
“There is some evidence that language models implicitly encode such knowledge (Da and Kusai, 2019; Weir et al., 2020); however, coverage of different types of knowledge may be inconsistent, with evidence to suggest that these models fail to capture some types of semantic knowledge such as visual…”
BackgroundBeyond Language: Learning Commonsense from Images for Reasoning
Cui, W., Lan, Y., Pang, L., Guo, J., & Cheng, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.392 · Cited by 5
Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning TasksPreprint
Bauer, L., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.10193 · Cited by 5
Understanding in Artificial IntelligencePreprint
Maetschke, S., Iraola, D. M., Barnard, P., ShafieiBavani, E., Zhong, P., Xu, Y., & Yepes, A. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06573 · Cited by 5
“[58] investigated a number of aspects of BERT’s commonsense representation abilities to provide a better understanding of the capability of these models.”
BackgroundRevisiting the Prepositional-Phrase Attachment Problem Using Explicit Commonsense KnowledgePreprint
Xin, Y., Lieberman, H. A., & Chin, P. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.00924 · Cited by 2
“Work has been done on constructing post-facto explanations for opaque inference processes (Petroni et al., 2019; Da and Kasai, 2019; Shwartz et al., 2020), but it is questionable how general the models and explanations really are.”
BackgroundAcquiring and modelling abstract commonsense knowledge via conceptualization
He, M. (2022) · doi:10.14711/thesis-991013098357503412 · Cited by 5
Linzbach, S., Dimitrov, D., Kallmeyer, L., Evang, K., Jabeen, H., & Dietze, S. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.201 · Cited by 3
Understanding Natural Language Beyond Surface by LLMs
Yong, Y. (2024) · Journal of Advances in Civil and Mechanical Engineering · doi:10.64030/3067-2457.01.01.02 · Cited by 1
“In the paper “Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Ap-titude of Deep Contextual Representations” (Da and Kasai, 2019) a method was developed through attribute classification in the semantic datasets and compared the contextual model to traditional word embedding.”
BackgroundMethodInfluentialAdversarial Transformer Language Models for Contextual Commonsense InferencePreprint
Colon-Hernandez, P., Lieberman, H. A., Xin, Y., Yin, C., Breazeal, C., & Chin, P. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05406 · Cited by 3
“Research into the knowledge that pre-trained language models learn has yielded good results in that they do contain various types of factual knowledge, as well as some commonsense knowledge [10, 11, 44].”
BackgroundToward context-based text-to-3D scene generation
Henlein, A. (2022) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.73448 · Cited by 1
Augmenting Structure with Text for Improved Graph Learning
Safavi, T. (2022) · Deep Blue (University of Michigan) · doi:10.7302/6246 · Cited by 2
Chavez, A. D. J. G., Gutiérrez-López, E., Flores, D., Gonzalez-Fraga, J. A., Montes-y-Gómez, M., & Chávez, E. (2026) · IEEE Access · doi:10.1109/access.2026.3717256
“Analyses of these representations demonstrate their effectiveness in encoding world knowledge [57], [58].”
Metadata
- Authors
- Da, J., & Kasai, J.
- Affiliations
- University of Washington (US)
- ORCID
- Kasai, Jungo
- Venue
- Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing
- Published
- 2019
- Pages
- 1-12
- OpenAlex
- W2985797697
- Cited by
- 22
- References
- 34
- Citations by year
- 2024: 1 · 2023: 2 · 2022: 2 · 2021: 13 · 2020: 4
- FWCI
- 1.82
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- commonsense reasoning, BERT, pre-trained language models, context representation, knowledge graph embedding, fine-tuning, embedding space analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.