Understanding Commonsense Inference Aptitude of Deep Contextual Representations
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models.Thus, we investigate and challenge several aspects of BERT's commonsense representation abilities.First, we probe BERT's ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas.Next, we show that, by augmenting BERT's pretraining data with additional data related to the deficient attributes, we are able to improve performance on a downstream commonsense reasoning task while using a minimal amount of data.Finally, we develop a method of fine-tuning knowledge graphs embeddings alongside BERT and show the continued importance of explicit knowledge graphs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These studies have tended to be more positive than those which are based on question answering tasks, often reporting results that suggest the LLMs perform well above baseline, though still below humans.”
From Symbols and grounding in large language models (2023) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 8×
“We do this since we need to build a classification layer for the specific task, as noted in Devlin et al. (2019).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 8×
“We do this since we need to build a classification layer for the specific task, as noted in Devlin et al. (2019).”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We propose a method to evaluate BERT’s representations and compare to previous non-contextual GloVe (Pennington et al., 2014) baselines, using simple logistic classifiers.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 4×
“Several works have improved on BERT’s language objectives (Yang et al., 2019; Liu et al., 2019b), but we keep the language objectives in BERT intact for comparison.”
BackgroundResultInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“On the other hand, NLP systems have recently improved dramatically with contextualized word representations in a wide range of tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019).”
BackgroundBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Other work extends BERT to domains where its original training data does not suffice (Beltagy et al., 2019; Lee et al., 2019).”
BackgroundXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 2×
“Several works have improved on BERT’s language objectives (Yang et al., 2019; Liu et al., 2019b), but we keep the language objectives in BERT intact for comparison.”
BackgroundHow to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 1×
“Explicitly, ERNIE, Zhang et al. (2019) introduces information to contextual representations during pretraining.”
BackgroundRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 6×
“(2019) explores fine-tuning on indomain data, similarly to our fine-tuning on the RACE dataset (Lai et al., 2017).”
BackgroundMethodInfluentialMcCarthy, J. (1960) · Cited by 931
A study on similarity and relatedness using distributional and WordNet-based approaches
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., & Soroa, A. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics on - NAACL '09 · doi:10.3115/1620754.1620758 · Cited by 860 · cited 1×
“Similar to our attribute classification investigation, several other works have used applied semantic norm datasets to computational linguistics (Agirre et al., 2009; Bruni et al., 2012; Kiela et al., 2016).”
ResultHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · cited 1×
“Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
MethodATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
Sap, M., Le Bras, R., Allaway, E., Bhagavatula, C., Lourie, N., Rashkin, H., Roof, B., Smith, N. A., & Choi, Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013027 · Cited by 740
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1009 · Cited by 690 · cited 1×
“Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
MethodLinguistic Knowledge and Transferability of Contextual Representations
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., & Smith, N. A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1112 · Cited by 621 · cited 4×
“needed capacity (Liu et al., 2019a)).”
BackgroundResultInfluentialA Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598 · cited 2×
“Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
BackgroundMethodWinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579
Commonsense reasoning and commonsense knowledge in artificial intelligence
Davis, E., & Marcus, G. (2015) · Communications of the ACM · doi:10.1145/2701413 · Cited by 434
Distributional Semantics in Technicolor
Bruni, E., Boleda, G., Baroni, M., & Tran, N. K. (2012) · Iris (University of Trento) · Cited by 415
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Roemmele, M., Bejan, C. A., & Gordon, A. S. (2011) · Cited by 334
ConceptNet 5: A Large Semantic Network for Relational Knowledge
Speer, R., & Havasi, C. (2013) · Theory and Applications of Natural Language Processing · doi:10.1007/978-3-642-35085-6_6 · Cited by 223 · cited 4×
“the same information that external commonsense knowledge graphs (such as ConceptNet (Speer and Havasi, 2013)) provide.”
BackgroundMethodInfluentialThe Centre for Speech, Language and the Brain (CSLB) concept property norms
Devereux, B. J., Tyler, L. K., Geertzen, J., & Randall, B. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0420-4 · Cited by 177 · cited 3×
“We present a method of analyzing commonsense knowledge in word representations through attribute classification on the semantic norm dataset (Devereux et al., 2014), and compare a contextual model to a traditional word type representation.”
BackgroundMethodInfluentialSciBERT: Pretrained Contextualized Embeddings for Scientific Text
Beltagy, I., Cohan, A., & Lo, K. (2019) · Cited by 173
Reporting bias and knowledge acquisition
Gordon, J., & Van Durme, B. (2013) · Proceedings of the 2013 workshop on Automated knowledge base construction · doi:10.1145/2509558.2509563 · Cited by 169
Ordinal Common-sense Inference
Zhang, S., Rudinger, R., Duh, K., & Van Durme, B. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00068 · Cited by 109 · cited 2×
“Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
BackgroundMethodMCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ostermann, S., Modi, A., Roth, M., Thater, S., & Pinkal, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x36g84ob95p · Cited by 92
How to Fine-Tune BERT for Text Classification?
Sun, C. C., Qiu, X., Xu, Y., & Huang, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05583 · Cited by 89 · cited 1×
“Implicitly, Sun et al. (2019) explores fine-tuning on indomain data, similarly to our fine-tuning on the RACE dataset (Lai et al., 2017).”
MethodWinoGrande: An Adversarial Winograd Schema Challenge at ScalePreprint
Sakaguchi, K., Bras, R. J. L., Bhagavatula, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.10641 · Cited by 84
WebChild 2.0 : Fine-Grained Commonsense Knowledge Distillation
Tandon, N., de Melo, G., & Weikum, G. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4020 · Cited by 81 · cited 3×
“We query a suite of knowledge bases (ConceptNet (Speer and Havasi, 2013), WebChild (Tandon et al., 2017), ATOMIC (Sap et al.”
MethodMCScript: A Novel Dataset for Assessing Machine Comprehension Using Script KnowledgePreprint
Ostermann, S., Modi, A., Roth, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05223 · Cited by 68 · cited 2×
“Prior work has developed several bench- 9 mark datasets to assess commonsense knowledge of NLP models (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017; Zellers et al., 2018, 2019; Ostermann et al., 2018, 2019; Sakaguchi et al., 2019).”
MethodDo Neural Language Representations Learn Physical Commonsense?
Forbes, M., Holtzman, A., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1908.02899 · Cited by 60 · cited 1×
“Forbes et al. (2019) investigates the commonsense aptitude of contextual representations.”
BackgroundLucy, L., & Gauthier, J. (2017) · Proceedings of the First Workshop on Language Grounding for Robotics · doi:10.18653/v1/w17-2810 · Cited by 59 · cited 1×
“For baseline tests, we use the pretrained 300 dimensional GloVe embeddings,1 as they have shown to perform better than word2vec embeddings (Lucy and Gauthier, 2017).”
MethodMCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants
Ostermann, S., Roth, M., & Pinkal, M. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1012 · Cited by 44 · cited 3×
“0 (Ostermann et al., 2019) for several investigations in this paper.”
BackgroundMethodInfluentialVirtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research
Kiela, D., Bulat, L., Verő, A. L., & Clark, S. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07432 · Cited by 22 · cited 1×
“Similar to our attribute classification investigation, several other works have used applied semantic norm datasets to computational linguistics (Agirre et al., 2009; Bruni et al., 2012; Kiela et al., 2016).”
Result
Research works printed but not linked: 5
- Elia Bruni, Gemma Boleda, Marco Baroni, and Nam- Khanh Tran. 2012. Distributional semantics in tech- nicolor. In Proc. of ACL.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proc. of NAACL-HLT.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI 2011 Spring Symposium.
- Lenhart Schubert. 2015. What kinds of knowledge are needed for genuine understanding? In Proc of Cog- nitum.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In Proc. of ACL, Florence, Italy. Association for Computational Linguistics.
Cited by
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“This definition is in the spirit of the RTE definition (Dagan et al., 2005), which similarly supports a more flexible use of the notion of entailment.”
BackgroundSymbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
“[95] perform similar analyses.”
Semantic Structure in Deep Learning
Pavlick, E. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 66
Metadata
- Authors
- Da, J., & Kasai, J.
- Affiliations
- University of Washington (US)
- ORCID
- Kasai, Jungo
- Venue
- Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing
- Published
- 2019
- Pages
- 1-12
- OpenAlex
- W2985797697
- Cited by
- 22
- References
- 34
- Citations by year
- 2024: 1 · 2023: 2 · 2022: 2 · 2021: 13 · 2020: 4
- FWCI
- 1.82
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- commonsense reasoning, BERT, pre-trained language models, context representation, knowledge graph embedding, fine-tuning, embedding space analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.