A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, James Allen. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“Here we count event as any hyponym of ‘event’ or ‘process’ in WordNet (Miller., 1995).”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
Kronenfeld, D. B. (1978) · Language · doi:10.1353/lan.1978.0028 · Cited by 3,081
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Kronenfeld, D. B., Schank, R. C., & Abelson, R. P. (1978) · Language · doi:10.2307/412850 · Cited by 3,054
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Hunt, E., Schank, R., & Abelson, R. (1979) · The American Journal of Psychology · doi:10.2307/1421499 · Cited by 2,944
Understanding natural language
Winograd, T. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,799
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
“Cloze Procedure”: A New Tool for Measuring Readability
Taylor, W. L. (1953) · Journalism Quarterly · doi:10.1177/107769905303000401 · Cited by 2,332
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06724 · Cited by 2,264
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009
Self-citation5.7 The WORDNET lexical database of English
(2019) · Computer Corpus Lexicography · doi:10.1515/9781474471459-050 · Cited by 1,865
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · cited 2×
“Skip-thoughts Model : This model uses Skip-thoughts’ Sentence2Vec embedding (Kiros et al., 2015) which models the semantic space of novels.”
MethodForster, E. M. (1927) · University of Michigan Library Repository · Cited by 1,011
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 885
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C. J. C., & Renshaw, E. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 610
Unsupervised Learning of Narrative Event Chains
Chambers, N., & Jurafsky, D. (2008) · Cited by 546
Self-citationUnsupervised learning of narrative schemas and their participants
Chambers, N., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690231 · Cited by 389
Self-citationThe third PASCAL recognizing textual entailment challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing - RTE '07 · doi:10.3115/1654536.1654538 · Cited by 376
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Roemmele, M., Bejan, C. A., & Gordon, A. S. (2011) · Cited by 334
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2016) · International Conference on Learning Representations · Cited by 261
Toward A Model Of Children''s Story Comprehension
Eugene Charniak, D. (1972) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 213
Learning Latent Personas of Film Characters
Bamman, D., O'Connor, B., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473477 · Cited by 179
Statistical Script Learning with Multi-Argument Events
Pichotta, K., & Mooney, R. J. (2014) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1024 · Cited by 139
CaTeRS: Causal and Temporal Relation Scheme for Semantic Annotation of Event Structures
Mostafazadeh, N., Grealish, A., Chambers, N., Allen, J. F., & Vanderwende, L. (2016) · doi:10.18653/v1/w16-1007 · Cited by 126
Self-citationLearning Script Knowledge with Web Experiments
Regneri, M., Koller, A., & Pinkal, M. (2010) · Cited by 125
The ICWSM 2009 Spinn3r Dataset
Burton, K., Java, A., & Soboroff, I. M. (2009) · Cited by 124
Event Schema Induction with a Probabilistic Entity-Driven Model
Chambers, N. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1185 · Cited by 118
Self-citationSkip N-grams and Ranking Functions for Predicting Script Events
Jans, B., Vulić, I., Bethard, S. J., & Moens, M. (2012) · Lirias · Cited by 110
Probabilistic Frame InductionPreprint
Cheung, J. C. K., Poon, H., & Vanderwende, L. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4813 · Cited by 106
Self-citationDeep semantic analysis of text
Allen, J. F., Swift, M. D., & de Beaumont, W. (2008) · Conference on Semantics in Text Processing (STEP) · doi:10.3115/1626481.1626508 · Cited by 104
Self-citationGenerating Coherent Event Schemas at Scale
Balasubramanian, N., Soderland, S. G., Mausam, & Etzioni, O. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1178 · Cited by 102
Identifying Personal Stories in Millions of Weblog Entries
Gordon, A. S., & Swanson, R. (2009) · Cited by 95
McIntyre, N., & Lapata, M. (2009) · doi:10.3115/1687878.1687910 · Cited by 95
Script Induction as Language Modeling
Rudinger, R., Rastogi, P., Ferraro, F., & Durme, B. V. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1195 · Cited by 94
Schubert, L. K., & Hwang, C. H. (2000) · Cited by 84
Say Anything: A Massively Collaborative Open Domain Story Writing Companion
Swanson, R., & Gordon, A. S. (2006) · Lecture notes in computer science · doi:10.1007/978-3-540-89454-4_5 · Cited by 78
Understanding script-based stories using commonsense reasoning
Mueller, E. T. (2004) · Cognitive Systems Research · doi:10.1016/j.cogsys.2004.06.001 · Cited by 72
Generative Event Schema Induction with Entity Disambiguation
Nguyen, K.-H., Tannier, X., Ferret, O., & Besançon, R. (2015) · doi:10.3115/v1/p15-1019 · Cited by 61
Learning a Probabilistic Model of Event Sequences From Internet Weblog Stories
Manshadi, M., Swanson, R., & Gordon, A. S. (2008) · Cited by 50
Searching for Storiness: Story-Generation from a Reader's Perspective
Bailey, P. (1999) · Cited by 46
(2003)
Research works printed but not linked: 4
- Gonzalo M´ endez, Pablo Gerv´ as, and Carlos Le´ on. 2014. A model of character affinity for agent-based story generation. In 9th International Conference on Knowledge, Information and Creativity Support Systems , Limassol, Cyprus, 11/2014. Springer-Verlag, Springer-Verlag.
- Erik T. Mueller. 2007. Modeling space and time in narratives about restaurants. LLC , 22(1):67-84.
- Karl Pichotta and Raymond J. Mooney. 2014b. Statistical script learning with multi-argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2014) , Gothenburg, Sweden, April.
- M. Riedl and Carlos Le´ on. 2008. Toward vignette-based story generation for drama management systems. In Workshop on Integrating Technologies for Interactive Stories - 2nd International Conference on INtelligent TEchnologies for interactive enterTAINment , 8-10/1.
Cited by
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“Next we study the variance of few-shot learning performance on the StoryCloze, Natural Questions, and Web Questions benchmarks presented in Section 6.1.”
BackgroundInfluentialConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
Metadata
- Authors
- Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J.
- Affiliations
- University of Rochester (US)
- United States Naval Academy (US)
- Microsoft (United States) (US)
- Virginia Tech (US)
- Florida Institute for Human and Machine Cognition (US)
- Venue
- Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2016
- Pages
- 839-849
- OpenAlex
- W2466175319
- Cited by
- 598
- References
- 46
- Citations by year
- 2026: 16 · 2025: 32 · 2024: 35 · 2023: 95 · 2022: 95 · 2021: 111 · 2020: 76 · 2019: 58 · 2018: 41 · 2017: 31 · 2016: 8
- FWCI
- 24.79
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Funders
- Nuance Foundation
- Defense Advanced Research Projects Agency (W911NF-15-1-0542)
- Advanced Research Projects Agency
- Office of Naval Research
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, narrative comprehension, corpus construction, reading comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.