A structured distributional model of sentence meaning and processing
Open accessAbstract
Most compositional distributional semantic models represent sentence meaning with a single vector. In this paper, we propose a structured distributional model (SDM) that combines word embeddings with formal semantics and is based on the assumption that sentences represent events and situations. The semantic representation of a sentence is a formal structure derived from discourse representation theory and containing distributional vectors. This structure is dynamically and incrementally built by integrating knowledge about events and their typical participants, as they are activated by lexical items. Event knowledge is modelled as a graph extracted from parsed corpora and encoding roles and relationships between participants that are represented as distributional vectors. SDM is grounded on extensive psycholinguistic research showing that generalized knowledge about events stored in semantic memory plays a key role in sentence comprehension.We evaluate SDMon two recently introduced compositionality data sets, and our results show that combining a simple compositionalmodel with event knowledge constantly improves performances, even with dif ferent types of word embeddings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924 · see citing works
Pustejovsky, J. D. (1995) · Oxford University Press eBooks · doi:10.1093/oso/9780195136975.003.0018 · Cited by 3,959 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581 · see citing works
Baroni, M., Dinu, G., & Kruszewski, G. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1023 · Cited by 1,391 · see citing works
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Levy, O., Goldberg, Y., & Dagan, I. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00134 · Cited by 1,307 · see citing works
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · see citing works
The WaCky wide web: a collection of very large linguistically processed web-crawled corpora
Baroni, M., Bernardini, S., Ferraresi, A., & Zanchetta, E. (2009) · Language Resources and Evaluation · doi:10.1007/s10579-009-9081-4 · Cited by 1,220 · see citing works
What do we mean by prediction in language comprehension?
Kuperberg, G. R., & Jaeger, T. F. (2016) · Language, Cognition and Neuroscience · doi:10.1080/23273798.2015.1102299 · Cited by 1,087 · see citing works
(2017) · International Conference on Learning Representations · Cited by 1,075 · see citing works
Meaning, Use, and Interpretation of Language
Bäuerle, R., Schwarze, C., & von Stechow, A. (1983) · De Gruyter eBooks · doi:10.1515/9783110852820 · Cited by 948 · see citing works
Composition in Distributional Models of Semantics
Mitchell, J., & Lapata, M. (2010) · Cognitive Science · doi:10.1111/j.1551-6709.2010.01106.x · Cited by 946 · see citing works
Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension
McRae, K., Spivey-Knowlton, M. J., & Tanenhaus, M. K. (1998) · Journal of Memory and Language · doi:10.1006/jmla.1997.2543 · Cited by 742 · see citing works
Distributional Memory: A General Framework for Corpus-Based Semantics
Baroni, M., & Lenci, A. (2010) · Computational Linguistics · doi:10.1162/coli_a_00016 · Cited by 660 · see citing works
Self-citationThe statistics of word cooccurrences : word pairs and collocations
Evert, S. (2005) · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · doi:10.18419/opus-2556 · Cited by 641 · see citing works
The proactive brain: memory for predictions
Bar, M. (2009) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2008.0310 · Cited by 583 · see citing works
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1198 · Cited by 572 · see citing works
Learning Distributed Representations of Sentences from Unlabelled Data
Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566 · see citing works
Distributional Models of Word Meaning
Lenci, A. (2017) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-030514-125254 · Cited by 448 · see citing works
Self-citationMUC (Memory, Unification, Control) and beyond
Peter Hagoort, A. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00416 · Cited by 441 · see citing works
On the Meaning of Words and Dinosaur Bones: Lexical Knowledge Without a Lexicon
Elman, J. L. (2009) · Cognitive Science · doi:10.1111/j.1551-6709.2009.01023.x · Cited by 436 · see citing works
File Change Semantics and the Familiarity Theory of Definiteness
Heim, I. (1983) · De Gruyter eBooks · doi:10.1515/9783110852820.164 · Cited by 425 · see citing works
Integrating Verbs, Situation Schemas, and Thematic Role Concepts
Ferretti, T. R., McRae, K., & Hatherell, A. (2001) · Journal of Memory and Language · doi:10.1006/jmla.2000.2728 · Cited by 414 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04207 · Cited by 374 · see citing works
Vector Space Models of Word Meaning and Phrase Meaning: A Survey
Erk, K. (2012) · Language and Linguistics Compass · doi:10.1002/lnco.362 · Cited by 321 · see citing works
What you can cram into a single vector: Probing sentence embeddings for linguistic propertiesPreprint
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01070 · Cited by 315 · see citing works
A basis for generating expectancies for verbs from nouns
McRae, K., Hare, M. L., Elman, J. L., & Ferretti, T. R. (2005) · Memory & Cognition · doi:10.3758/bf03193221 · Cited by 248 · see citing works
Frege in Space: A Program for Compositional Distributional Semantics
Baroni, M., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Linguistic Issues in Language Technology · doi:10.33011/lilt.v9i.1321 · Cited by 241 · see citing works
In defense of abstract conceptual representations
Binder, J. R. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0909-1 · Cited by 241 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y. (2024) · arXiv (Cornell University) · doi:10.57702/tcr06vve · Cited by 228 · see citing works
The balance between memory and unification in semantics: A dynamic account of the N400
Baggio, G., & Peter Hagoort, A. (2011) · Language and Cognitive Processes · doi:10.1080/01690965.2010.542671 · Cited by 221 · see citing works
100 Million Words of English:The British National Corpus (BNC)
Leech, G. (1992) · Seoul National University Open Repository (Seoul National University) · Cited by 198 · see citing works
Bar, M., Aminoff, E., Mason, M. F., & Fenske, M. (2007) · Hippocampus · doi:10.1002/hipo.20287 · Cited by 186 · see citing works
Generalized event knowledge activation during online sentence comprehension
Metusalem, R., Marta Kutas, A., Urbach, T. P., Hare, M. L., McRae, K., & Elman, J. L. (2012) · Journal of Memory and Language · doi:10.1016/j.jml.2012.01.001 · Cited by 179 · see citing works
Hare, M. L., Jones, M. N., Thomson, C. E., Kelly, S., & McRae, K. (2009) · Cognition · doi:10.1016/j.cognition.2009.01.009 · Cited by 172 · see citing works
Effects of event knowledge in processing verbal arguments
Bicknell, K., Elman, J. L., Hare, M. L., McRae, K., & Marta Kutas, A. (2010) · Journal of Memory and Language · doi:10.1016/j.jml.2010.08.004 · Cited by 160 · see citing works
People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible
McRae, K., & Matsuki, K. (2009) · Language and Linguistics Compass · doi:10.1111/j.1749-818x.2009.00174.x · Cited by 142 · see citing works
Paczynski, M., & Kuperberg, G. R. (2012) · Journal of Memory and Language · doi:10.1016/j.jml.2012.07.003 · Cited by 137 · see citing works
A Simple, Similarity-based Model for Selectional Preferences
Erk, K. (2007) · Cited by 134 · see citing works
Probing for semantic evidence of composition by means of simple classification tasks
Ettinger, A., Elgohary, A., & Resnik, P. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2524 · Cited by 131 · see citing works
MUC (Memory, Unification, Control)
Peter Hagoort, A. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-407794-2.00028-6 · Cited by 109 · see citing works
A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences
Erk, K., Padó, S., & Padó, U. (2010) · Computational Linguistics · doi:10.1162/coli_a_00017 · Cited by 103 · see citing works
Representing Meaning with a Combination of Logical and Distributional Models
Beltagy, I., Roller, S., Cheng, P., Erk, K., & Mooney, R. J. (2016) · Computational Linguistics · doi:10.1162/coli_a_00266 · Cited by 69 · see citing works
Exploring Semantic Properties of Sentence Embeddings
Zhu, X., Li, T., & de Melo, G. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2100 · Cited by 65 · see citing works
How the brain processes different dimensions of argument structure complexity: Evidence from fMRI
Meltzer‐Asscher, A., Mack, J. E., Barbieri, E., & Thompson, C. K. (2015) · Brain and Language · doi:10.1016/j.bandl.2014.12.005 · Cited by 63 · see citing works
A practical and linguistically-motivated approach to compositional distributional semantics
Paperno, D., Pham, N. T., & Baroni, M. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-1009 · Cited by 63 · see citing works
Formal Distributional Semantics: Introduction to the Special Issue
Boleda, G., & Herbelot, A. (2016) · Computational Linguistics · doi:10.1162/coli_a_00261 · Cited by 55 · see citing works
Composing and Updating Verb Argument Expectations: A Distributional Semantic Model
Lenci, A. (2011) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 49 · see citing works
Self-citationJointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model
Pham, N. T., Kruszewski, G., Lazaridou, A., & Baroni, M. (2015) · doi:10.3115/v1/p15-1094 · Cited by 46 · see citing works
Neurocognitive mechanisms of verb argument structure processing
Thompson, C. K., & Meltzer‐Asscher, A. (2014) · Language faculty and beyond · doi:10.1075/lfab.10.07tho · Cited by 42 · see citing works
Logical Metonymy in a Distributional Model of Sentence Comprehension
Chersoni, E., Lenci, A., & Blache, P. (2017) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s17-1021 · Cited by 37 · see citing works
Self-citationPadó, U. (2007) · Publications of the UdS (Saarland University) · doi:10.22028/d291-23491 · Cited by 36 · see citing works
Meaning and the Dynamics of Interpretation
Kamp, H. (2013) · BRILL eBooks · doi:10.1163/9789004252882 · Cited by 34 · see citing works
Question-Answering with Grammatically-Interpretable Representations
Palangi, H., Smolensky, P., He, X., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12004 · Cited by 34 · see citing works
RELPRON: A Relative Clause Evaluation Data Set for Compositional Distributional Semantics
Rimell, L., Maillard, J., Polajnar, T., & Clark, S. C. (2016) · Computational Linguistics · doi:10.1162/coli_a_00263 · Cited by 33 · see citing works
Early sensitivity of left perisylvian cortex to relationality in nouns and verbs
Williams, A., Reddigari, S., & Liina Pylkkänen, A. (2017) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.04.029 · Cited by 33 · see citing works
Improving unsupervised vector-space thematic fit evaluation via role-filler prototype clustering
Greenberg, C., Sayeed, A., & Demberg, V. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1003 · Cited by 29 · see citing works
The mechanism of additive composition
Tian, R., 岡崎直観, Okazaki, N., & Inui, K. (2018) · Tokyo Tech Research Repository (Tokyo Institute of Technology) · Cited by 28 · see citing works
Event participant modelling with neural networks
Tilk, O., Demberg, V., Sayeed, A., Klakow, D., & Thater, S. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1017 · Cited by 27 · see citing works
Thematic fit evaluation: an aspect of selectional preferences
Sayeed, A., Greenberg, C., & Demberg, V. (2016) · doi:10.18653/v1/w16-2518 · Cited by 23 · see citing works
Measuring Thematic Fit with Distributional Feature Overlap
Santus, E., Chersoni, E., Lenci, A., et al. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1068 · Cited by 22 · see citing works
Self-citationAn Exploration of Semantic Features in an Unsupervised Thematic Fit Evaluation Framework
Sayeed, A., Shkadzko, P., & Demberg, V. (2015) · Italian Journal of Computational Linguistics · doi:10.4000/ijcol.298 · Cited by 20 · see citing works
Bar, M. (2011) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195395518.003.0010 · Cited by 19 · see citing works
Conceptual Versus Referential Affordance in Concept Composition
McNally, L., & Boleda, G. (2017) · Language, cognition and mind · doi:10.1007/978-3-319-45977-6_10 · Cited by 19 · see citing works
Large-scale evaluation of dependency-based DSMs: Are they worth the effort?
Lapesa, G., & Evert, S. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2063 · Cited by 18 · see citing works
Representing Verbs with Rich Contexts: an Evaluation on Verb Similarity
Chersoni, E., Santus, E., Lenci, A., et al. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1205 · Cited by 14 · see citing works
Self-citationModeling covert event retrieval in logical metonymy: probabilistic and distributional accounts
Zarcone, A., Utt, J., & Padó, S. (2012) · North American Chapter of the Association for Computational Linguistics · Cited by 13 · see citing works
Learning distributed event representations with a multi-task approach
Hong, X., Sayeed, A., & Demberg, V. (2018) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2002 · Cited by 11 · see citing works
Question-Answering with Grammatically-Interpretable RepresentationsPreprint
Palangi, H., Smolensky, P., He, X., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08432 · Cited by 10 · see citing works
The mechanism of additive composition
Tian, R., Okazaki, N., & Inui, K. (2017) · Machine Learning · doi:10.1007/s10994-017-5634-8 · Cited by 9 · see citing works
Event Knowledge in Sentence Processing: A New Dataset for the Evaluation of Argument Typicality
Vassallo, P., Chersoni, E., Santus, E., et al. (2018) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 9 · see citing works
Self-citationBridging Formal and Conceptual SemanticsSelected papers of BRIDGE-14
Balogh, K., Petersen, W., Filip, H., et al. (2017) · düsseldorf university press eBooks · Cited by 8 · see citing works
Expectation Vectors: A Semiotics Inspired Approach to Geometric Lexical-Semantic Representation
Washtell, J. (2010) · Cited by 6 · see citing works
Is Structure Necessary for Modeling Argument Expectations in Distributional Semantics?Preprint
Chersoni, E., Santus, E., Blache, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.00998 · Cited by 6 · see citing works
Self-citationElman, J. L. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9780262027236.003.0005 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
- Authors
- Chersoni, E., Santus, E., Pannitto, L., Lenci, A., Blache, P., & Huang, C.-R.
- Affiliations
- Hong Kong Polytechnic University (HK)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- University of Pisa (IT)
- Centre National de la Recherche Scientifique (FR)
- Corresponding author
- Chersoni, E.
- Venue
- Natural Language Engineering
- Publisher
- Cambridge University Press (CUP)
- Published
- 2019-07
- Volume
- 25
- Issue
- 4
- Pages
- 483-502
- ISSN
- 1351-3249 (ISSN-L)
- 1469-8110
- OpenAlex
- W2965758391
- Cited by
- 19
- References
- 77
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 5 · 2021: 6 · 2020: 2 · 2019: 3
- FWCI
- 1.24
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Natural Language Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- compositional distributional semantics, sentence meaning, sentence processing, Discourse Representation Theory, event knowledge, event attendees, word embeddings, sentence comprehension, semantic memory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
