The DOI matches its Crossref record.
Distributional vectors encode referential attributes
Open accessAbstract
Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany).In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants).In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes.We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound.Further analysis suggests that our model is able to "objectify" distributional representations for entities, anchoring them more firmly in the external world in measurable ways.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Probes were first introduced in NLP and have since been extensively explored in the field.
“Probes were first introduced in NLP (Köhn, 2015;Gupta et al., 2015) and have since been extensively explored in the field (Conneau et al., 2018;Tenney et al., 2019;Rogers et al., 2020;Gurnee et al., 2023;Peters et al., 2018;Burns et al., 2023;Marks & Tegmark, 2024).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Gupta et al., 2015
Section 2.3 (Attribute Prediction; page not provided) describes training logistic-regression models on word-vector dimensions to predict referential attributes, an early example of probing representations in NLP. The paper does not establish that probes were first introduced in NLP by this work. The sentence’s claim about subsequent extensive exploration is attached to other citations, not this one. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWhen attempting to localize a human-interpretable concept within the network, an intuitive approach is to 'probe' for it.
“When attempting to localize a human-interpretable concept within the network, an intuitive approach is to 'probe' for it (Köhn, 2015;Gupta et al., 2015;Alain & Bengio, 2017;Ettinger et al., 2016).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Gupta et al., 2015
The Introduction describes training a regression model on distributional vectors to retrieve interpretable attributes of countries and cities, which is an example of probing a representation. The paper does not attempt to localize a concept within a network; it tests what can be recovered from word vectors. (LLM, conf=0.83)
Experimental — accuracy not yet measuredRelative coordinates can be approximately recovered with simple techniques like multidimensional scaling, co-occurrence statistics, or probing word embeddings.
“relative coordinates can be approximately recovered with simple techniques like multidimensional scaling, co-occurrence statistics, or probing word embeddings (Louwerse & Zwaan, 2009;Mikolov et al., 2013a;Gupta et al., 2015;Konkol et al., 2017).”
From Language Models Represent Space and Time (2023) · cites this paper as Gupta et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 26
- Marco Baroni and Alessandro Lenci. 2010. Dis- tributional Memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673-721.
- Greg Carlson. 2009. Generics and concepts. In Jefrey Pelletier, editor, Kinds, Things and Stuff, pages 16- 35. Oxford University Press, Oxford, UK.
- Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723-763.
- Katrin Erk. 2012. Vector space models of word mean- ing and phrase meaning: A survey. Language and Linguistics Compass, 6(10):635-653.
- Luana Fȃgȃrȃs ¸an, Eva Maria Vecchi, and Stephen Clark. 2015. From distributional semantics to fea- ture norms: grounding semantic models in human perceptual data. In Proceedings of IWCS, pages 52- 57, London, UK.
- André Freitas and Edward Curry. 2014. Natural language queries over heterogeneous linked data graphs: A distributional-compositional semantics approach. In Proceedings of IUI, pages 279-288, Haifa, Israel.
- André Freitas, João Carlos Pereira da Silva, Edward Curry, and Paul Buitelaar. 2014. A distributional semantics approach for selective reasoning on com- monsense graph knowledge bases. In Proceedings of NLDB, pages 21-32, Montpellier, France.
- Alinda Friedman, Dennis Kerkman, and Norman Brown. 2002. Spatial location judgments: A cross- national comparison of estimation bias in subjective North American geography. Psychonomic Bulletin & Review, 9(3):615-623.
- Andrea Frome, Greg Corrado, Jon Shlens, Samy Ben- gio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A deep visual-semantic embedding model. In Proceedings of NIPS, pages 2121-2129, Lake Tahoe, NV.
- Aurélie Herbelot. 2015. Mr Darcy and Mr Toad, gen- tlemen: distributional names and their kinds. In Pro- ceedings of IWCS, pages 151-161, London, UK.
- Felix Hill, Roi Reichart, and Anna Korhonen. 2014. Multi-modal models for concrete and abstract con- cept meaning. Transactions of the Association for Computational Linguistics, 2:285-296.
- Brendan Johns and Michael Jones. 2012. Perceptual inference through global lexical similarity. Topics in Cognitive Science, 4(1):103-120.
- Willis F. Kern and James R. Bland. 1948. Solid Men- suration with Proofs. Wiley, New York, 2nd edition.
- Angeliki Lazaridou, Elia Bruni, and Marco Baroni. 2014. Is this a wampimuk? Cross-modal map- ping between distributional semantics and the visual world. In Proceedings of ACL, pages 1403-1414, Baltimore, MD.
- Chris Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Re- trieval. Cambridge University Press, Cambridge, UK.
- Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of NAACL, pages 746-751, Atlanta, Georgia.
- Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf- sky. 2009. Distant supervision for relation extrac- tion without labeled data. In Proceedings of ACL, pages 1003-1011, Singapore.
- Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012. Selecting corpus-semantic models for neu- rolinguistic decoding. In Proceedings of *SEM, pages 114-123, Montreal, Canada.
- Gregory Murphy. 2002. The Big Book of Concepts. MIT Press, Cambridge, MA.
- Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161-199.
- Mark Palatucci, Dean Pomerleau, Geoffrey Hinton, and Tom Mitchell. 2009. Zero-shot learning with se- mantic output codes. In Proceedings of NIPS, pages 1410-1418, Vancouver, Canada.
- Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. 2015. How well do distributional mod- els capture different types of semantic knowledge? In Proceedings of ACL (Volume 2: Short Papers), pages 726-730, Beijing, China.
- Richard Socher, Danqi Chen, Christopher Manning, and Andrew Ng. 2013a. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NIPS, pages 926-934, Lake Tahoe, NV.
- Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. 2013b. Zero-shot learning through cross-modal transfer. In Proceedings of NIPS, pages 935-943, Lake Tahoe, NV.
- Peter Turney and Patrick Pantel. 2010. From fre- quency to meaning: Vector space models of se- mantics. Journal of Artificial Intelligence Research, 37:141-188.
- Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for re- lation extraction. In Proceedings of EMNLP, pages 1366-1371, Seattle, WA.
Cited by
Sorted by citations
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
Language Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46
Probing Pretrained Models of Source Codes
Troshin, S., & Chirkova, N. (2022) · Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2022.blackboxnlp-1.31 · Cited by 23
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
Metadata
- Authors
- Gupta, A., Boleda, G., Baroni, M., & Padó, S.
- Affiliations
- University of Stuttgart (DE)
- Venue
- Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
- Published
- 2015
- Pages
- 12-21
- OpenAlex
- W2250382531
- Cited by
- 79
- References
- 0
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 6 · 2023: 6 · 2022: 9 · 2021: 6 · 2020: 11 · 2019: 14 · 2018: 6 · 2017: 7 · 2016: 6 · 2015: 1
- FWCI
- 3.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-sa)
- Institutional Research Information System (Università degli Studi di Trento) (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-sa)
- Funders
- European Commission (655577, 283554, HORIZON2020)
- Deutsche Forschungsgemeinschaft (SFB732)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- distributional semantics, word embeddings, attribute prediction, entity attributes, regression models, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
