Do Language Embeddings capture Scales?
Open accessAbstract
Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense and factual knowledge.One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects.We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning.We identify contextual information in pre-training and numeracy as two key factors affecting their performance, and show that a simple method of canonicalizing numbers can have a significant effect on the results. 1 * Both authors contributed equally.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LMs structure their representations of perceptual concepts such as size and color in a way that credibly mirrors the physical world.
“LMs have been shown to structure their representations of perceptual concepts such as size and color in a way that credibly mirrors the physical world (Abdou et al., 2021;Patel and Pavlick, 2022;Zhang et al., 2020;Ilharco et al., 2021;i.a.).”
From Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024) · cites this paper as Zhang et al., 2020
The Results and Conclusion report that object embeddings contain significant information about scales such as length and mass, supporting the size-related part of the claim. The paper does not establish the color-related part, and its Conclusion calls the signals weak and says the models remain far from common-sense scale understanding. Page numbers are not provided in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredUsing commonsense knowledge can help to gain better numeral representation.
“Currently, some works [193, 246, 315] point out that using commonsense knowledge can help to gain better numeral representation.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [315]
Section 4 and the Introduction describe using scientific notation for numbers during pretraining to improve representations of scalar magnitude. The Conclusion discusses better numeracy as a route toward commonsense scale understanding. None of these passages says that using commonsense knowledge improves numeral representations; I also searched for that claim directly. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe correctness of numerals in generated text, such as date, quantities and scalars, is important for readers.
“However, the correctness of numerals in generated text, such as date, quantities and scalars are important for readers [246, 315, 321].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [315]
Section 4 examines numerical representations for scalar probing, and Section 6 reports results for scale prediction. Neither passage addresses the correctness of numerals in generated text or its importance to readers; searches for those points and for dates and quantities found no relevant passage in our copy. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Pixel Recurrent Neural NetworksPreprint
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1601.06759 · Cited by 3,111 · see citing works
A metric for distributions with applications to image databases
Rubner, Y., Tomasi, C., & Guibas, L. (2002) · International Conference on Computer Vision · doi:10.1109/iccv.1998.710701 · Cited by 1,765 · see citing works
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393 · see citing works
Does String-Based Neural MT Learn Source Syntax?
Shi, X., Padhi, I., & Knight, K. K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1159 · Cited by 302 · see citing works
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.04487 · Cited by 251
Do NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227 · see citing works
Transferable Adversarial Training: A General Approach to Adapting Deep Classifiers
Liu, H., Long, M., Wang, J., & Jordan, M. I. (2019) · International Conference on Machine Learning · Cited by 209 · see citing works
Information-Theoretic Probing for Linguistic Structure
Pimentel, T., Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., & Cotterell, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.420 · Cited by 173 · see citing works
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 166 · see citing works
Probing for semantic evidence of composition by means of simple classification tasks
Ettinger, A., Elgohary, A., & Resnik, P. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2524 · Cited by 131 · see citing works
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.557 · Cited by 122 · see citing works
Verb Physics: Relative Physical Knowledge of Actions and Objects
Forbes, M., & Choi, Y. (2017) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p17-1025 · Cited by 80 · see citing works
Temporal Common Sense Acquisition with Minimal Supervision
Zhou, B., Qiang, N., Khashabi, D., & Roth, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.678 · Cited by 78 · see citing works
Self-citationHow Large Are Lions? Inducing Distributions over Quantitative Attributes
Elazar, Y., Mahabal, A., Ramachandran, D., Bedrax-Weiss, T., & Roth, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1388 · Cited by 58 · see citing works
Self-citationDesigning and Interpreting Probes with Control TasksPreprint
Hewitt, J., & Liang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03368 · Cited by 40 · see citing works
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · see citing works
Baringhaus, L., & Henze, N. (2017) · Journal of nonparametric statistics · doi:10.1080/10485252.2017.1285029 · Cited by 35 · see citing works
Extracting Commonsense Properties from Embeddings with Limited Human Guidance
Yang, Y., Birnbaum, L., Wang, J., et al. (2018) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p18-2102 · Cited by 23 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Deep contextualized word representations
Peters, M. E., Neumann, M., Iyyer, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.05365
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · arXiv · doi:10.48550/arxiv.2005.00683
Information-Theoretic Probing for Linguistic Structure
Pimentel, T., Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., & Cotterell, R. (2020) · arXiv · doi:10.48550/arxiv.2004.03061
Flagged first, then by citations
Research works printed but not linked: 3
- Julian McAuley Jianmo Ni, Jiacheng Li. 2019. Jus- tifying recommendations using distantly-labeled re- views and fined-grained aspects. In Empirical Meth- ods in Natural Language Processing (EMNLP).
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119.
- Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel recurrent neural net- works. In International Conference on Machine Learning, pages 1747-1756.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data
Zhao, Y., Li, Y., Li, C., & Zhang, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.454 · Cited by 71 · see citing works
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58 · see citing works
Wulff, P., Buschhüter, D., Westphal, A., Mientus, L., Nowak, A., & Borowski, A. (2022) · Journal of Science Education and Technology · doi:10.1007/s10956-022-09969-w · Cited by 52 · see citing works
NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-Task Financial Forecasting
Yang, L., Li, J., Dong, R., Zhang, Y., & Smyth, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21414 · Cited by 44 · see citing works
Injecting the BM25 Score as Text Improves BERT-Based Re-rankers
Askari, A., Abolghasemi, A., Pasi, G., Kraaij, W., & Verberne, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_5 · Cited by 38 · see citing works
ML-LJP: Multi-Law Aware Legal Judgment Prediction
Liu, Y., Wu, Y., Zhang, Y., Sun, C., Lü, W., Wu, F. Q., & Kuang, K. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591731 · Cited by 32 · see citing works
Precision information extraction for rare disease epidemiology at scale
Kariampuzha, W. Z., Alyea, G., Qu, S., Sanjak, J. S., Mathé, E. A., Sid, E., Chatelaine, H., Yadaw, A. S., Xu, Y., & Zhu, Q. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04011-y · Cited by 22 · see citing works
TaskLAMA: Probing the Complex Task Understanding of Language Models
Yuan, Q., Kazemi, M., Xu, X., Noble, I., Imbrasaitė, V., & Ramachandran, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29918 · Cited by 15 · see citing works
Self-citationProbing BERT for Ranking Abilities
Wallat, J., Beringer, F., Anand, A., & Anand, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_17 · Cited by 14 · see citing works
Injecting the score of the first-stage retriever as text improves BERT-based re-rankers
Askari, A., Abolghasemi, A., Pasi, G., Kraaij, W., & Verberne, S. (2024) · Discover Computing · doi:10.1007/s10791-024-09435-8 · Cited by 10 · see citing works
Introducing Semantic Information for Numerical Attribute Prediction over Knowledge Graphs
Xue, B., Li, Y., & Zou, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19433-7_1 · Cited by 8 · see citing works
UniMath: A Foundational and Multimodal Mathematical Reasoner
Liang, Z., Yang, T., Zhang, J., & Zhang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.440 · Cited by 7 · see citing works
Self-citationHolmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Waldis, A., Perlitz, Y., Choshen, L., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 9 · see citing works
Impression-Aware Recommender Systems
Maurera, F. B. P., Dacrema, M. F., Castells, P., & Cremonesi, P. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3712292 · Cited by 9 · see citing works
New Frontiers of Information Extraction
Chen, M., Huang, L., Li, M., Zhou, B. Z., Ji, H., & Roth, D. (2022) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2022.naacl-tutorials.3 · Cited by 5 · see citing works
Self-citationMining Numbers in Text: A Survey
Yoshida, M. W., & Kita, K. (2021) · IntechOpen eBooks · doi:10.5772/intechopen.98540 · Cited by 4 · see citing works
A survey on mathematical reasoning and optimization with large language models
Forootani, A., Aliabadi, D. E., & Thrän, D. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200712 · Cited by 13 · see citing works
Fine-grained entity disambiguation through numeric pattern awareness in transformer models
Jang, J., Kim, S.-M., Moon, H., Shin, S.-H., Yun, M., & Wiseman, C. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01936-3 · Cited by 1 · see citing works
Bueno, M. C., Gemmell, C., Dalton, J., Lotufo, R., & Nogueira, R. F. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.3 · Cited by 1 · see citing works
Reading Preference Analysis Through Eye-Tracking and Large Language Models
Ying, S., Zhang, D., Wu, H., Yu, C., Liu, Y., & He, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1071-6_7
- Authors
- Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
- Published
- 2020
- Pages
- 292-299
- OpenAlex
- W4212865381
- Cited by
- 22
- References
- 16
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 3 · 2023: 6 · 2022: 8 · 2021: 1
- FWCI
- 1.43
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- pre-trained language models, language embedding, numerical knowledge, commonsense reasoning, numeracy, contextual information, linguistic knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
