Do NLP Models Know Numbers? Probing Numeracy in Embeddings
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, Matt Gardner. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 We train a probing model to decode a number from its word embedding over a random 80% of the integers from [-500, 500], e.g., '71' → 71.0. We plot the model's predictions for all numbers from [-2000, 2000]. The model accurately decodes numbers within the training range (in blue), i.e., pre-trained embeddings like GloVe and BERT capture numeracy. However, the probe fails to extrapolate to larger numbers (in red). The Char-CNN (e) and Char-LSTM (f) are trained jointly with the probing model.](https://figures.cochat.ai/paper-figures/153bcb5dac394f6194eb8e7aa3ffa6ae/do-nlp-models-know-numbers-probing-numeracy-in-embeddings-figure-1.png)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“, 2018), BERT (Devlin et al., 2019), and GloVe (Pennington et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“, 2018), BERT (Devlin et al., 2019), and GloVe (Pennington et al.”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“We find that all widely-used pre-trained embed-dings, e.g., ELMo (Peters et al., 2018), BERT (De-vlin et al., 2019), and GloVe (Pennington et al., 2014), capture numeracy: number magnitude is present in the embeddings, even for numbers in the thousands.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“We find that all widely-used pre-trained embed-dings, e.g., ELMo (Peters et al., 2018), BERT (De-vlin et al., 2019), and GloVe (Pennington et al., 2014), capture numeracy: number magnitude is present in the embeddings, even for numbers in the thousands.”
MethodSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“The dataset’s input-output format is a superset of SQuAD (Rajpurkar et al., 2016): the an... JaMarcus Russell completed a 91-yard touchdown pass to rookie wide receiver Chaz Schilens .”
MethodUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 2×
“, influence functions (Koh and Liang, 2017; Brunet et al., 2019)).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“As in typical span selection models (Seo et al., 2017), an LSTM reads the list of token embeddings, and a weight matrix and softmax function assign a probability to each index using the model’s hidden state.”
BackgroundMethodGraves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 2×
“Synthetic Numerical Tasks Similar to our synthetic numerical reasoning tasks, other work considers sorting (Graves et al., 2014), counting (Weiss et al., 2018), or decoding tasks (Trask et al., 2018).”
MethodAssessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Dupoux, E., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 737 · cited 1×
“Khandel-wal et al. (2018) show how language models leverage context, while Linzen et al. (2016) demonstrate that language models understand subject-verb agreement.”
BackgroundLinguistic Knowledge and Transferability of Contextual Representations
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., & Smith, N. A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1112 · Cited by 621 · cited 2×
“Probing Models Our probes of numeracy parallel work in understanding the linguistic capabilities (literacy) of neural models (Conneau et al., 2018; Liu et al., 2019).”
BackgroundSelf-citation
Research works printed but not linked: 5
- [1]1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Finegrained anal- ysis of sentence embeddings using auxiliary predic- tion tasks. In ICLR.
- [13]13. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguis- tic knowledge and transferability of contextual rep- resentations. In NAACL.
- [14]14. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Ad- vances in pre-training distributed word representa- tions. In LREC.
- [24]24. Andrew Trask, Felix Hill, Scott Reed, Jack W. Rae, Chris Dyer, and Phil Blunsom. 2018. Neural arith- metic logic units. In NeurIPS.
- [29]29. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehen- sion. In ICLR.
Cited by
Items 101–200 of 286
Shen, K., & Kejriwal, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.01258 · Cited by 16
“Some of these experiments have shown that, even though BERT encodes information about entity types, relations, semantic roles, and proto-roles well, it struggles with the representations of numbers (Wallace et al. 2019b) and is also brittle to named entity replacements (Balasubramanian et al. 2020).”
BackgroundCombining BERT with numerical variables to classify injury leave based on accident description
Ramos, P. M. S., Macêdo, J. B., Maior, C. B. S., Moura, M. D. C., & Lins, I. D. (2022) · Proceedings of the Institution of Mechanical Engineers Part O Journal of Risk and Reliability · doi:10.1177/1748006x221140194 · Cited by 15
“46 Thus, an alternative approach is to concatenate the non-textual variables for BERT embedding and classify the resulting vector through a neural network.”
Cancer Registry Coding via Hybrid Neural Symbolic Systems in the Cross-Hospital Setting
Dai, H.-J., Yang, Y., Wang, T., Lin, Y.-J., Lu, P.-J., Wu, C.-Y., Chang, Y., Lee, Y.-Q., Zhang, Y.-C., Hsu, Y.-C., et al. (2021) · IEEE Access · doi:10.1109/access.2021.3099175 · Cited by 15
“Wallace et al. [30], we noticed that the inclusion of the pre-trained word representation features enables the CRF model to learn the numeracy for integers and their corresponding word forms ( e.g. 42 and forty-two).”
MethodSelf-citationExploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning
Wang, C., Zheng, B., Niu, Y., & Zhang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_61 · Cited by 27
How to Query Language Models?Preprint
Adolphs, L., Dhuliawala, S., & Hofmann, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01928 · Cited by 16
“…individual relations for this data, we see that the performance is largely driven by predicting a person’s birth and death place; the birth-date relation doesn’t play a significant role because BERT is incapable of accurately predicting numbers (i.e., dates) (Lin et al., 2020; Wallace et al., 2019).”
BackgroundKitchenScale: Learning to predict ingredient quantities from recipe contexts
Choi, D., Gim, M., Badreddine, S., Kim, H., Park, D., & Kang, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120041 · Cited by 18
“Various tasks including probing numerical knowledge (Lin et al., 2020; Yamane et al., 2020; Elazar et al., 2019) and probing numerical embedding (Wallace et al., 2019) showed the possibility of PLMs achieving numeracy.”
BackgroundMethodInfluentialMitigating Temporal Misalignment by Discarding Outdated Facts
Zhang, M., & Choi, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.879 · Cited by 39
Probing for Multilingual Numerical Understanding in Transformer-Based Language Models
Johnson, D., Mak, D., Barker, A., & Loessberg-Zahl, L. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.18 · Cited by 13
Subsentence Extraction from Text Using Coverage-Based Deep Learning Language Models
Lim, J., Sa, I., Ahn, H. S., Gasteiger, N., Lee, S. J., & MacDonald, B. (2021) · Sensors · doi:10.3390/s21082712 · Cited by 13
“Within this context and drawing inspiration from [37], we propose a recursive approach that estimates the length of subsentence in an input sentence.”
BackgroundMeasurement Extraction with Natural Language Processing: A Review
Göpfert, J., Kuckertz, P., Weinand, J. M., Kotzur, L., & Stolten, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.161 · Cited by 26
“Character-level embeddings, for example, outperform word- and subword-level methods (Wallace et al., 2019).”
BackgroundMethodInfluentialNT5?! Training T5 to Perform Numerical ReasoningPreprint
Yang, P., Chen, Y.-T., Chen, Y., & Cer, D. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07307 · Cited by 13
“For works analyzing the mathematical reasoning ability of models over text refer to Wallace et al. (2019); Ravichander et al. (2019).”
BackgroundLearn Your Tokens: Word-Pooled Tokenization for Language Modeling
Thawani, A., Ghanekar, S., Zhu, X., & Pujara, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.662 · Cited by 20
Arithmetic with language models: From memorization to computation
Maltoni, D., & Ferrara, M. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106550 · Cited by 21
“The blue cells show that correlation remains quite similar across the encoder layers as if the encoder was not performing any significant computation (this is confirmed in Section 5 where by totally removing the encoder we achieve similar results).”
MethodResultInjecting the score of the first-stage retriever as text improves BERT-based re-rankers
Askari, A., Abolghasemi, A., Pasi, G., Kraaij, W., & Verberne, S. (2024) · Discover Computing · doi:10.1007/s10791-024-09435-8 · Cited by 15
“Wallace et al. [16] analyze the ability of BERT models to work with numbers and come to the conclusion that the models capture numeracy and are able to do numerical reasoning; however the models appeared to struggle with interpreting floats.”
What Do You See in this Patient? Behavioral Testing of Clinical NLP Models
van Aken, B., Herrmann, S., & Löser, A. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.7 · Cited by 14
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite TasksPreprint
Mishra, S. R., Mitra, A., Varshney, N., Sachdeva, B., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.08516 · Cited by 15
“, 2019] have become standard tools across several language understanding tasks, they can not perform complex forms of reasoning, specifically numerical reasoning [Wallace et al., 2019].”
BackgroundKartashov, N., & Vlassis, N. N. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.117742 · Cited by 14
TM-generation model: a template-based method for automatically solving mathematical word problems
Lee, D., Ki, K. S., Kim, B., & Gweon, G. (2021) · The Journal of Supercomputing · doi:10.1007/s11227-021-03855-9 · Cited by 11
“Moreover, Wallace et al. [18] argued that language models such as BERT can have numeracy and the ability to understand numbers.”
BackgroundDoes BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms
Tan, M., & Jiang, J. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_156 · Cited by 27
“Examples of probing tasks include POS tagging and parsing (He-witt and Liang, 2019; Wu et al., 2020) as well as semantic reasoning tasks such as understanding numbers (Wallace et al., 2019).”
BackgroundSurvey on Tabular Data Privacy and Synthetic Data Generation in Industry 4.0
Koubeissy, H., Amine, A., Kamradt, M., & Makhoul, A. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06823-5 · Cited by 11
Have You Seen That Number? Investigating Extrapolation in Question Answering Models
Kim, J.-H., Hong, G., Kim, K.-M., Kang, J., & Myaeng, S.-H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.563 · Cited by 26
“Although Wallace et al. (2019) reveals that NAQANet (Dua et al., 2019) struggles to deal with numbers outside the training range, showing a drop in performance in the extrapolation setting, they simply treat it as one of the failure modes in NAQANet and provide no further analysis on this alarming…”
BackgroundMethodMasked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context
Spokoyny, D., Lee, I., Zhao, J., & Berg-Kirkpatrick, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.2 · Cited by 12
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · arXiv · doi:10.48550/arxiv.2106.09231 · Cited by 10
“semantic knowledge (Tenney et al., 2019; Wallace et al., 2019; Ettinger, 2020) and world knowledge (Davison et al.”
BackgroundNumeric Magnitude Comparison Effects in Large Language Models
Shah, R. C., Marupudi, V., Koenen, R., Bhardwaj, K., & Varma, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.383 · Cited by 28
Academic expert finding using BERT pre-trained language model
Mannix, I. A., & Yulianti, E. (2024) · International Journal of Advances in Intelligent Informatics · doi:10.26555/ijain.v10i2.1497 · Cited by 9
“This is based on a study in [24], which revealed that the BERT model is still not as proficient as the BM25 model in capturing exact matches and based on the study [28], it was found that BERT can capture numeracy information.”
MethodZhou, F., Parrish, R., Afzal, M., Saha, A., Haynes, R. B., Iorio, A., & Lokker, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104825 · Cited by 9
Introducing Semantic Information for Numerical Attribute Prediction over Knowledge Graphs
Xue, B., Li, Y., & Zou, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19433-7_1 · Cited by 13
What do Large Language Models Learn beyond Language?
Madasu, A., & Srivastava, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.516 · Cited by 11
SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems
Emami, P., Li, Z., Sinha, S., & Nguyen, T. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2405.19653 · Cited by 10
“Limitations : Current BERT-style tokenizers struggle with numerical values [54]; for one example, they interpolate poorly to unseen numbers (Fig.”
MachineLearningLM: Scaling Many-shot In-context Learning via Continued PretrainingPreprint
Dong, H., Zhang, P., Lu, M., Shen, Y., & Ke, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.06806 · Cited by 10
“|9 where 11 is bigger than 9 while 1.11 is smaller than 1.9)) rather than numerically (Spithourakis & Riedel, 2018; Wallace et al., 2019; Golkar et al., 2024; Singh & Strouse, 2024; Hugging Face, 2024).”
InfluentialA Pretraining Numerical Reasoning Model for Ordinal Constrained Question Answering on Knowledge Base
Feng, Y., Zhang, J., He, G., Zhao, W. X., Liu, L., Liu, Q., Li, C., & Chen, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.159 · Cited by 13
“Numerical Reasoning has been studied for various tasks such as word embedding (Naik et al., 2019; Wallace et al., 2019), arithmetic word problems (AWP) (Wang et al.”
MethodTowards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings
Sengupta, S., Heaton, C., Cui, S., Sarkar, S., & Mitra, P. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821824 · Cited by 11
“Numbers were excluded since it is known that BERT does not deal with numbers well [27].”
Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models
Arakelyan, E., Liu, Z., & Augenstein, I. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.27 · Cited by 24
Do Language Models Understand Measurements?
Park, S., Ryu, S., & Choi, E. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.128 · Cited by 19
Learning Numeracy: A Simple Yet Effective Number Embedding Approach Using Knowledge Graph
Duan, H., Yang, Y., & Tam, K. Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.221 · Cited by 16
“In task 2, we evaluate our method on several numeracy-tasks including List Maximum, Decoding, and Addition of numbers (Wallace et al., 2019).”
BackgroundMethodInfluentialPretrained Language Model based Web Search Ranking: From Relevance to SatisfactionPreprint
Li, C., Wang, X., Li, D., Liu, Y., Lu, Y., Wang, S., Cheng, Z., Gu, S., & Yin, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.01599 · Cited by 12
“In general, we leverage the capabilities of PLMs on comprehending both text and numbers [57], and modularize each dimension of user satisfaction as an input field in SAT-Ranker.”
On self-supervision in historical handwritten document segmentation
Baloun, J., Prantl, M., Lenc, L., Martínek, J., & Král, P. (2025) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-025-00538-6 · Cited by 8
“To address this, we draw inspiration from the probing techniques used with large language models, as demonstrated in [29], where a probing model is trained on top of a frozen language model for a downstream task.”
Learning to Generate Correct Numeric Values in News Headlines
Chu, J., Chen, C.-C., Huang, H., & Chen, H. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366424.3382676 · Cited by 7
“As shown in Figure 1 of three news article-headline pairs, the common processes for values in articles to go through are copy (2.6% in the first pair), round (1.7% from 1.67% in the second pair), and paraphrase (6K from 5981 in the third pair), which might require numerical reasoning [3].”
BackgroundInfluentialBueno, M. C., Gemmell, C., Dalton, J., Lotufo, R., & Nogueira, R. F. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.3 · Cited by 15
Investigating Execution Trace Embedding for Test Case Prioritization
Jabbar, E., Hemmati, H., & FELDT, R. H. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00036 · Cited by 7
“Research [15] have shown that most embedding techniques cannot work well with numerical data, especially large values.”
BackgroundWhat BERT Based Language Model Learns in Spoken Transcripts: An Empirical Study
Kumar, A., Sundararaman, M. N., & Vepa, J. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.25 · Cited by 10
Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems
Xu, J., Zhou, M., He, X., Shi, H., & Zhang, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.542 · Cited by 12
How Can We Know When Language Models KnowPreprint
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2020) · arXiv (Cornell University) · Cited by 11
Test2Vec: An Execution Trace Embedding for Test Case PrioritizationPreprint
Jabbar, E., Zangeneh, S., Hemmati, H., & FELDT, R. H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.15428 · Cited by 9
“Research [78] have shown that most NLP Manuscript submitted to ACM Fig.”
FinIR 2020: The First Workshop on Information Retrieval in Finance
Feng, F., Luo, C., He, X., Liu, Y., & Chua, T. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401462 · Cited by 6
“This wave of research has also aroused great interest in training language models to properly encode financial documents especially the numerical information [14].”
BackgroundMining Numbers in Text: A Survey
Yoshida, M., & Kita, K. (2021) · Information Systems · doi:10.5772/intechopen.98540
Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
Jang, M., Mtumbuka, F., & Lukasiewicz, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.156 · Cited by 13
“…when generating representations (Pham et al., 2021; Gupta et al., 2021; Sinha et al., 2021a), struggle to comprehend number-related representations (Wallace et al., 2019; Lin et al., 2020; Nogueira et al., 2021), and display a lack of semantic content understanding (Ravichander et al., 2020;…”
You Are My Type! Type Embeddings for Pre-trained Language Models
Saeed, M., & Papotti, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.336 · Cited by 10
“We believe this is due to BERT’s inability to precisely capture numeracy (Wallace et al., 2019).”
BackgroundReinforcement Learning for IoT Interoperability
Kotstein, S., & Decker, C. (2019) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c.2019.00010 · Cited by 7
Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
Clark, C., Salvador, J., Schwenk, D., Bonafilia, D., Yatskar, M., Kolve, E., Herrasti, A., Choi, J., Mehta, S., Skjonsberg, S., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.141 · Cited by 13
“This gives the model some prior knowledge of what the icons are, and a sense of ordering among the quantized tokens (Wallace et al., 2019).”
BackgroundInformation-Theoretic Measures of Dataset Difficulty.Preprint
Ethayarajh, K., Choi, Y., & Swayamdipta, S. (2021) · arXiv (Cornell University) · Cited by 9
McMaster, C., Liew, D. F. L., & Pires, D. E. V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.14920 · Cited by 6
“Wallace et al. [7] demonstrated that language models trained to decode token embeddings of numbers (i.e. predict the floating point number from its embedding) can only accurately recapture within the range of numbers they were trained on.”
How Large Language Models Transform Urban Planning and Shape Tomorrow’s Cities?
Agrawal, K., & Göktaş, P. (2025) · The Springer series in applied machine learning · doi:10.1007/978-3-031-86039-3_8 · Cited by 9
Vulnerability detection based on transformer and high‐quality number embedding
Cao, Y., Dong, Y., & Peng, J. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8292 · Cited by 6
Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models
Derby, S., Devereux, B. J., & Miller, P. C. (2021) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2021.cmcl-1.25 · Cited by 7
“A number of these studies rely on probing techniques, where supervised models are trained to predict specific linguistic phenomena from model activations (Adi et al., 2016; Wallace et al., 2019; Tenney et al., 2019b; Hewitt and Liang, 2019).”
BackgroundProbing Numeracy and Logic of Language Models of Code
Baltaji, R., & Thakkar, P. (2023) · doi:10.1109/intense59150.2023.00006 · Cited by 6
“However, [5] has shown these models can also reason about numerical properties of text.”
BackgroundMethodInfluentialImproving Numeracy by Input Reframing and Quantitative Pre-Finetuning Task
Chen, C.-C., Takamura, H., Kobayashi, I., & Miyao, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.4 · Cited by 6
“Some previous studies (Spithourakis and Riedel, 2018; Chen et al., 2019) have discussed the prediction of the masked number tasks, while others (Wallace et al., 2019; Naik et al., 2019; Zhang et al., 2020) have explored numeracy from the perspective of embedding properties.”
BackgroundDetecting context-based in-claim numerals in Financial Earnings Conference Calls
Ghosh, S., & Naskar, S. K. (2022) · International Journal of Information Technology · doi:10.1007/s41870-022-00952-7 · Cited by 5
“It has also been mentioned by Wallace et al. in the paper [30].”
MethodSeeing past words: Testing the cross-modal capabilities of pretrained V&L modelsPreprint
Pârcălăbescu, L., Albert Gatt, A., Frank, A., & Calixto, I. (2020) · arXiv (Cornell University) · Cited by 5
Random Embeddings and Linear Regression can Predict Protein FunctionPreprint
Lu, T., Lu, A. X., & Moses, A. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14661 · Cited by 7
“Further, some random embedders do better than others, which works have attributed to differences in model architecture [20].”
NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision TreesPreprint
Ye, W., Liu, H., & Jiang, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.28587 · Cited by 6
“These formulations expose two distinct concerns: numerical training adds a mapping that must generalize beyond its supervision, while restricting inputs to isolated numbers removes the linguistic context needed to interpret a full question (Wallace et al., 2019).”
Memorization of Named Entities in Fine-Tuned BERT Models
Diera, A., Lell, N., Garifullina, A., & Scherp, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40837-3_16 · Cited by 6
“The lower values in the Money and Cardinal types reinforce the findings that the subword tokenization in BERT is a suboptimal method to encode numerical values [55].”
Veerendranath, V., Shah, V., & Ghate, K. (2024) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2024.semeval-1.211 · Cited by 6
A Sequence-to-sequence Approach for Numerical Slot-filling Dialog Systems
Shi, H. (2020) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.34 · Cited by 4
“This observation is consist with one recent paper, which also reported the good performance of neural models on addition calculation within the training range (Wallace et al., 2019).”
MethodMy-Trac: System for Recommendation of Points of Interest on the Basis of Twitter Profiles
Rivas, A., González‐Briones, A., Cea-Morán, J. J., Prat-Pèrez, A., & Corchado, J. M. (2021) · Electronics · doi:10.3390/electronics10111263 · Cited by 4
Srivastava, V. P., Pilli, S., Bhat, S., Pedanekar, N., & Karande, S. S. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.12 · Cited by 5
“Experiments in (Wallace et al., 2019) are aimed to investigate the numerical reasoning capabilities of an LM. Similarly, (Petroni et al., 2019) presents an in-depth analysis of relational knowledge present in pre-trained LMs. Penha and Hauff (2020) probe the contextual LMs (BERT and RoBERTa) for…”
BackgroundLearning Multi-Step Reasoning by Solving Arithmetic Tasks
Wang, T., & Lü, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.106 · Cited by 15
LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI
Tarunesh, I., Aditya, S., & Choudhury, M. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09691-y · Cited by 5
“All models struggle with templates pertaining to addition and subtraction (often showing label bias) (Naik et al., 2018; Pikekos et al., 2021; Wallace et al., 2019).”
Beyond Topicality: Including Multidimensional Relevance in Cross-encoder Re-ranking
Upadhyay, R., Askari, A., Pasi, G., & Viviani, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_16 · Cited by 4
Sakamoto, T., & Aizawa, A. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.14 · Cited by 4
“However, unlike the case of words, sub-words derived from numerals often have little relationship to the meaning of the original numerals, which can make it difficult to understand the meaning of numerals (Wallace et al., 2019).”
BackgroundEnhancing Knowledge Bases with Quantity Facts
Ho, V. T., Stepanova, D., Milchevski, D., Strötgen, J., & Weikum, G. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511932 · Cited by 4
“In [21, 30, 38, 42, 44] word embeddings are adapted to account for numerical expressions.”
BackgroundAd astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task
Tikhonova, M., Mikhailov, V., Pisarevskaya, D., Malykh, V., & Shavrina, T. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000225 · Cited by 8
Dynamic Conversational Chatbot for Assessing Primary Students
Weerasinghe, E., Kotuwegedara, T., Amarasena, R., Jayasinghe, P., & Manathunga, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11647-6_89 · Cited by 4
What Has Been Enhanced in my Knowledge-Enhanced Language Model?
Hou, Y., Fu, G., & Sachan, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.102 · Cited by 6
“The poor performance of language models in handling numbers (Wallace et al., 2019) provides an alternative explanation for the observation that K-Adapter does not integrate triples with TR relations.”
BackgroundEvaluating Transformer’s Ability to Learn Mildly Context-Sensitive Languages
Wang, S., & Steinert‐Threlkeld, S. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.21 · Cited by 7
Wu, Y., Ting, H., Lam, T., & Luo, R. (2021) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3459930.3469557 · Cited by 6
“[18, 19] also show that BERT-based models are inadequate to capture the magnitude of numbers in the text.”
BackgroundR3: A Reading Comprehension Benchmark Requiring Reasoning ProcessesPreprint
Wang, R., Tao, K., Song, D., Zhang, Z., Ma, X., Su, X. H., & Dai, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01251 · Cited by 4
“, 2018), and that higher accuracy does not ensure more robustness or generalization (Jia and Liang, 2017; Wallace et al., 2019).”
BackgroundEnhancing Documents with Multidimensional Relevance Statements in Cross-encoder Re-rankingPreprint
Upadhyay, R., Askari, A., Pasi, G., & Viviani, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10979 · Cited by 4
“cross-encoder re-ranking? Previous studies have shown that BERT and other NLP model effectiveness can depend on the way in which numbers are encoded [32].”
BackgroundLompo, B. A., Le, T.-D., Jouvet, P. A., & Noumeir, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.10171 · Cited by 5
“Although [5] claimed that most word embedding models capture numeracy, several limitations persist.”
Flexible O&M for Telecom Networks at Huawei: A Language Model-based Approach
Wu, S., Qin, Z., Wang, Z., Rao, S., Ye, Q., Quan, X., & Tian, G. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481919 · Cited by 4
“Despite their importance, numeric values are difficult to be learned by a language model as discussed in [35].”
BackgroundChen, C.-C., Huang, H., & Chen, H. (2021) · doi:10.18653/v1/2021.emnlp-tutorials.2 · Cited by 3
“The numeral in the textual data (Naik et al., 2019; Wallace et al., 2019; Chen et al., 2018, 2019a, 2020c) and the numeracy of the neural network models (Spithourakis and Riedel, 2018; Chen et al., 2019b) attract lots of attentions recently.”
BackgroundCapturing the Spectrum of Social Media Conflict: A Novel Multi-objective Classification Model
Warke, O., Jose, J. M., Breitsohl, J., & Wang, J. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672510 · Cited by 3
Enhancing neural network predictions with finetuned numeric embeddings for stock trend forecasting
Trivedi, A., & Sangeetha, S. (2025) · Soft Computing · doi:10.1007/s00500-025-10483-5 · Cited by 4
“We contrast the predicted results with those obtained by existing methodologies (Thakkar and Chaudhari 2020b;SethiaandRaut2018;Huetal.2018;Jiangetal.2019; Shah et al. 2022; Wallace et al. 2019; Dey and Das 2023).”
InfluentialAnalysing Word Representation from the Input and Output Embeddings in Neural Network Language Models
Derby, S., Miller, P. C., & Devereux, B. J. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.36 · Cited by 6
“Examples include probing tasks, which are supervised machine learning problems that look to decode salient linguistic features from embedding vectors (Adi et al., 2016; Wallace et al., 2019; Tenney et al., 2019).”
BackgroundDeepEx: A Robust Weak Supervision System for Knowledge Base Augmentation
Moreira, J., & Barbosa, L. (2021) · Journal on Data Semantics · doi:10.1007/s13740-021-00134-x · Cited by 4
The Value of Numbers in Clinical Text Classification
Miok, K., Corcoran, P., & Spasić, . (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5030040 · Cited by 4
“In addition, embedding characters, in this case digits, is more effective than embedding subwords as the positions of digits can be used to capture scales [15–17].”
BackgroundMethodGuerin, N., & Chemla, E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.560 · Cited by 4
“…et al., 2019; Jumelet and Hupkes, 2018; Marvin and Linzen, 2018) for which they obtain strong results, but to a lesser extent for their semantic abilities (Rogers et al., 2020; Balasubramanian et al., 2020; Wallace et al., 2019; Ettinger, 2019) for which they show more fragile performances.”
BackgroundProbing with Noise: Unpicking the Warp and Weft of Embeddings
Klubička, F., & Kelleher, J. D. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.34 · Cited by 6
Interpretable Disease Prediction from Clinical Text by Leveraging Pattern Disentanglement
Ehghaghi, M., Zhou, P.-Y., Cheng, W. Y., Rajabi, S., Kuo, C.-H., & Lee, E.-S. A. (2023) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi58575.2023.10313408 · Cited by 3
“However, LLMs have been shown to have issues in the clinical domain, such as failure to interpret numerical values like body temperature [21], as well as issues with the misinterpretation of medical-specific negation (e.g., ”abstinence from alcohol” becomes ”alcohol dependence syndrome”) [8].”
BackgroundFrugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
Dutta, S., Pandey, I., Singh, J., Manchanda, S., Chakrabarti, S., & Chakraborty, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29750 · Cited by 14
Floating-Point Embedding: Enhancing the Mathematical Comprehension of Large Language Models
Jin, X., Mao, C., Yue, D., & Leng, T. (2024) · Symmetry · doi:10.3390/sym16040478 · Cited by 5
Kaddari, Z., Mellah, Y., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Conference on Intelligent Systems and Computer Vision (ISCV) · doi:10.1109/iscv49265.2020.9204142 · Cited by 6
“It is known that pre-trained language models naturally encode numeracy [15] but it is still unclear to us why reasoning over intervals rather than specific values did not increase accuracy.”
BackgroundNQuAD: 70,000+ Questions for Machine Comprehension of the Numerals in Text
Chen, C.-C., Huang, H.-H., & Chen, H.-H. (2021) · International Conference on Information and Knowledge Management · doi:10.1145/3459637.3482155 · Cited by 15
“Additionally, all of the previous works examine the numeracy on English dataset [5, 18, 26, 27].”
BackgroundInformation Extraction of Air-Traffic Control Instructions via Pre-trained Models
Wang, X., Ding, H., Xu, Q., & Liu, Z. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-9423-3_11 · Cited by 3
Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering
Kim, J.-H., Kang, J., Kim, K.-M., Hong, G., & Myaeng, S.-H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.138 · Cited by 3
“…other state-of-the-art models like QDGAT (Chen et al., 2020a) and NumNet (Ran et al., 2019), since they disregard the numeracy understanding part (Wallace et al., 2019) by simply employing specialized heads that learn to assign {-1, 0, +1} on numbers for summation, and they do not actually…”
BackgroundInfluentialWaLDORf: Wasteless Language-model Distillation On Reading-comprehensionPreprint
Tian, J. Y., Kreuzer, A. P., Chen, P.-H., & Will, H.-M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06638 · Cited by 3
“In academia, a plethora of research has arisen not only towards advancing and augmenting these state-of-the-art VLLMs[9, 10, 21, 22, 35, 45] but also towards studying their inner workings[1, 8, 17, 32, 33, 43, 47, 50–52] - a eld sometimes called ”BERTology”.”
BackgroundProbing Language Models for Understanding of Temporal ExpressionsPreprint
Thukral, S., Kukreja, K., & Kavouras, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01113 · Cited by 5
“On the semantic side, for instance, BERT struggles with representations of numbers (Wallace et al., 2019) and cannot reason based on its world knowledge (Rogers et al., 2020).”
Background
Metadata
- Authors
- Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Peking University (CN)
- University of California, Irvine (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 5306-5314
- OpenAlex
- W2974596953
- Cited by
- 227
- References
- 28
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 17 · 2023: 38 · 2022: 41 · 2021: 61 · 2020: 43 · 2019: 2
- FWCI
- 14.06
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Greater South Information System
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- numeracy, numerical reasoning, word embeddings, language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.