Do NLP Models Know Numbers? Probing Numeracy in Embeddings
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, Matt Gardner. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 We train a probing model to decode a number from its word embedding over a random 80% of the integers from [-500, 500], e.g., '71' → 71.0. We plot the model's predictions for all numbers from [-2000, 2000]. The model accurately decodes numbers within the training range (in blue), i.e., pre-trained embeddings like GloVe and BERT capture numeracy. However, the probe fails to extrapolate to larger numbers (in red). The Char-CNN (e) and Char-LSTM (f) are trained jointly with the probing model.](https://figures.cochat.ai/paper-figures/153bcb5dac394f6194eb8e7aa3ffa6ae/do-nlp-models-know-numbers-probing-numeracy-in-embeddings-figure-1.png)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“, 2018), BERT (Devlin et al., 2019), and GloVe (Pennington et al.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“, 2018), BERT (Devlin et al., 2019), and GloVe (Pennington et al.”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“We find that all widely-used pre-trained embed-dings, e.g., ELMo (Peters et al., 2018), BERT (De-vlin et al., 2019), and GloVe (Pennington et al., 2014), capture numeracy: number magnitude is present in the embeddings, even for numbers in the thousands.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“We find that all widely-used pre-trained embed-dings, e.g., ELMo (Peters et al., 2018), BERT (De-vlin et al., 2019), and GloVe (Pennington et al., 2014), capture numeracy: number magnitude is present in the embeddings, even for numbers in the thousands.”
MethodSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“The dataset’s input-output format is a superset of SQuAD (Rajpurkar et al., 2016): the an... JaMarcus Russell completed a 91-yard touchdown pass to rookie wide receiver Chaz Schilens .”
MethodUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 2×
“, influence functions (Koh and Liang, 2017; Brunet et al., 2019)).”
BackgroundBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“As in typical span selection models (Seo et al., 2017), an LSTM reads the list of token embeddings, and a weight matrix and softmax function assign a probability to each index using the model’s hidden state.”
BackgroundMethodGraves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 2×
“Synthetic Numerical Tasks Similar to our synthetic numerical reasoning tasks, other work considers sorting (Graves et al., 2014), counting (Weiss et al., 2018), or decoding tasks (Trask et al., 2018).”
MethodAssessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Dupoux, E., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 737 · cited 1×
“Khandel-wal et al. (2018) show how language models leverage context, while Linzen et al. (2016) demonstrate that language models understand subject-verb agreement.”
BackgroundLinguistic Knowledge and Transferability of Contextual Representations
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., & Smith, N. A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1112 · Cited by 621 · cited 2×
“Probing Models Our probes of numeracy parallel work in understanding the linguistic capabilities (literacy) of neural models (Conneau et al., 2018; Liu et al., 2019).”
BackgroundSelf-citation
Research works printed but not linked: 5
- [1]1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Finegrained anal- ysis of sentence embeddings using auxiliary predic- tion tasks. In ICLR.
- [13]13. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguis- tic knowledge and transferability of contextual rep- resentations. In NAACL.
- [14]14. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Ad- vances in pre-training distributed word representa- tions. In LREC.
- [24]24. Andrew Trask, Felix Hill, Scott Reed, Jack W. Rae, Chris Dyer, and Phil Blunsom. 2018. Neural arith- metic logic units. In NeurIPS.
- [29]29. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehen- sion. In ICLR.
Cited by
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
“Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small, but fail when the numbers are larger [9, 88, 139].”
BackgroundA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
“This is in line with the recent findings on adversarial attacks, with models disturbed by nonsensical inputs (Wallace et al., 2019a), and suggests that BERT’s encoding of syntactic structure does not indicate that it actually relies on that knowledge .”
BackgroundResultPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
TaPas: Weakly Supervised Table Parsing via Pre-training
Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.398 · Cited by 535
“This can assist the model when processing questions that involve superlatives, as word pieces may not represent numbers informatively (Wallace et al., 2019).”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
“Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small, but fail when the numbers are larger [9, 88, 139].”
BackgroundoLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 271
“…often linear, randomly initialized probe, to determine whether the representations hold relevant information (Ettinger et al., 2016; Adi et al., 2016; Belinkov and Glass, 2019; He-witt and Manning, 2019; Wallace et al., 2019; Rozen et al., 2019; Warstadt et al., 2019; Richard-son et al., 2019).”
BackgroundAttention is Not Only a Weight: Analyzing Transformers with Vector Norms
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.574 · Cited by 377
“The first category assesses a single word or phraselevel linguistic capabilities of BERT, such as its performance on part-of-speech tagging and word sense disambiguation performance (Tenney et al., 2019; Jawahar et al., 2019; Reif et al., 2019; Lin et al., 2019; Wallace et al., 2019).”
BackgroundWhat Happens To BERT Embeddings During Fine-tuning?
Merchant, A., Rahimtoroghi, E., Pavlick, E., & Tenney, I. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.4 · Cited by 139
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.89 · Cited by 162
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
“…(Petroni et al., 2019; Jiang et al., 2020b; Roberts et al., 2020; Bouraoui et al., 2020), commonsense knowledge (Trinh and Le, 2018; Kocijan et al., 2019; Talmor et al., 2019a; Bosselut et al., 2019), and simple numerical operations (Wallace et al., 2019; Talmor et al., 2019a; Geva et al., 2020).”
BackgroundTABBIE: Pretrained Representations of Tabular Data
Iida, H., Thai, D., Manjunatha, V., & Iyyer, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.270 · Cited by 238
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.557 · Cited by 117
Probing Pre-Trained Language Models for Cross-Cultural Differences in Values
Arora, A., Kaffee, L.-A., & Augenstein, I. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.12 · Cited by 57
“This can be syntactic (Hewitt and Manning, 2019), semantic (Vuli´c et al., 2020), numerical (Wallace et al., 2019), relational (Petroni et al., 2019) or factual knowledge (Jiang et al., 2020) picked up by PLMs. Probes can be created on both, at the word or sentence level (Mosbach et al., 2020).”
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 222
Annotating Columns with Pre-trained Language Models
Suhara, Y., Li, J., Li, Y., Zhang, D., Demiralp, Ç., Chen, C., & Tan, W.-C. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3517906 · Cited by 205
“This results aligns with the findings from the NLP literature [16, 55] that Transformer models can partially handle numeric data.”
ResultTransformers: “The End of History” for Natural Language Processing?
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_41 · Cited by 149
LLM-Prop: predicting the properties of crystalline materials using large language models
Rubungo, A. N., Arnold, C. B., Rand, B. P., & Dieng, A. B. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01536-2 · Cited by 145
“Ithasalsoproveninprevious studiesthatLLMsstruggletounderstandnumericalinformationwhereit oftentreatsthemasnormaltokensyetdonotbene fi t from the quantitative meaning that numerical values convey 36 – 39 , How much data does LLM-Prop need to achieve SOTA results?”
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.146 · Cited by 71
“semantic knowledge (Tenney et al., 2019; Wallace et al., 2019; Ettinger, 2020) and world knowledge (Davison et al.”
BackgroundTeaching Arithmetic to Small Transformers
Lee, N., Sreenivasan, K., Lee, J. D., Lee, K., & Papailiopoulos, D. (2023) · arXiv · doi:10.48550/arxiv.2307.03381
Understanding tables with intermediate pre-training
Eisenschlos, J., Krichene, S., & Müller, T. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.27 · Cited by 86
MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving
Liang, Z., Zhang, J., Wang, L., Qin, W., Lan, Y., Shao, J., & Zhang, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.74 · Cited by 153
“…semantic 042 representations (Li et al., 2019; Zhang et al., 2020b; 043 Wu et al., 2021c,a; Yu et al., 2021a), injecting sym- 044 bolic constraints (Wang et al., 2018, 2019; Liu et al., 045 2019a; Xie and Sun, 2019) and how to align seman- 046 tic space (text descriptions) and huge combinato- 047…”
BackgroundMethodInfluentialINFOTABS: Inference on Tables as Semi-structured Data
Gupta, V., Mehta, M., Nokhiz, P., & Srikumar, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.210 · Cited by 77
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89
“Numeracy and Temporal Reasoning in LMs Our work contributes to the larger body of work studying numeracy in word embeddings and language models (Spithourakis and Riedel, 2018; Wallace et al., 2019).”
BackgroundSelf-citationHanna, M., Liu, O., & Variengien, A. (2023) · neural information processing systems · doi:10.52202/075280-3322 · Cited by 430
Hanna, M., Liu, O., & Variengien, A. (2023) · arXiv · doi:10.48550/arxiv.2305.00586 · Cited by 84
“We find, as in Wallace et al. [49], that structure already exists in the static year embeddings, though the years 02-09 are clustered apart from the rest.”
BackgroundResultClinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration
van Aken, B., Papaioannou, J.-M., Mayrdorfer, M., Budde, K., Gers, F., & Loeser, A. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.75 · Cited by 63
Semantic Structure in Deep Learning
Pavlick, E. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 66
(2021) · doi:10.18653/v1/2021.naacl-main.53 · Cited by 78
A Closer Look at How Fine-tuning Changes BERT
Zhou, Y., & Srikumar, V. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.75 · Cited by 102
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.-Y., Papailiopoulos, D., & Lee, K. (2022) · arXiv · doi:10.48550/arxiv.2206.06565 · Cited by 67
“[135] Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner.”
BackgroundCommonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
Bhargava, P., & Ng, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21496 · Cited by 104
Towards Observability for Production Machine Learning Pipelines
Shankar, S., & Parameswaran, A. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3565838.3565853 · Cited by 70
Drug–Drug Interaction Relation Extraction Based on Deep Learning: A Review
Dou, M., Tang, J., Tiwari, P., Ding, Y., & Guo, F. (2024) · ACM Computing Surveys · doi:10.1145/3645089 · Cited by 60
“In terms of training methods, they can be roughly divided into three categories: word2vec 6 /glove 7 [26, 52, 75, 92, 104, 120], character level embedding[6, 65, 115], and BERT 8 (Bidirectional Encoder Representations from Transformers) [4, 34, 56, 138].”
Deidentification of free-text medical records using pre-trained bidirectional transformers
Johnson, A. E. W., Bulgarelli, L., & Pollard, T. J. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384455 · Cited by 55
“Despite training on a large corpus of text, BERT based models struggle with numeracy, in particular due to the poor representation of large numbers in sub-word tokenization [38].”
BackgroundRELIC: Investigating Large Language Model Responses using Self-Consistency
Cheng, F., Zouhar, V., Arora, S., Sachan, M., Strobelt, H., & El-Assady, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641904 · Cited by 37
“In these cases, numbers which occur frequently in the data, such as 1000000 are represented as a unique token, but a 7705453 might be represented as a sequence of 770 , 545 , and 3 .”
BackgroundBootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
Orr, L., Leszczynski, M., Arora, S., Wu, S., Guha, N., Xiao, L., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10363 · Cited by 40
“This may be because the BERT model represents certain numbers with sub-word tokens and is known to under-perform other language models on numbers [28].”
Data-to-text generation with macro planning
Puduppully, R., & Lapata, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00381 · Cited by 53
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Razeghi, Y., Logan, R. L., Gardner, M., & Singh, S. (2022) · arXiv · doi:10.48550/arxiv.2202.07206 · Cited by 53
“Numeracy and Temporal Reasoning in LMs Our work contributes a larger body of work dedicated to studying numeracy in word embeddings and language models (Spithourakis & Riedel, 2018; Wallace et al., 2019).”
MethodSelf-citationReasoning Like Program Executors
Pi, X., Liu, Q., Chen, B., Ziyadi, M., Lin, Z., Fu, Q., Gao, Y., Lou, J., & Chen, W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.48 · Cited by 73
Web Table Retrieval using Multimodal Deep Learning
Shraga, R., Roitman, H., Feigenblat, G., & Canim, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401120 · Cited by 53
“We used GloVe [25] (300-dimension) word embeddings, which have been shown to handle both text and (to some extent) numeric data [35].”
MethodNumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-Task Financial Forecasting
Yang, L., Li, J., Dong, R., Zhang, Y., & Smyth, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21414 · Cited by 75
Semantics Driven Embedding Learning for Effective Entity Alignment
Zhong, Z., Zhang, M., Fan, J., & Dou, C. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00205 · Cited by 70
“[28] has reported that ELMo has the best performance in capturing numeracy among all pre-trained methods.”
BackgroundMethodGenerative Table Pre-training Empowers Models for Tabular Prediction
Zhang, T., Wang, S., Yan, S., Li, J., & Liu, Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.917 · Cited by 78
Probing for the Usage of Grammatical Number
Lasri, K., Pimentel, T., Lenci, A., Poibeau, T., & Cotterell, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.603 · Cited by 91
Math Word Problem Solving with Explicit Numerical Values
Wu, Q., Zhang, Q., Wei, Z., & Huang, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.455 · Cited by 62
“Numerical Value Representations: Some recent studies have explored the numerical value representations in language models (Naik et al., 2019; Chen et al., 2019; Wallace et al., 2019).”
BackgroundMathematical Language Models: A Survey
Liu, W., Hu, H., Zhou, J., Ding, Y., Li, J., Zeng, J., He, M., Chen, Q., Jiang, B., Zhou, A., et al. (2025) · ACM Computing Surveys · doi:10.1145/3773985 · Cited by 57
“In the field of math problem solving in artificial intelligence, math problem solving refers to the process of using algorithms, computational models, and increasingly LLMs to understand, explain, and solve mathematical problems.”
Injecting the BM25 Score as Text Improves BERT-Based Re-rankers
Askari, A., Abolghasemi, A., Pasi, G., Kraaij, W., & Verberne, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_5 · Cited by 80
TABBIE: Pretrained Representations of Tabular DataPreprint
Iida, H., Thai, D. N., Manjunatha, V., & Iyyer, M. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2105.02584 · Cited by 244
“Additionally, BERT has been shown to encode some degree of numeracy (Wallace et al., 2019), which helps represent cells with numerical content.”
BackgroundTOPReward: Token Probabilities as Hidden Zero-Shot Rewards for RoboticsPreprint
Chen, S., Harrison, C., Lee, Y.-C., Yang, A. J., Ren, Z., Ratliff, L. J., Duan, J., Fox, D., & Krishna, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.19313 · Cited by 49
“In this paper, we hypothesize that the reason for this is not a lack of visual understanding in open-source VLMs but rather the instability of numeric token generation—LLMs are poorly calibrated when asked to produce precise numerical outputs (Wallace et al., 2019; Yuan et al., 2023).”
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 97
Intrinsic Probing through Dimension Selection
Hennigen, L. T., Williams, A., & Cotterell, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.15 · Cited by 58
Methods for Numeracy-Preserving Word Embeddings
Sundararaman, D., Si, S., Subramanian, V., Wang, G., Hazarika, D., & Carin, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.384 · Cited by 71
“To observe the numerical properties of DICE, we consider two tasks: Task 1 deals with the numer-ation (NUM) and magnitude (MAG) properties as proposed by (Naik et al., 2019); Task 2 performs list maximum , decoding , and addition as proposed by (Wallace et al., 2019).”
BackgroundMethodResultInfluentialOn Attention Redundancy: A Comprehensive Study
Bian, Y., Huang, J., Cai, X., Yuan, J., & Church, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.72 · Cited by 68
“BERT does not excel at numbers, negation, inferences and role-based event prediction (Wallace et al., 2019; Ettinger, 2020).”
BackgroundYoran, O., Talmor, A., & Berant, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.416 · Cited by 35
PromptEM: prompt-tuning for low-resource generalized entity matching
Wang, P., Zeng, X., Chen, L., Ye, F., Mao, Y., Zhu, J., & Gao, Y. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3565816.3565836 · Cited by 53
Do Language Embeddings capture Scales?
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.439 · Cited by 35
An Empirical Investigation of Contextualized Number Prediction
Berg-Kirkpatrick, T., & Spokoyny, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.385 · Cited by 45
Zero-shot Generalization in Dialog State Tracking through Generative Question Answering
Li, S., Cao, J. L., Sridhar, M., Zhu, H., Li, S.-W. D., Hamza, W., & McAuley, J. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.91 · Cited by 53
Self-citationFlores, C. A., Figueroa, R. L., & Pezoa, J. E. E. (2021) · IEEE Access · doi:10.1109/access.2021.3064000 · Cited by 39
“In this sense, it has been shown that BERT may not work properly representing numbers, while regular expressions allow representing complex sequential patterns, including numerical attributes [18], [22], [23].”
BackgroundML-LJP: Multi-Law Aware Legal Judgment Prediction
Liu, Y., Wu, Y., Zhang, Y., Sun, C., Lü, W., Wu, F. Q., & Kuang, K. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591731 · Cited by 70
“[36] divides the existing methods into string-based methods [13, 40, 58] and real-based methods [35, 40].”
MethodNumber Cookbook: Number Understanding of Language Models and How to Improve It
Yang, H., Hu, Y., Kang, S., Lin, Z., & Zhang, M. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2411.03766 · Cited by 42
“…mentioned in the main text, our paper also includes some related works, which are: Probing numerical understanding in natural language Earlier studies (Wallace et al., 2019; Johnson et al., 2020) have explored LMs’ understanding of numbers presented in natural languages, such as English or Chinese.”
CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming ConstraintsPreprint
Paulus, A., Rolínek, M., Musil, V., Amos, B., & Martius, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.02343 · Cited by 34
Sundaram, S. S., Gurajada, S., Padmanabhan, D., Abraham, S. S., & Fisichella, M. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1534 · Cited by 35
“Studies (Spithourakis & Riedel, 2018; Wallace et al., 2019) have delved deep into the numerical and mathematical ability of language models.”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks
Nogueira, R., Jiang, Z., & Lin, J. (2021) · arXiv · doi:10.48550/arxiv.2102.13019 · Cited by 29
“Character-level models such as ELMO (Peters et al., 2018) have stronger numeracy than sub-word models such as BERT (Devlin et al., 2019), perhaps because two numbers that are similar in value can have very different sub-word tokenizations (Wallace et al., 2019).”
BackgroundMethodInfluentialInfusing Finetuning with Semantic Dependencies
Wu, Z., Peng, H., & Smith, N. A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00363 · Cited by 39
“Wallace et al. (2019) observed that BERT displays suboptimal numeracy knowledge.”
BackgroundA survey on mathematical reasoning and optimization with large language models
Forootani, A., Aliabadi, D. E., & Thrän, D. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200712 · Cited by 32
Jiang, C., Nian, Z., Guo, K., Chu, S., Zhao, Y., Shen, L., & Tu, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.235 · Cited by 27
Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks
Pârcălăbescu, L., Gatt, A., Frank, A., & Calixto, I. (2020) · Pure Amsterdam UMC · doi:10.48550/arxiv.2012.12352 · Cited by 28
“This is in stark contrast to recent findings with text-only pretrained language models, which have a good grasp of numeracy and perform well when interpolating quantities (Wallace et al., 2019).”
ResultBalsebre, P., Yao, D., Cong, G., & Hai, Z. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512026 · Cited by 27
“In a recent study [37], Wallace and Wang et al.”
MethodPrecision information extraction for rare disease epidemiology at scale
Kariampuzha, W. Z., Alyea, G., Qu, S., Sanjak, J. S., Mathé, E. A., Sid, E., Chatelaine, H., Yadaw, A. S., Xu, Y., & Zhu, Q. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04011-y · Cited by 31
“BERT-based models have been shown to have some limitations on numeracy, thus the BERT-based model relies much more heavily on contextual information and the attention mechanism rather than sub-word embeddings to differentiate the significance between different numbers [119].”
MethodMass-Producing Failures of Multimodal Systems with Language Models
Tong, S., Jones, E., & Steinhardt, J. (2023) · neural information processing systems · doi:10.52202/075280-1275 · Cited by 60
Testolin, A. (2024) · Applied Sciences · doi:10.3390/app14020744 · Cited by 24
“Although some initial investigations suggested that standard embedding techniques could capture number semantic fairly well (Wallace et al., 2019), subsequent studies pointed out that these representations are in fact inadequate at dealing precisely with numbers (Naik et al.”
BackgroundDirectProbe: Studying Representations without Classifiers
Zhou, Y., & Srikumar, V. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.401 · Cited by 43
Do Language Embeddings capture Scales?Preprint
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Findings · doi:10.48550/arxiv.2010.05345 · Cited by 101
“Wallace et al. (2019) showed that BERT and ELMo had a limited amount of numeracy or numerical reasoning ability, when restricted to numbers of small magnitude.”
BackgroundMethodInfluentialCan language representation models think in bets?
Tang, Z., & Kejriwal, M. (2023) · Royal Society Open Science · doi:10.1098/rsos.221585 · Cited by 23
“Owing in part to the close connection between language and cognition [23,24], a growing body of research is seeking to deduce the cognitive abilities (or lack thereof) of LRMs [25,26].”
BackgroundCharacter-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT
van Noord, R., Toral, A., & Bos, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.371 · Cited by 45
Do Language Embeddings capture Scales?
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.27 · Cited by 22
“4 Numeracy through Scientific Notation Wallace et al. (2019) showed that BERT and ELMo had a limited amount of numeracy or numerical reasoning ability, when restricted to numbers of small magnitude.”
BackgroundMethodInfluentialPremise Selection in Natural Language Mathematical Texts
Ferreira, D. M., & Freitas, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.657 · Cited by 22
“Wallace et al. (2019) found that CNNs are useful for tasks involving understanding and working with numbers; however, it still struggles to extrapolate beyond the values seen during training.”
BackgroundCan Language Models Understand Physical Concepts?
Li, L., Xu, J., Dong, Q., Zheng, C., Sun, X., Kong, L., & Liu, Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.726 · Cited by 36
Neural Methods for Data-to-text Generation
Sharma, M., Gogineni, A. K., & Ramakrishnan, N. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3660639 · Cited by 31
“Outside of the D2T niche, there are research communities focused on solving these individual problems - NLG [100, 235, 320] and numerical reasoning [295, 317, 334, 349].”
Chen, C.-C., Huang, H., & Chen, H. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412100 · Cited by 22
“Analyzing the numeral information in documents is an emerging topic, and attracts much attention [7, 18, 23, 26, 29].”
BackgroundNatural Language Premise Selection: Finding Supporting Statements for Mathematical TextPreprint
Ferreira, D. M., & Freitas, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14959 · Cited by 22
“o verify on how state-of-the-art embedding models perform with such specific dataset. BERT (Devlin et al., 2019) is reported to have performed in different NLP tasks, including understanding numeracy (Wallace et al., 2019). In order to use BERT, we formulate the problem as a pairwise relevance classification problem, wh…”
BackgroundMethodNumeracy enhances the Literacy of Language Models
Thawani, A., Pujara, J., & Ilievski, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.557 · Cited by 31
“Value embeddings (Wallace et al., 2019) project the scalar magnitude of the number to be encoded into a vector space of same dimensionality as the lookup word embeddings.”
BackgroundInfluentialNumGPT: Improving Numeracy Ability of Generative Pre-trained ModelsPreprint
Jin, Z., Jiang, X., Wang, X., Liu, Q., Wang, Y., Ren, X., & Qu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03137 · Cited by 25
Self-citationA comparison of text preprocessing techniques for hate and offensive speech detection in Twitter
Glazkova, A. V. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01156-y · Cited by 20
“Probably, it can be related to the fact that pretrained language models are more focused on understanding words than numbers (Wallace et al. 2019; Rogers et al. 2020).”
AND does not mean OR: Using Formal Languages to Study Language Models’ Representations
Traylor, A., Feiman, R., & Pavlick, E. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.21 · Cited by 19
“More distantly related is work on LMs’ ability to reason about numbers (Wallace et al., 2019) or perform multi-hop reasoning (Yang et al., 2018).”
BackgroundProbing Language Models for Understanding of Temporal Expressions
Thukral, S., Kukreja, K., & Kavouras, C. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.31 · Cited by 32
Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation
Le, T.-D., Noumeir, R., Rambaud, J., Sans, G., & Jouvet, P. A. (2022) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2022.3209900 · Cited by 22
“It has been proven that the pre-trained token representations (word2vec) can naturally encode the numeric values [11].”
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
“…et al., 2019; Tenney et al., 2019; Clark et al., 2019; Liu et al., 2019a; Hewitt and Manning, 2019) and semantic knowledge (Tenney et al., 2019; Wallace et al., 2019; Yaghoobzadeh et al., 2019; Liu et al., 2019a) in PLMs. Popular classifiers include linear classifier, logistics regression,…”
BackgroundInfluentialThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
“…et al., 2019; Tenney et al., 2019; Clark et al., 2019; Liu et al., 2019a; Hewitt and Manning, 2019) and semantic knowledge (Tenney et al., 2019; Wallace et al., 2019; Yaghoobzadeh et al., 2019; Liu et al., 2019a) in PLMs. Popular classifiers include linear classifier, logistics regression,…”
BackgroundInfluentialInvestigating Numeracy Learning Ability of a Text-to-Text Transfer Model
Pal, K. K., & Baral, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.265 · Cited by 18
Sevastjanova, R., Kalouli, A.-L., Beck, C., Hauptmann, H., & El‐Assady, M. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14541 · Cited by 26
Character-based Neural Semantic Parsing
van Noord, R. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.169308968 · Cited by 18
A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models
Stolfo, A., Jin, Z., Shridhar, K., Schoelkopf, B., & Sachan, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.32 · Cited by 17
Akhtar, M., Shankarampeta, A., Gupta, V., Patil, A., Cocarascu, O., & Simperl, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.1028 · Cited by 17
Shen, K., & Kejriwal, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0295925 · Cited by 17
Cadeddu, A., Chessa, A., Leo, V. D., Fenu, G., Motta, E., Osborne, F., Recupero, D. R., Salatino, A. A., & Secchi, L. (2024) · Information · doi:10.3390/info15070398 · Cited by 20
“Regarding numeric value injection as text, previous works [43,44] have proven that BERT can handle numeric values expressed as text.”
Method
Metadata
- Authors
- Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Peking University (CN)
- University of California, Irvine (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 5306-5314
- OpenAlex
- W2974596953
- Cited by
- 227
- References
- 28
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 17 · 2023: 38 · 2022: 41 · 2021: 61 · 2020: 43 · 2019: 2
- FWCI
- 14.06
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Greater South Information System
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- numeracy, numerical reasoning, word embeddings, language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.