CoChat
Sign Up
Cited by 17Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data

Open access
Akhtar, Mubashara, Shankarampeta, Abhilash, Gupta, Vivek, Patil, Arpit, Cocarascu, Oana, Simperl, Elena·2023·Findings of the Association for Computational Linguistics: EMNLP 2023
DOI 10.18653/v1/2023.findings-emnlp.1028
Open this paper in CoChat — free accountOpen PDF

Abstract

Numbers are crucial for various real-world domains such as finance, economics, and science.Thus, understanding and reasoning with numbers are essential skills for language models to solve different tasks.While different numerical benchmarks have been introduced in recent years, they are limited to specific numerical aspects mostly.In this paper, we propose a hierarchical taxonomy for numerical reasoning skills with more than ten reasoning types across four levels: representation, number sense, manipulation, and complex reasoning.We conduct a comprehensive evaluation of state-of-the-art models to identify reasoning challenges specific to them.Henceforth, we develop a diverse set of numerical probes employing a semi-automated approach.We focus on the tabular Natural Language Inference (TNLI) task as a case study and measure models' performance shifts.Our results show that no model consistently excels across all numerical reasoning types.Among the probed models, FlanT5 (few-/zero-shot) and GPT-3.5 (few-shot) demonstrate strong overall numerical reasoning skills compared to other models.Label-flipping probes indicate that models often exploit dataset artifacts to predict the correct labels. 1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.1028

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are especially relevant, but are rarely examined within one unified setting.

    “Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are therefore especially relevant, but are rarely examined within one unified setting [11][12][13][14][15][16][17][18].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [16]

    The Introduction says earlier benchmarks focused on a few numerical-reasoning types, and this paper evaluates a wider range within one framework. That supports the numerical-grounding and limited-prior-coverage parts of the claim. The cited passages do not establish that compositionality, entity binding, and long-context evidence use are rarely examined together with numerical grounding. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Numerical reasoning introduces further difficulties, especially when values must be compared, computed, or correctly grounded to specific entities and variables.

    “Numerical reasoning introduces further difficulties, especially when values must be compared, computed, or correctly grounded to specific entities and variables [16-18, 51, 52].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022. PubHealthTab: A public health table-based dataset for evidence-based fact checking. In Find- ings of the Association for Computational Linguistics: NAACL 2022, pages 1-16, Seattle, United States. As- sociation for Computational Linguistics. · DOI
  2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 2357-2367, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  3. Pierre Barrouillet and Michel Fayol. 1998a. From algo- rithmic computing to direct retrieval: Evidence from number and alphabetic arithmetic in children and adults. Memory &amp Cognition, 26(2):355-368. · DOI
  4. Laura Bofferding. 2019. Understanding Negative Num- bers, pages 251-277. Springer International Publish- ing, Cham. · DOI
  5. Justin W. Bonny and Stella F. Lourenco. 2013. The ap- proximate number system and its relation to early math achievement: Evidence from the preschool years. Journal of Experimental Child Psychology, 114(3):375-388. · DOI
  6. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úl- far Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Se- curity 2021, August 11-13, 2021, pages 2633-2650. USENIX Association.
  7. Anthony Chen, Panupong Pasupat, Sameer Singh, Hon- grae Lee, and Kelvin Guu. 2023. PURR: efficiently editing language model hallucinations by denoising language model corruptions. CoRR, abs/2305.14908. · DOI · Paper page
  8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks. CoRR, abs/2211.12588. · DOI · Paper page
  9. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. CoRR, abs/1809.02922. · arXiv
  10. Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with interme- diate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281-296, Online. Association for Computational Lin- guistics. · DOI
  11. Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, and Dan Roth. 2023. Generic temporal reasoning with dif- ferential analysis and explanation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12013-12029, Toronto, Canada. Association for Computational Linguistics. · DOI
  12. Karen C Fuson. 2012. Children's counting and concepts of number. Springer Science & Business Media.
  13. Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 946-958, Online. Association for Computa- tional Linguistics. · DOI
  14. Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xiao- man Zhao, and Xiaoyong Du. 2022. PASTA: Table- operations aware fact verification via sentence-table cloze pre-training. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 4971-4983, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
  15. Vivek Gupta, Riyaz A. Bhat, Atreya Ghosal, Manish Shrivastava, Maneesh Singh, and Vivek Srikumar. 2022a. Is my model using the right evidence? sys- tematic probes for examining evidence-based tabular reasoning. Transactions of the Association for Com- putational Linguistics, 10:659-679. · DOI
  16. Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics. · DOI
  17. Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yu- jie He, Temma Choji, and Vivek Srikumar. 2022b. Right for the right reason: Evidence extraction for trustworthy tabular reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3268-3283, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI
  18. Hongwei Han, Jialiang Xu, Mengyu Zhou, Yijia Shao, Shi Han, and Dongmei Zhang. 2022. LUNA: lan- guage understanding with number augmentations on transformers via number plugins and pre-training. CoRR, abs/2212.02691. · DOI
  19. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
  20. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computa- tional Linguistics. · DOI
  21. Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset con- struction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887-896, Berlin, Germany. Association for Computational Lin- guistics. · DOI
  22. Shotaro Ishihara. 2023. Training data extraction from pre-trained language models: A survey. CoRR, abs/2305.16157. · DOI
  23. Aashna Jena, Vivek Gupta, Manish Shrivastava, and Julian Eisenschlos. 2022. Leveraging data recasting to enhance tabular reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4483-4496, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  24. Zhihua Jin, Xin Jiang, Xingbo Wang, Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. 2021. Numgpt: Improving numeracy ability of generative pre-trained models. CoRR, abs/2109.03137. · arXiv
  25. Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. 2021. Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031-7037, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  26. Elida V. Laski and Robert S. Siegler. 2007. Is 27 a big number? correlational and causal connections among numerical categorization, number line estima- tion, and numerical magnitude comparison. Child Development, 78(6):1723-1743. · DOI
  27. Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. 2022. MWP-BERT: Numeracy-augmented pre-training for math word problem solving. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, pages 997-1009, Seattle, United States. Association for Computational Linguistics. · DOI
  28. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre- Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862-6868, Online. Association for Computational Linguistics. · DOI
  29. Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: table pre-training via learning a neural SQL executor. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  30. Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos, and Georgios Paliouras. 2022. FiNER: Financial numeric entity recognition for XBRL tagging. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 4419-4431, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
  31. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. Mathdial: A dialogue tutor- ing dataset with rich pedagogical properties grounded in math reasoning problems. CoRR, abs/2305.14536. · DOI
  32. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christo- foros Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: a survey. CoRR, abs/2302.07842. · DOI · Paper page
  33. Matteo Muffo, Aldo Cocco, and Enrico Bertino. 2022. Evaluating transformer language models on arith- metic operations using number decomposition. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 291-297. European Language Resources Association.
  34. Darko Odic and Ariel Starr. 2018. An introduction to the approximate number system. Child Development Perspectives, 12(4):223-229. · DOI
  35. Kuntal Kumar Pal and Chitta Baral. 2021. Investigating numeracy learning ability of a text-to-text transfer model. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 3095-3101, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  36. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. As- sociation for Computational Linguistics. · DOI · Paper page
  37. Sungjin Park, Seungwoo Ryu, and Edward Choi. 2022. Do language models understand measurements? In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 1782-1792, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
  38. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics. · DOI · Paper page
  39. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
  40. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
  41. Abhilash Shankarampeta, Vivek Gupta, and Shuo Zhang. 2022. Enhancing tabular reasoning with pat- tern exploiting training. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 706-726, Online only. Association for Computational Linguis- tics.
  42. Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vin- cent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Dar- rell, and Denny Zhou. 2023. Flan-moe: Scaling instruction-finetuned language models with sparse mixture of experts. CoRR, abs/2305.14705. · DOI
  43. Avijit Thawani, Jay Pujara, and Filip Ilievski. 2021a. Numeracy enhances the literacy of language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6960-6967, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI
  44. Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021b. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-656, Online. As- sociation for Computational Linguistics. · DOI
  45. Shyam Upadhyay and Ming-Wei Chang. 2017. An- notating derivations: A new evaluation strategy and dataset for algebra word problems. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 1, Long Papers, pages 494-504, Valencia, Spain. Association for Computational Linguistics.
  46. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know num- bers? probing numeracy in embeddings. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 5307-5315, Hong Kong, China. Association for Computational Linguis- tics. · DOI · Paper page
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  48. Jemma Catherine Whyte and Rebecca Bull. 2008. Num- ber games, magnitude representation, and basic num- ber skills in preschoolers. Developmental Psychol- ogy, 44(2):588-596. · DOI
  49. Jialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei Zhang. 2022. Towards robust numerical question answering: Diagnosing numerical capabili- ties of NLP systems. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7950-7966, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
  50. Peng-Jian Yang, Ying-Ting Chen, Yuechan Chen, and Daniel Cer. 2021. Nt5?! training T5 to perform numerical reasoning. CoRR, abs/2104.07307. · arXiv
  51. Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language mod- els are versatile decomposers: Decompose evidence and questions for table-based reasoning. CoRR, abs/2301.13808. · DOI
  52. Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020. Do language embeddings capture scales? In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 4889-4896, Online. Association for Computa- tional Linguistics. · DOI
  53. Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. 2022. ReasTAP: Injecting ta- ble reasoning skills during pre-training via synthetic reasoning examples. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9006-9018, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
  54. Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with min- imal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7579-7589, Online. Association for Computational Linguistics. · DOI
  55. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual con- tent in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277-3287, Online. Association for Computational Linguistics. · DOI · Paper page

Cited by

Sorted by citations

Authors
Akhtar, M., Shankarampeta, A., Gupta, V., Patil, A., Cocarascu, O., & Simperl, E.
Affiliations
  • King's College London (GB)
  • University of Pennsylvania (US)
  • University of Utah (US)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
15391-15405
DOI
10.18653/v1/2023.findings-emnlp.1028
OpenAlex
W4389518637
Cited by
17
References
9
Citations by year
2026: 7 · 2025: 9 · 2024: 1
FWCI
2.70
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
numerical reasoning, language models, tabular data, natural language inference, probing, number sense, GPT-3.5, benchmark evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review