Can Neural Networks Do Arithmetic? A Survey on the Elementary Numerical Skills of State-of-the-Art Deep Learning Models
Open accessAbstract
Creating learning models that can exhibit sophisticated reasoning abilities is one of the greatest challenges in deep learning research, and mathematics is rapidly becoming one of the target domains for assessing scientific progress in this direction. In the past few years there has been an explosion of neural network architectures, datasets, and benchmarks specifically designed to tackle mathematical problems, reporting impressive achievements in disparate fields such as automated theorem proving, numerical integration, and the discovery of new conjectures or matrix multiplication algorithms. However, despite this notable success it is still unclear whether deep learning models possess an elementary understanding of quantities and numbers. This survey critically examines the recent literature, concluding that even state-of-the-art architectures and large language models often fall short when probed with relatively simple tasks designed to test basic numerical and arithmetic knowledge.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs quickly fail in arithmetic as operands increase, especially in multiplication.
“Another fundamental failure is that LLMs quickly fail in arithmetic as operands increase (Yuan et al., 2023; Testolin, 2024), especially in multiplication .”
From Large Language Model Reasoning Failures (2026) · cites this paper as Testolin, 2024
Section 3.3 (Large language models; page unavailable) says GPT-3’s calculation accuracy rapidly decreases as the number of digits increases, supporting the claim about larger operands. It does not establish that the decline is especially pronounced in multiplication. Its discussion of difficulty with a higher number of operands concerns other neural architectures, not LLMs. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Harnad, S. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90087-6 · Cited by 4,235 · cited 1× · see citing works
“On the one hand, grounding structured mathematical knowledge into some form of intrinsic meaning is a long-standing problem in symbolic AI (Searle, 1980; Harnad, 1990).”
BackgroundConnectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Hybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412 · see citing works
Ethical principles for artificial intelligence in education
Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B.-P. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11316-w · Cited by 1,228 · see citing works
Early math matters: Kindergarten number competence and later mathematics outcomes.
Jordan, N. C., Kaplan, D. E., Ramineni, C., & Locuniak, M. N. (2009) · Developmental Psychology · doi:10.1037/a0014939 · Cited by 1,172 · see citing works
Developing conceptual understanding and procedural skill in mathematics: An iterative process.
Rittle‐Johnson, B., Siegler, R. S., & Alibali, M. W. (2001) · Journal of Educational Psychology · doi:10.1037/0022-0663.93.2.346 · Cited by 1,155 · see citing works
Deep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853 · see citing works
The logic theory machine-A complex information processing system
Newell, A., & Simon, H. (1956) · IEEE Transactions on Information Theory · doi:10.1109/tit.1956.1056797 · Cited by 846 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Early Childhood Mathematics Education Research
Sarama, J. H., & Clements, D. H. (2009) · Routledge eBooks · doi:10.4324/9780203883785 · Cited by 717 · see citing works
Anders, Y., Roßbach, H., Weinert, S., Ebert, S., Kuger, S., Lehrl, S., & von Maurice, J. (2012) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2011.08.003 · Cited by 622 · see citing works
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · see citing works
Neurocognitive start-up tools for symbolic number representations
Manuela Piazza, A. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.09.008 · Cited by 488 · see citing works
Discovering faster matrix multiplication algorithms with reinforcement learning
Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera-Paredes, B., Barekatain, M., Novikov, A., R. Ruiz, F. J., Schrittwieser, J., Swirszcz, G., et al. (2022) · Nature · doi:10.1038/s41586-022-05172-4 · Cited by 478 · see citing works
Advancing mathematics by guiding human intuition with AI
Davies, A., Veličković, P., Buesing, L., Blackwell, S., Zheng, D., Tomašev, N., Tanburn, R., Battaglia, P., Blundell, C., Juhász, A., et al. (2021) · Nature · doi:10.1038/s41586-021-04086-x · Cited by 454 · see citing works
How Important is Where you Start? Early Mathematics Knowledge and Later School Success
Claessens, A., & Engel, M. (2013) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146811311500603 · Cited by 415 · see citing works
Deep Neural Solver for Math Word Problems
Wang, Y., Liu, X., & Shi, S. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1088 · Cited by 404 · see citing works
Which preschool mathematics competencies are most predictive of fifth grade achievement?
Nguyen, T., Watts, T. W., Duncan, G. J., Clements, D. H., Sarama, J. S., Wolfe, C. B., & Spitler, M. E. (2016) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2016.02.003 · Cited by 402 · see citing works
Nieder, A. (2016) · Nature reviews. Neuroscience · doi:10.1038/nrn.2016.40 · Cited by 389 · see citing works
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385 · see citing works
Number and language: how are they related?
Gelman, R., & Butterworth, B. L. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.11.004 · Cited by 373 · see citing works
Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization
Shalev‐Shwartz, S., & Zhang, T. (2014) · Mathematical Programming · doi:10.1007/s10107-014-0839-0 · Cited by 363 · see citing works
Solving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358 · see citing works
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335 · see citing works
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · see citing works
Are NLP Models really able to Solve Simple Math Word Problems?
Patel, A., Bhattamishra, S., & Goyal, N. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.168 · Cited by 307 · see citing works
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · see citing works
Emergence of a 'visual number sense' in hierarchical generative models
Stoianov, I., & Zorzi, M. (2012) · Nature Neuroscience · doi:10.1038/nn.2996 · Cited by 300 · see citing works
Origins of Mathematical Intuitions
Dehaene, S. (2009) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2009.04469.x · Cited by 266 · see citing works
Analysing Mathematical Reasoning Abilities of Neural Models
Saxton, D., Grefenstette, E., Hill, F., & Kohli, P. (2019) · arXiv · doi:10.48550/arxiv.1904.01557 · Cited by 241 · see citing works
Is There Really an Evolved Capacity for Number?
Núñez, R. E. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.03.005 · Cited by 240 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · arXiv · doi:10.48550/arxiv.1903.00161 · Cited by 238 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Clements, D. H., Sarama, J. H., & Liu, X. (2008) · Educational Psychology · doi:10.1080/01443410701777272 · Cited by 231 · see citing works
Do NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227 · see citing works
Neural GPUs Learn AlgorithmsPreprint
Kaiser, Ł., & Sutskever, I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08228 · Cited by 224 · see citing works
A Survey of Deep Learning for Mathematical ReasoningPreprint
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10535 · Cited by 213 · see citing works
MathPrompter: Mathematical Reasoning using Large Language Models
Imani, S., Du, L., & Shrivastava, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.4 · Cited by 212 · see citing works
A Goal-Driven Tree-Structured Neural Model for Math Word Problems
Xie, Z., & Sun, S. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/736 · Cited by 203 · see citing works
Bender, A., & Beller, S. (2012) · Cognition · doi:10.1016/j.cognition.2012.05.005 · Cited by 187 · see citing works
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2123433119 · Cited by 187 · see citing works
LILA: A Unified Benchmark for Mathematical Reasoning
Mishra, S., Finlayson, M., Lu, P., Tang, L., Welleck, S., Baral, C., Rajpurohit, T., Tafjord, O., Sabharwal, A., Clark, P., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.17517 · Cited by 179
Kellman, P. J., Massey, C. M., & Son, J. Y. (2009) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2009.01053.x · Cited by 169 · see citing works
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 166 · see citing works
How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation
Huang, D.-Q., Shi, S., Lin, C.-Y., Yin, J., & Ma, W.-Y. (2016) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p16-1084 · Cited by 158 · see citing works
Translating a Math Word Problem to a Expression Tree
Wang, L., Wang, Y., Cai, D., Zhang, D., & Liu, X. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1132 · Cited by 156 · see citing works
Neural Arithmetic Logic UnitsPreprint
Trask, A., Hill, F., Reed, S., Rae, J. W., Dyer, C., & Blunsom, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00508 · Cited by 153 · see citing works
Ontogenetic Origins of Human Integer Representations
Carey, S. E., & Barner, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.004 · Cited by 151 · see citing works
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · see citing works
Leibovich, T., & Ansari, D. (2016) · Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale · doi:10.1037/cep0000070 · Cited by 147 · see citing works
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 3× · see citing works
“The scratchpad method forces the model to explicitly produce intermediate computation steps, which are iteratively refined to generate the final answer (Nye et al., 2021).”
BackgroundMethodGraph-to-Tree Learning for Solving Math Word Problems
Zhang, J., Wang, L., Lee, R. K.-W., Bin, Y., Wang, Y., Shao, J., & Lim, E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.362 · Cited by 139 · see citing works
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based FormalismsPreprint
Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13319 · Cited by 137 · see citing works
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
Miao, S.-Y., Liang, C.-C., & Su, K.-Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.92 · Cited by 135 · see citing works
Neuro-Symbolic Artificial Intelligence: The State of the Art
Hitzler, P., & Sarker, M. K. (2021) · Neuro-Symbolic Artificial Intelligence · doi:10.3233/faia342 · Cited by 130 · see citing works
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Mishra, S. R., Mitra, A., Varshney, N., Sachdeva, B., Clark, P. A., Baral, C. R., & Kalyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.246 · Cited by 122 · see citing works
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.557 · Cited by 122 · see citing works
Early Numeracy Assessment: The Development of the Preschool Early Numeracy Scales
Purpura, D. J., & Lonigan, C. J. (2015) · Early Education and Development · doi:10.1080/10409289.2015.991084 · Cited by 120 · see citing works
NumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108 · see citing works
Number Processing and Calculation – Normative Data from Healthy Adults
Delazer, M., Girelli, L., Granà, A., & Domahs, F. (2003) · The Clinical Neuropsychologist · doi:10.1076/clin.17.3.331.18092 · Cited by 108 · see citing works
From number sense to number symbols. An archaeological perspective
d’Errico, F., Doyon, L., Colagè, I., Queffelec, A., Vraux, E. L., Giacobini, G., Vandermeersch, B., & Maureille, B. (2018) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2016.0518 · Cited by 104 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
Generative Language Modeling for Automated Theorem Proving
Polu, S., & Sutskever, I. (2020) · arXiv · doi:10.48550/arxiv.2009.03393 · Cited by 82 · see citing works
LILA: A Unified Benchmark for Mathematical Reasoning
Mishra, S. R., Finlayson, M., Lu, P., Tang, L., Welleck, S., Baral, C. R., Rajpurohit, T., Tafjord, O., Sabharwal, A., Clark, P. E., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.392 · Cited by 79 · see citing works
(2021) · doi:10.18653/v1/2021.naacl-main.53 · Cited by 78 · see citing works
Bias and Generalization in Deep Generative Models: An Empirical StudyPreprint
Zhao, S., Ren, H., Yuan, A., Song, J., Goodman, N. D., & Ermon, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03259 · Cited by 74 · see citing works
Choi, C. Q. (2021) · IEEE Spectrum · doi:10.1109/mspec.2021.9563958 · Cited by 71 · see citing works
The role of materiality in numerical cognition
Overmann, K. A. (2015) · Quaternary International · doi:10.1016/j.quaint.2015.05.026 · Cited by 68 · see citing works
Testolin, A., Zou, W. Y., & McClelland, J. L. (2020) · Developmental Science · doi:10.1111/desc.12940 · Cited by 62 · see citing works
Self-citationVisual sense of number vs. sense of magnitude in humans and machines
Testolin, A., Dolfi, S., Rochus, M., & Zorzi, M. (2020) · Scientific Reports · doi:10.1038/s41598-020-66838-5 · Cited by 59 · see citing works
Self-citationImpact of Pretraining Term Frequencies on Few-Shot Reasoning
Razeghi, Y., Logan, R. L., Gardner, M., & Singh, S. (2022) · arXiv · doi:10.48550/arxiv.2202.07206 · Cited by 53 · see citing works
Exploring Numeracy in Word Embeddings
Naik, A., Ravichander, A., Rose, C., & Hovy, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1329 · Cited by 49 · see citing works
Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., Shoham, Y., Bata, H., Levine, Y., Leyton-Brown, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.00445 · Cited by 49 · see citing works
Exploring Length Generalization in Large Language Models
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2207.04901 · Cited by 44 · cited 2× · see citing works
“The related question of whether and how recurrent networks generalize to sequences longer than those encountered during training has also been of enduring interest, and definitive solutions to such challenging issue have not yet been found (Anil et al., 2022).”
BackgroundMathematics, word problems, common sense, and artificial intelligencePreprint
Davis, E. (2023) · Bulletin of the American Mathematical Society · doi:10.48550/arxiv.2301.09723 · Cited by 44 · see citing works
Do Language Embeddings capture Scales?
Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.439 · Cited by 37 · see citing works
TALM: Tool Augmented Language Models
Parisi, A., Zhao, Y., & Fiedel, N. (2022) · arXiv · doi:10.48550/arxiv.2205.12255 · Cited by 35 · see citing works
Methods for Numeracy-Preserving Word Embeddings
Sundararaman, D., Si, S., Subramanian, V., Wang, G., Hazarika, D., & Carin, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.384 · Cited by 34 · see citing works
Symbolic Behaviour in Artificial IntelligencePreprint
Santoro, A., Lampinen, A. K., Mathewson, K. W., Lillicrap, T., & Raposo, D. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03406 · Cited by 34 · see citing works
The cultural challenge in mathematical cognition
Beller, S., Bender, A., Chrisomalis, S., Jordan, F. M., Overmann, K. A., Saxe, G. B., & Schlimm, D. (2018) · Journal of Numerical Cognition · doi:10.5964/jnc.v4i2.137 · Cited by 30 · see citing works
Investigating the Limitations of Transformers with Simple Arithmetic Tasks
Nogueira, R., Jiang, Z., & Lin, J. (2021) · arXiv · doi:10.48550/arxiv.2102.13019 · Cited by 29 · see citing works
Discrete and continuous representations and processing in deep learning: Looking forward
Cartuyvels, R., Spinks, G., & Moens, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.07.002 · Cited by 27 · see citing works
Jiang, C., Nian, Z., Guo, K., Chu, S., Zhao, Y., Shen, L., & Tu, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.235 · Cited by 25 · see citing works
The cultural origins of symbolic number.
O’Shaughnessy, D. M., Gibson, E. A. F., & Piantadosi, S. T. (2021) · Psychological Review · doi:10.1037/rev0000289 · Cited by 24 · see citing works
Turan, E., & Smedt, B. D. (2023) · Cognitive Development · doi:10.1016/j.cogdev.2023.101318 · Cited by 24 · see citing works
Banino, A., Balaguer, J., & Blundell, C. (2021) · arXiv · doi:10.48550/arxiv.2107.05407 · Cited by 24 · see citing works
Teaching Algorithmic Reasoning via In-context LearningPreprint
Zhou, H., Nova, A., Larochelle, H., Courville, A. C., Neyshabur, B., & Sedghi, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09066 · Cited by 24 · see citing works
Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model
Pal, K. K., & Baral, C. R. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.265 · Cited by 18 · see citing works
Learning to Prove Theorems by Learning to Generate TheoremsPreprint
Wang, M., & Deng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07019 · Cited by 17 · see citing works
NumGPT: Improving Numeracy Ability of Generative Pre-trained ModelsPreprint
Jin, Z., Jiang, X., Wang, X., Liu, Q., Wang, Y., Ren, X., & Qu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03137 · Cited by 16 · see citing works
The Challenge of Modeling the Acquisition of Mathematical Concepts
Testolin, A. (2020) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.00100 · Cited by 15 · see citing works
Self-citationSymbolic Brittleness in Sequence Models: On Systematic Generalization in Symbolic Mathematics
Welleck, S., West, P., Cao, J., & Choi, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20841 · Cited by 13 · see citing works
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic GeneralizationPreprint
Csordás, R., Irie, K., & Schmidhuber, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07732 · Cited by 11 · see citing works
Have You Seen That Number? Investigating Extrapolation in Question Answering Models
Kim, J.-H., Hong, G., Kim, K.-M., Kang, J., & Myaeng, S.-H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.563 · Cited by 10 · see citing works
NT5?! Training T5 to Perform Numerical ReasoningPreprint
Yang, P., Chen, Y.-T., Chen, Y., & Cer, D. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07307 · Cited by 9 · see citing works
Neural Sequence-to-grid Module for Learning Symbolic Rules
Kim, S., Nam, H., Kim, J., & Jung, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16994 · Cited by 7 · see citing works
Learning Mathematical Properties of Integers
Ryskina, M., & Knight, K. K. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.30 · Cited by 6 · see citing works
“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility
Gupta, H., Varshney, N., Mishra, S. R., Pal, K. K., Sawant, S. A., Scaria, K. J., Goyal, S., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.30 · Cited by 5 · see citing works
Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?
Kudo, K., Aoki, Y., Kuribayashi, T., Brassard, A., Yoshikawa, M., Sakaguchi, K., & Inui, K. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.98 · Cited by 5 · see citing works
Boccato, T., Testolin, A., & Zorzi, M. (2021) · Entropy · doi:10.3390/e23070857 · Cited by 4 · see citing works
Self-citationCognolato, S., & Testolin, A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892619 · Cited by 4 · see citing works
Self-citationSelf-Communicating Deep Reinforcement Learning Agents Develop External Number Representations
Sabathiel, S., Solstad, T., Testolin, A., & Petruzzellis, F. (2022) · Proceedings of the Northern Lights Deep Learning Workshop · doi:10.7557/18.6291 · Cited by 4 · see citing works
Self-citationA Hybrid System for Systematic Generalization in Simple Arithmetic ProblemsPreprint
Petruzzellis, F., Testolin, A., & Sperduti, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.17249 · Cited by 2 · see citing works
Self-citationLearning to solve arithmetic problems with a virtual abacus
Petruzzellis, F., Chen, L., & Testolin, A. (2023) · Proceedings of the Northern Lights Deep Learning Workshop · doi:10.7557/18.6805 · Cited by 1 · see citing works
Self-citationLogical Tasks for Measuring Extrapolation and Rule ComprehensionPreprint
Fujisawa, I., & Kanai, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.07727 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Bundy, A. (1983). The Computer Modelling of Mathematical Reasoning. Academic Press.
- Dehaene, S. (2011). The number sense: How the mind creates mathematics. New York, NY, USA: Oxford University Press.
- Faldu, K., Sheth, A., Kikani, P., Gaur, M., and Avasthi, A. (2021). Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems. arXiv Prepr. doi:10.48550/arXiv.2111.05364. · DOI
- Geva, M., Gupta, A., and Berant, J. (2020). Injecting Numerical Reasoning Skills into Language Models. in The 58th annual meeting of the Association for Computational Linguistics, eds.
- Lample, G., and Charton, F. (2019). Deep Learning for Symbolic Mathematics. arXiv Prepr. doi:10.48550/arXiv.1912.0141. · DOI
- Lee, D., Szegedy, C., Rabe, M. N., Loos, S. M., and Bansal, K. (2019). Mathematical reasoning in latent space. arXiv Prepr. doi:10.48550/arXiv.1909.11851. · DOI
- Madsen, A., and Johansen, A. R. (2020). Neural arithmetic units. in International Conference on Learning Representations doi:10.48550/arXiv.2001.05016. · DOI
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
“The difficulty LLMs have in interpreting numbers 48 and solving simple arithmetic 49 is an active research topic 50,51 .”
Gaber, F., Shaik, M., Allega, F., Bilecz, A. J., Busch, F., Goon, K., Franke, V., & Akalin, A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01684-1 · Cited by 165 · see citing works
Liu, P., Eti, S., Yüksel, S., Dınçer, H., Gökalp, Y., Ergün, E., & Aysan, A. F. (2024) · Renewable Energy · doi:10.1016/j.renene.2024.121081 · Cited by 8 · see citing works
Visual enumeration remains challenging for multimodal generative AI
Testolin, A., Hou, K., & Zorzi, M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0331566 · Cited by 8 · see citing works
Self-citationAli, T. E., Ali, F. I., Pataki, N., & Zoltán, A. D. (2024) · International Conference on Software and System Engineering (ICoSSE) · doi:10.1109/icosse62619.2024.00017 · Cited by 7 · see citing works
Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
Petruzzellis, F., Testolin, A., & Sperduti, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72344-5_18 · Cited by 5 · see citing works
Self-citationFractal Analysis of GPT-2 Token Embedding Spaces: Stability and Evolution of Correlation Dimension
Lee, M. (2024) · Fractal and Fractional · doi:10.3390/fractalfract8100603 · Cited by 5 · see citing works
Sadia, H., & Alam, P. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202400032 · Cited by 4 · see citing works
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
Hirst, E., & Gherardini, T. S. (2024) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.109.106006 · Cited by 3 · see citing works
Large Language Models in Radiologic Numerical Tasks: A Thorough Evaluation and Error Analysis
Nowroozi, A., Bondarenko, M., Serapio, A., Schnitzler, T., Brar, S. S., & Sohn, J. H. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01824-9 · Cited by 2 · see citing works
Rethinking Mathematical Intuition in the Age of AI
Friedman, M., & Bar-On, K. K. (2026) · The Mathematical Intelligencer · doi:10.1007/s00283-026-10561-y · Cited by 1 · see citing works
Local Attention Neural Network for Median Filtering: Enhancing Nonlinear Local Operations
Jung, S., Kim, J. W., Choi, Y., & Kim, J.-S. (2024) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc62082.2024.10827697
Hill Space is All You NeedPreprint
DuJardin, J. (2025) · doi:10.36227/techrxiv.175339930.03949307/v2
Shi, N., Xu, L., Zhu, T., Zhou, W., Meng, W., & Tan, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5606507
Shi, N., Xu, L., Zhu, T., Zhou, W., Meng, W., & Tan, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5606506
Adapting Large Language Models for Solving Mathematical Equation
Kartik, D., Gupta, D., & Singh, V. P. (2025) · International Conference on Contemporary Computing · doi:10.1109/ic366947.2025.11290279
Shi, N., Xu, L., Zhu, T., Zhou, W., Meng, W., & Tan, Y. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103729
Executing Signal Control From Text: LLM Fidelity on a CV Platoon-Based Optimization Algorithm
Alzahrani, A. S., & Alahmadi, M. D. (2026) · IEEE Access · doi:10.1109/access.2026.3673185
Diffractive Arithmetic Processing by Conditional Optical Generative Models
Wang, F., Sun, Y., Zhang, Z., Yang, C., Chen, N., Wang, S., Han, J., Li, Y., Qu, G., Wang, S., et al. (2026) · Laser & Photonics Review · doi:10.1002/lpor.202502983
Labuem, S., & Gaite, C. (2026) · ABDI UNISAP Jurnal Pengabdian Kepada Masyarakat · doi:10.59632/abdiunisap.v4i1.672
A Survey on the Use of Machine Learning within Computer Algebra
Shafiq, U., England, M., & Zaidi, N. A. (2026) · ACM Computing Surveys · doi:10.1145/3831699
Can AI assist in reducing diagnostic error? A narrative review
Scott, I. A. (2026) · Diagnosis · doi:10.1515/dx-2026-0021
How Do Vision-Language Models Represent Numbers? An Investigation on Visual Enumeration
Shafiezadeh, S., Hou, K., Testolin, A., & Zorzi, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38401-0_56
Self-citationP, A. R. K., & Gowrishankar, S. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12507-z
- Authors
- Testolin, A.
- Affiliations
- University of Padua (IT)
- Corresponding author
- Testolin, Alberto
- ORCID
- Testolin, Alberto
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-01-15
- Volume
- 14
- Issue
- 2
- Pages
- 744
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4390876225
- Cited by
- 24
- References
- 88
- Citations by year
- 2026: 10 · 2025: 7 · 2024: 7
- FWCI
- 3.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Mathematics › Statistics and Probability › Cognitive and developmental aspects of mathematical skills
- Keywords
- arithmetic reasoning, numerical understanding, deep learning models, large language models, mathematical reasoning, automated theorem proving, numerical integration, probing tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
