CoChat
Sign Up
Cited by 3Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

How Good Are Large Language Models at Arithmetic Reasoning in Low-Resource Language Settings?—A Study on Yorùbá Numerical Probes with Minimal Contamination

Open access
Oyesanmi, Fiyinfoluwa, Olukanmi, Peter O.·2025·Applied Sciences
DOI 10.3390/app15084459
Open this paper in CoChat — free accountOpen PDF

Abstract

We study the performance of large language models (LLMs) in natural language understanding and natural language reasoning tasks in a low-resourced-language (LRL) setting. Using Yorùbá, an LRL, we curated a set of numerical probes with minimal contamination. The probes comprise three sets of questions—the first covers basic arithmetic, the second covers date and time (calendar system), and the last focuses on numerals and counting systems. Assessed in a zero-shot setup, three LLMs (ChatGPT, Gemini, and PaLM) were evaluated based on several metrics. The best-performing model, ChatGPT, generated some correct answers, showing logical steps in attaining the answers in Yorùbá (with an accuracy of 56% in set one, and 44% in set two). The second-best model (with an accuracy of 56% in set one, and 32% in set two) is Gemini. PaLM (with an accuracy of 16% in set one, and 8% in set two) showed the answers without logic. The three models performed poorly on the Yorùbá numerals question set (ChatGPT scored 8%, and Gemini and PaLM each had 0% accuracy). The study also revealed that there is significant room for improvement in the state of the art of LLMs when it comes to Yorùbá numerals.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/app15084459

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Numerical reasoning introduces further difficulties, especially when values must be compared, computed, or correctly grounded to specific entities and variables.

    “Numerical reasoning introduces further difficulties, especially when values must be compared, computed, or correctly grounded to specific entities and variables [16-18, 51, 52].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [18]

    Section 4.1 reports difficulties with computations and with representing Yorùbá numerals accurately; it says low numerical-representation accuracy affected the models’ final computations. The passages do not address comparing values or grounding values to specific entities and variables, so those parts of the citing sentence are not supported by this source. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are especially relevant, but are rarely examined within one unified setting.

    “Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are therefore especially relevant, but are rarely examined within one unified setting [11][12][13][14][15][16][17][18].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [18]

    Sections 3.1 and 4.1 examine combined arithmetic operations, step-by-step reasoning, and accurate representation of Yorùbá numerals, supporting the compositional-reasoning and numerical-grounding parts of the claim. The paper does not establish limitations in entity binding or long-context evidence use, or say that these limitations are rarely examined together. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Kasai, J.; Kasai, Y.; Sakaguchi, K.; Yamada, Y.; Radev, D. Evaluating gpt-4 and chatgpt on japanese medical licensing examinations. arXiv 2023, arXiv:2303.18027. · arXiv
  2. Wu, J.; Wu, X.; Qiu, Z.; Li, M.; Lin, S.; Zhang, Y.; Zheng, Y.; Yuan, C.; Yang, J. Large language models leverage external knowledge to extend clinical insight beyond language boundaries. J. Am. Med. Inform. Assoc. 2024, 34 , 2054-2064. [CrossRef] [PubMed] · DOI · Paper page
  3. Wang, X.; Hu, Z.; Lu, P.; Zhu, Y.; Zhang, J.; Subramaniam, S.; Loomba, A.R.; Zhang, S.; Sun, Y.; Wang, W. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv 2023, arXiv:2307.10635. · arXiv
  4. Imani, S.; Du, L.; Shrivastava, H. Mathprompter: Mathematical reasoning using large language models. arXiv 2023, arXiv:2303.05398. · arXiv
  5. Peng, X.; Geng, X. Self-controller: Controlling LLMs with Multi-round Step-by-step Self-awareness. arXiv 2024, arXiv:2410.00359. · arXiv
  6. Ji, K.; Chen, J.; Gao, A.; Xie, W.; Wan, X.; Wang, B. LLMs Could Autonomously Learn Without External Supervision. arXiv 2024, arXiv:2406.00606. · arXiv
  7. Qin, C.; Zhang, A.; Zhang, Z.; Chen, J.; Yasunaga, M.; Yang, D. Is ChatGPT a general-purpose natural language processing task solver? arXiv 2023, arXiv:2302.06476. · arXiv
  8. Kojima, T.; Gu, S.S.; Reid, M.; Matsuo, Y.; Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. arXiv 2022, arXiv:2205.11916. · arXiv
  9. Li, L.; Wang, Y.; Zhao, H.; Kong, S.; Teng, Y.; Li, C.; Wang, Y. Reflection-Bench: Probing AI intelligence with reflection. arXiv 2024, arXiv:2410.16270. · arXiv
  10. Fang, X.; Xu, W.; Tan, F.A.; Zhang, J.; Hu, Z.; Qi, Y.; Nickleach, S.; Socolinsky, D.; Sengamedu, S.H.; Faloutsos, C. Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding-A Survey. arXiv 2024, arXiv:2402.17944. · arXiv
  11. Kadlčík, M.; Štefánik, M. Self-training Language Models for Arithmetic Reasoning. arXiv 2024, arXiv:2407.08400. · arXiv
  12. Cai, C.; Zhao, X.; Liu, H.; Jiang, Z.; Zhang, T.; Wu, Z.; Hwang, J.N.; Li, L. The Role of Deductive and Inductive Reasoning in Large Language Models. arXiv 2024, arXiv:2410.02892. · arXiv
  13. He, P.; Li, Z.; Xing, Y.; Li, Y.; Tang, J.; Ding, B. Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning. arXiv 2024, arXiv:2410.19000. · arXiv
  14. Akhtar, M.; Shankarampeta, A.; Gupta, V.; Patil, A.; Cocarascu, O.; Simperl, E.P.B. Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data. arXiv 2023, arXiv:2311.02216. · arXiv
  15. Shao, Z.; Huang, F.; Huang, M. Chaining Simultaneous Thoughts for Numerical Reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 7-9 December 2022.
  16. Mishra, S.; Mitra, A.; Varshney, N.; Sachdeva, B.S.; Clark, P.; Baral, C.; Kalyan, A. NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, 22-27 May 2022.
  17. Zhou, F.; Dong, H.; Liu, Q.; Cheng, Z.; Han, S.; Zhang, D. Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems. arXiv 2022, arXiv:2210.05075. · arXiv
  18. Heston, T.F.; Khun, C. Prompt Engineering in Medical Education. Int. Med. Educ. 2023, 2, 198-205. [CrossRef] · DOI
  19. Wong, W. Practical Approach to Knowledge-based Question Answering with Natural Language Understanding and Advanced Reasoning. arXiv 2007, arXiv:0707.3559. · arXiv
  20. Yu, F.; Zhang, H.; Wang, B. Natural Language Reasoning, A Survey. Acm Comput. Surv. 2023, 56, 1-39. [CrossRef] · DOI
  21. Rybak, P. Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matching. In Proceedings of the International Conference on Language Resources and Evaluation, Turin, Italy, 20-25 May 2024.
  22. Cahyawijaya, S.; Winata, G.I.; Wilie, B.; Vincentio, K.; Li, X.; Kuncoro, A.; Ruder, S.; Lim, Z.Y.; Bahar, S.; Khodra, M.L.; et al. IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation. arXiv 2021, arXiv:2104.08200. · arXiv
  23. Joshua, A.M. Improving Question-Answering Capabilities in Large Language Models Using Retrieval Augmented Generation (RAG): A Case Study on Yoruba Culture and Language. In Proceedings of the 5th Workshop on African Natural Language Processing, Vienna, Austria, 11 May 2024.
  24. Kale, M.; Rastogi, A. Few-Shot Natural Language Generation by Rewriting Templates. arXiv 2020, arXiv:2004.15006. · arXiv
  25. Adelani, D.I.; Liu, H.; Shen, X.; Vassilyev, N.; Alabi, J.O.; Mao, Y.; Gao, H.; Lee, A.E.S. SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, 2-6 May 2023.
  26. Sánchez-Salido, E.; Morante, R.; Gonzalo, J.; Marco, G.; de Albornoz, J.C.; Plaza, L.; Amig'o, E.; Fern'andez, A.A.; Benito-Santos, A.; Espinosa, A.G.; et al. Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination. arXiv 2024, arXiv:2409.12746. · arXiv
  27. Perelkiewicz, M.; Poswiata, R. A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training. arXiv 2024, arXiv:2407.07630. · arXiv
  28. Liu, Y.; Xu, M.; Wang, S.; Yang, L.; Wang, H.; Liu, Z.; Kong, C.; Chen, Y.; Sun, M.; Yang, E. OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models. arXiv 2024, arXiv:2402.13524. · arXiv
  29. Wang, Y.; Zhao, Y. Gemini in reasoning: Unveiling commonsense in multimodal large language models. arXiv 2023, arXiv:2312.17661. · arXiv
  30. Anil, R.; Dai, A.M.; Firat, O.; Johnson, M.; Lepikhin, D.; Passos, A.; Shakeri, S.; Taropa, E.; Bailey, P.; Chen, Z.; et al. Palm 2 technical report. arXiv 2023, arXiv:2305.10403. · arXiv
  31. Shi, F.; Suzgun, M.; Freitag, M.; Wang, X.; Srivats, S.; Vosoughi, S.; Chung, H.W.; Tay, Y.; Ruder, S.; Zhou, D.; et al. Language models are multilingual chain-of-thought reasoners. arXiv 2022, arXiv:2210.03057. · arXiv
  32. Xu, F.; Lin, Q.; Han, J.; Zhao, T.; Liu, J.; Cambria, E. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond. IEEE Trans. Knowl. Data Eng. 2025, 37, 1620-1634. [CrossRef] · DOI
  33. Huang, Y.; Tang, K.; Chen, M. A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry. arXiv 2024, arXiv:2404.15777. · arXiv
  34. Nguyen, T.H.; Le, A.C.; Nguyen, V.C. ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models. arXiv 2024, arXiv:2404.11086. · arXiv
  35. Adetomiwa, A. Yorùbá Numeral System in 21st Century: Challenges and Prospects. 2023. Available online: https: //www.researchgate.net/publication/375096095_YORUBA_NUMERAL_SYSTEM_IN_21ST_CENTURY_CHALLENGES_ AND_PROSPECTS (accessed on 20 February 2025).

Cited by

Sorted by citations

Authors
Oyesanmi, F., & Olukanmi, P. O.
Affiliations
University of Johannesburg (ZA)
Corresponding author
  • Oyesanmi, Fiyinfoluwa
  • Olukanmi, Peter O.
ORCID
Venue
Applied Sciences
Publisher
MDPI AG
Published
2025-04-17
Volume
15
Issue
8
Pages
4459
ISSN
2076-3417 (ISSN-L)
DOI
10.3390/app15084459
OpenAlex
W4409535830
Cited by
3
References
3
Citations by year
2026: 2 · 2025: 1
FWCI
0.97
Type
journal article
Language
en
Open access
Licence
Locations
  • Applied Sciences (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, arithmetic reasoning, low-resource languages, Yoruba, numerals, counting systems, zero-shot evaluation, ChatGPT, Gemini, Arecaceae, language contamination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review