Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard
Open accessAbstract
A comparison between three chatbots which are based on large language models, namely ChatGPT-3.5, ChatGPT-4 and Google Bard is presented, focusing on their ability to give correct answers to mathematics and logic problems. In particular, we check their ability to Understand the problem at hand; Apply appropriate algorithms or methods for its solution; and Generate a coherent response and a correct answer. We use 30 questions that are clear, without any ambiguities, fully described with plain text only, and have a unique, well defined correct answer. The questions are divided into two sets of 15 each. The questions of Set A are 15 "Original" problems that cannot be found online, while Set B contains 15 "Published" problems that one can find online, usually with their solution. Each question is posed three times to each chatbot. The answers are recorded and discussed, highlighting their strengths and weaknesses. It has been found that for straightforward arithmetic, algebraic expressions, or basic logic puzzles, chatbots may provide accurate solutions, although not in every attempt. However, for more complex mathematical problems or advanced logic tasks, their answers, although written in a usually "convincing" way, may not be reliable. Consistency is also an issue, as many times a chatbot will provide conflicting answers when given the same question more than once. A comparative quantitative evaluation of the three chatbots is made through scoring their final answers based on correctness. It was found that ChatGPT-4 outperforms ChatGPT-3.5 in both sets of questions. Bard comes third in the original questions of Set A, behind the other two chatbots, while it has the best performance (first place) in the published questions of Set B. This is probably because Bard has direct access to the internet, in contrast to ChatGPT chatbots which do not have any communication with the outside world.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In answering 30 mathematical questions and logic problems, ChatGPT was superior to Bard on Set A (unavailable online), while Bard had an advantage on Set B (available online) due to internet access.
“The results revealed ChatGPT's superiority in Set (A) over Bard. Nevertheless, Bard's advantage emerged in Set (B) due to its capacity to access the internet directly and retrieve answers, a capability that ChatGPT does not possess [57].”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Transformer architecture : The transformer model, proposed by Vaswani et al. in 2017 [19], is the backbone of ChatGPT.”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“The term “hallucination” [21] describes the phenomenon where AI systems generate outputs that are unrealistic, incorrect, or nonsensical.”
BackgroundELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1× · see citing works
“In the realm of early chatbot development, a significant milestone was reached with the inception of ELIZA [1], an innovative software program crafted by Joseph Weizenbaum in the 1960s.”
MethodGilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · cited 2× · see citing works
“They can be put to the test in various fields to evaluate their understanding and problem-solving abilities, and they can even be tested against actual professional examinations [8].”
BackgroundInteracting with educational chatbots: A systematic review
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11177-3 · Cited by 841 · see citing works
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., Chevalier, A., & Berner, J. (2023) · Neural Information Processing Systems · doi:10.52202/075280-1205 · Cited by 630 · cited 3× · see citing works
“The assessment of ChatGPT’s mathematical capabilities was conducted utilizing a novel dataset introduced within the same research study [9].”
BackgroundMethodInfluentialMathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385 · see citing works
Manson, N. A. (2025) · Sonic Bodies · doi:10.1093/acprof:oso/9780199812097.003.0007 · Cited by 310 · cited 1×
“…figure in the domain of natural language processing and artificial intelligence, and it garnered widespread acclaim for its capacity to emulate conversations between a psychotherapist and a patient, thus representing one of the initial manifestations of AI-driven human–computer interactions [2].”
BackgroundFormal Verification of Smart Contracts Based on Users and Blockchain Behaviors Models
Abdellatif, T., & Brousmiche, K.-L. (2018) · IFIP International Conference on New Technologies, Mobility and Security (NTMS) · doi:10.1109/ntms.2018.8328737 · Cited by 183 · see citing works
Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment
Lai, U. H., Wu, K. S., Hsu, T.-Y., & Kan, J. K. C. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1240915 · Cited by 67 · cited 1× · see citing works
“Despite the ever-expanding body of research dedicated to the examination and evaluation of ChatGPT in several domains [7,8,15], the authors are not aware of comparable investigations being conducted for the other prominent chatbot systems, such as Bard for instance.”
BackgroundFormal Methods for the Verification of Smart Contracts: A Review
Krichen, M., Lahami, M., & Al-Haija, Q. A. (2022) · International Conference on Security of Information and Networks · doi:10.1109/sin56466.2022.9970534 · Cited by 66 · see citing works
Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems
Upadhyay, S., & Chang, M.-W. (2016) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1047 · Cited by 65 · cited 1×
“The evaluation of ChatGPT’s performance in this study was conducted employing the DRAW-1K dataset [11–13], which was meticulously arranged to encompass 1000 algebraic word problems.”
MethodMWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers
Lan, Y., Wang, L., Zhang, Q., Lan, Y., Dai, B. T., Wang, Y., Zhang, D., & Lim, E. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21723 · Cited by 58 · cited 1× · see citing works
“The evaluation of ChatGPT’s performance in this study was conducted employing the DRAW-1K dataset [11–13], which was meticulously arranged to encompass 1000 algebraic word problems.”
MethodLearning from Explicit and Implicit Supervision Jointly For Algebra Word Problems
Upadhyay, S., Chang, M.-W., Chang, K.-W., & Yih, W.-T. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1029 · Cited by 52 · see citing works
Cheng, H. (2023) · AI · doi:10.3390/ai4020021 · Cited by 45 · see citing works
Why Does ChatGPT Fall Short in Providing Truthful Answers?
Zheng, S., Huang, J., & Chang, K. C.-C. (2023) · arXiv · doi:10.48550/arxiv.2304.10513 · Cited by 40 · cited 1× · see citing works
“Understand the problem : Chatbots must be able to accurately interpret the user’s input, recognize the type of problem being posed, and identify the relevant information needed to solve it.”
BackgroundAn Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)Preprint
Shakarian, P., Koyyalamudi, A., Ngu, N., & Mareedu, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.13814 · Cited by 39 · see citing works
Alruqi, T. N., & Alzahrani, S. M. (2023) · AI · doi:10.3390/ai4030035 · Cited by 28 · cited 1× · see citing works
“As a result, it can understand and generate text in several languages, such as Spanish, French, German, Chinese, Arabic [20], and more.”
BackgroundTatai, G., Csordás, A., Kiss, Á., Szaló, A., & Laufer, L. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39396-2_2 · Cited by 27 · see citing works
Maximum Margin Reward Networks for Learning from Explicit and Implicit Supervision
Peng, H., Chang, M.-W., & Yih, W.-T. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1252 · Cited by 22 · cited 1× · see citing works
“In this study, the DRAW-1K dataset [8-10] was used to evaluate the performance of ChatGPT.”
MethodDesign an Intelligent System to automatically Tutor the Method for Solving Problems
Nguyen, H. D., Tran, D. A., Do, H. P., & Pham, V. T. (2020) · International Journal of Integrated Engineering · doi:10.30880/ijie.2020.12.07.024 · Cited by 21 · cited 1× · see citing works
“A chatbot is a computer program or artificial intelligence (AI) system designed to engage in conversation with human users through text or voice-based interactions [2].”
BackgroundAI Chatbots: Threat or Opportunity?
Bryant, A. (2023) · Informatics · doi:10.3390/informatics10020049 · Cited by 19 · see citing works
Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word ProblemsPreprint
Upadhyay, S., & Chang, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07197 · Cited by 7 · see citing works
Plevris, V., Papazafeiropoulos, G., & Ríos, A. J. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7940781 · Cited by 2 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 2×
“Some of the improvements and advanced features of GPT-4 include [18]: 1.”
Background
Flagged first, then by citations
15 of the 40 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A systematic literature review of empirical research on ChatGPT in education
Albadarin, Y., Saqr, M., Pope, N., & Tukiainen, M. (2024) · Discover Education · doi:10.1007/s44217-024-00138-2 · Cited by 191 · see citing works
AlShaikh, R., Al-Malki, N. A., & Almasre, M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25361 · Cited by 151 · see citing works
“Secondly, Bard tends to offer more detailed and precise answers to factual questions, while ChatGPT tends to provide more concise and occasionally vague responses [50].”
BackgroundPlevris, V., Papazafeiropoulos, G., & Ríos, A. J. (2023) · AI · doi:10.3390/ai4040048 · Cited by 105 · see citing works
Self-citationEnough of the chit-chat: A comparative analysis of four AI chatbots for calculus and statistics
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.22 · Cited by 50 · see citing works
A Systematic Literature Review of Empirical Research on ChatGPT in EducationPreprint
Albadarin, Y., Tukiainen, M., Saqr, M., & Pope, N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4562771 · Cited by 26 · see citing works
Teaching Gifted Students in Mathematics: A Literature Review
Trpin, A. (2024) · International Journal of Childhood Education · doi:10.33422/ijce.v5i1.498 · Cited by 13 · see citing works
“…praising the problem-solving abilities of GPT-4 and ChatGPT, particularly in exam-like scenarios (which may be the result of selective reporting), their overall mathematical proficiency does not meet the rigorous standards expected at the graduate level (Frieder et al., 2023, Plevris et al., 2023).”
BackgroundAnalyzing the Competitive Mathematical Problem-Solving Skills of ChatGPT
Teegavarapu, R. R., & Sanghvi, H. A. (2023) · International Conference on Data Science, Agents & Artificial Intelligence (ICDSAAI) · doi:10.1109/icdsaai59313.2023.10452659 · Cited by 5 · see citing works
“[15] A broader study about AI in mathematical research (Davis, 2023) concluded that limitations existed in AI when solving today’s mathematical problems, which makes the future of AI in pure mathematical research unclear.”
BackgroundIslam, H. M. A., Hasan, M., Ahmed, S., Fardin, A. I., & Nabil, M. H. (2024) · Preprints.org · doi:10.20944/preprints202402.1083.v1
Sorel, M., Gurrala, C., & Tadinada, A. (2025) · Cureus · doi:10.7759/cureus.89149
- Authors
- Plevris, V., Papazafeiropoulos, G., & Rios, A. J.
- Affiliations
- Qatar University (QA)
- National Technical University of Athens (GR)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 4
- Issue
- 4
- Pages
- 949-969
- ISSN
- 2673-2688 (ISSN-L)
- arXiv
- 2305.18618
- OpenAlex
- W4387909633
- Cited by
- 90
- References
- 40
- Citations by year
- 2026: 16 · 2025: 34 · 2024: 34 · 2023: 6
- FWCI
- 21.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- AI (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Qatar University QSpace (Qatar University) (submittedVersion, other-oa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- chatbots, large language models, ChatGPT-3.5, ChatGPT-4, Google Bard, mathematical problem solving, logic puzzles, arithmetic, algebra, answer accuracy, consistency, internet access, artificial intelligence in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
