The DOI matches its OpenAlex record; 6 of the 6 references it cites were found in a registry or an index.
Mathematical Capabilities of ChatGPT
Open accessAbstract
We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 2×
“In some cases, such as the LaMDA model [42] or BLOOM [19], both released also in 2022 by Google, an evaluation of mathematical reasoning capability is missing entirely.”
BackgroundMethod(2022) · doi:10.52202/068431-0278 · Cited by 358
History of Interactive Theorem Proving
Harrison, J., Urban, J., & Wiedijk, F. (2014) · Handbook of the History of Logic · doi:10.1016/b978-0-444-51624-4.50004-6 · Cited by 107
Cited by
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development
Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123618 · Cited by 1,452
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
ChatGPT for tourism: applications, benefits and risks
Carvalho, I., & Ivanov, S. (2024) · Tourism Review · doi:10.1108/tr-02-2023-0088 · Cited by 507
ChatGPT: A revolutionary tool for teaching and learning mathematics
Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023) · Eurasia Journal of Mathematics, Science and Technology Education · doi:10.29333/ejmste/13272 · Cited by 470
Use chat gpt to solve programming bugs
Surameery, N. M. S., & Shakor, M. Y. (2023) · International Journal of Information technology and Computer Engineering · doi:10.55529/ijitc.31.17.22 · Cited by 382
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 370
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., Lyden, S., Neal, P., & Sandison, C. (2023) · European Journal of Engineering Education · doi:10.1080/03043797.2023.2213169 · Cited by 336
Generative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293
A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges
Nazir, A., & Wang, Z. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100022 · Cited by 245
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Lai, C. Y., Cheung, K. Y., & Chan, C. S. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100178 · Cited by 230
Generative AI without guardrails can harm learning: Evidence from high school mathematics
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2422633122 · Cited by 309
An Examination of the Use of Large Language Models to Aid Analysis of Textual Data
(2024) · doi:10.1177/16094069241231168 · Cited by 227
Tossell, C. C., Tenhundfeld, N. L., Momen, A., Cooley, K., & de Visser, E. J. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3355015 · Cited by 218
Albayati, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100203 · Cited by 188
Role of activity-based learning and ChatGPT on students' performance in education
Al Shloul, T., Mazhar, T., Abbas, Q., Iqbal, M., Ghadi, Y. Y., Shahzad, T., Mallek, F., & Hamam, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100219 · Cited by 173
“ChatGPT ’ s mathematical abilities are significantly inadequate to those of an average college student (Frieder et al., 2024; Wardat et al., 2023).”
BackgroundLet's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · arXiv · doi:10.48550/arxiv.2302.13817 · Cited by 167
Let’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167
Ellis, A. R., & Slade, E. (2023) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2023.2223609 · Cited by 137
Language Models and Cognitive Automation for Economic Research
Korinek, A. (2023) · doi:10.3386/w30957 · Cited by 129
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
“However, ChatGPT’s performance was found to be below average in physics (Korte-meyer, 2023) and in various mathematical domains, such as Olympiad problem-solving, functional analysis, and symbolic integration (Frieder et al., 2023).”
BackgroundChatgpt: a meta-analysis after 2.5 months
Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., & Eger, S. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100541 · Cited by 115
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
“There is a great deal of lively debate surrounding its potential uses, impact on research [4], science [5], and education [1,3,6].”
BackgroundAssessing the Benefits of ChatGPT for Business: An Empirical Study on Organizational Performance
Chu, M.-N. (2023) · IEEE Access · doi:10.1109/access.2023.3297447 · Cited by 69
Han, X., Peng, H., & Liu, M. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100714 · Cited by 70
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R., Ribeiro, A., Robinson, C., Loeb, S., & Demszky, D. (2024) · doi:10.21203/rs.3.rs-5363154/v1 · Cited by 53
Mei, P., Brewis, D. N., Nwaiwu, F., Sumanathilaka, D., Alva-Manchego, F., & Demaree-Cotton, J. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100140 · Cited by 50
Towards AI-Assisted Synthesis of Verified Dafny Methods
Misu, M. R. H., Lopes, C. V., Ma, I., & Noble, J. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643763 · Cited by 38
“Dafny-Synthesis : https://github.com/Mondego/dafny-synthesis”
Background‘Fighting fire with fire’ — using LLMs to combat LLM hallucinations
Verspoor, K. (2024) · Nature · doi:10.1038/d41586-024-01641-0 · Cited by 33
Artificial intelligence—friend or foe in fake news campaigns
Węcel, K., Sawiński, M., Stróżyna, M., Lewoniewski, W., Księżniak, E., Stolarski, P., & Abramowicz, W. (2023) · Economics and Business Review · doi:10.18559/ebr.2023.2.736 · Cited by 28
Bonsu, E., & Baffour-Koduah, D. (2023) · doi:10.2139/ssrn.4387107 · Cited by 26
“Also, Lametti (2022) and Frieder et al. (2023) expressed that AI systems failed to solve mathematical problems. Consequently, Bang et al. (2023) comprehensively confirmed the performance of ChatGPT based on a 64.”
BackgroundNezhurina, M., Cipolina-Kun, L., Cherti, M., & Jitsev, J. (2024) · arXiv · doi:10.48550/arxiv.2406.02061 · Cited by 21
Markowitz, D. M. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02344-0 · Cited by 19
Exploring ChatGPT ‐Facilitated Scaffolding in Undergraduates' Mathematical Problem Solving
Zhou, R., He, X., Fan, Q., Li, Y., Li, Y., Xiao, X., & Fang, J. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70077 · Cited by 15
Plevris, V., Papazafeiropoulos, G., & Rios, A. J. (2023) · arXiv · doi:10.48550/arxiv.2305.18618 · Cited by 90
“The assessment of ChatGPT’s mathematical capabilities was conducted utilizing a novel dataset introduced within the same research study [9].”
BackgroundMethodInfluentialLanguage-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
Self-citationPARENTS’ VIEWS ON THE USE OF AI-BASED CHATBOTS SUCH AS ChatGPT IN HIGH SCHOOL (STEM) EDUCATION
Feser, M. S. (2024) · Journal of Baltic Science Education · doi:10.33225/jbse/24.23.04 · Cited by 7
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024) · arXiv · doi:10.48550/arxiv.2410.03017 · Cited by 6
“As a result, LMs often generate bad pedagogical responses, such as giving away answers rather than fostering critical thinking (Frieder et al., 2023, Singer, 2023, Wang and Demszky, 2023), and these shortcomings have negatively impacted students’ educational outcomes (Bastani et al., 2024, Nie et…”
Lv, P., Wang, X., Nie, Y., Wu, R., & Wang, L. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1633686 · Cited by 5
From Tool to Method: Developing AI Literacy in an Upper-Level History Course
Dumančić, M., & Stickle, T. (2026) · Journal of Teaching and Learning with Technology · doi:10.14434/jotlt.v14i1.41669
Metadata
- Authors
- Frieder, S., Pinchetti, L., Chevalier, C., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., & Berner, J.
- Venue
- neural information processing systems
- Published
- 2023-01-01
- Pages
- 27699-27744
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133223925
- Cited by
- 312
- References
- 6
- FWCI
- 58.98
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.