CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Investigating the Effectiveness of ChatGPT in Mathematical Reasoning and Problem Solving: Evidence from the Vietnamese National High School Graduation Examination

Dao, Xuan-Quy, Le, Ngoc-Bich·2023·arXiv
DOI 10.48550/arxiv.2306.06331arXiv 2306.06331
Open this paper in CoChat — free account

Abstract

This study offers a complete analysis of ChatGPT's mathematics abilities in responding to multiple-choice questions for the Vietnamese National High School Graduation Examination (VNHSGE) on a range of subjects and difficulty levels. The dataset included 250 questions divided into four levels: knowledge (K), comprehension (C), application (A), and high application (H), and it included ten themes that covered diverse mathematical concepts. The outcomes demonstrate that ChatGPT's performance varies depending on the difficulty level and subject. It performed best on questions at Level (K), with an accuracy rate of $83\%$; but, as the difficulty level rose, it scored poorly, with an accuracy rate of $10\%$. The study has also shown that ChatGPT significantly succeeds in providing responses to questions on subjects including exponential and logarithmic functions, geometric progression, and arithmetic progression. The study found that ChatGPT had difficulty correctly answering questions on topics including derivatives and applications, spatial geometry, and Oxyz spatial calculus. Additionally, this study contrasted ChatGPT outcomes with Vietnamese students in VNHSGE and in other math competitions. ChatGPT dominated in the SAT Math competition with a success rate of $70\%$, followed by VNHSGE mathematics ($58.8\%)$. However, its success rates were lower on other exams, such as AP Statistics, the GRE Quantitative, AMC 10, AMC 12, and AP Calculus BC. These results suggest that ChatGPT has the potential to be an effective teaching tool for mathematics, but more work is needed to enhance its handling of graphical data and address the challenges presented by questions that are getting more challenging.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.06331

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ChatGPT is not good at answering questions on topics including derivatives and applications, Oxyz spatial calculus, and spatial geometry.

    “ChatGPT is not good at answering questions on topics including derivatives and applications, Oxyz spatial calculus, and spatial geometry [31].”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [31]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Jianxing He, Sally L Baxter, Jie Xu, Jiming Xu, Xingtao Zhou, and Kang Zhang. The practical implementation of artificial intelligence technologies in medicine. Nature medicine, 25(1):30-36, 2019.
  2. Lijia Chen, Pingping Chen, and Zhijian Lin. Artificial intelligence in education: A review. Ieee Access, 8:75264- 75278, 2020.
  3. Bill Cope, Mary Kalantzis, and Duane Searsmith. Artificial intelligence for education: Knowledge and its assessment in ai-enabled learning ecologies. Educational Philosophy and Theory, 53(12):1229-1245, 2021.
  4. Xuan-Quy Dao, Ngoc-Bich Le, and Thi-My-Thanh Nguyen. Ai-powered moocs: Video lecture generation. In 2021 3rd International Conference on Image, Video and Signal Processing, pages 95-102, 2021.
  5. Thi-My-Thanh Nguyen, Thanh-Hai Diep, Bac-Bien Ngo, Ngoc-Bich Le, and Xuan-Quy Dao. Design of online learning platform with vietnamese virtual assistant. In 2021 6th International Conference on Intelligent Information Technology, pages 51-57, 2021.
  6. Raju Vaishya, Mohd Javaid, Ibrahim Haleem Khan, and Abid Haleem. Artificial intelligence (ai) applications for covid-19 pandemic. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 14(4):337-339, 2020.
  7. Shanshan Gao. Innovative teaching of integration of artificial intelligence and university mathematics in big data environment. In IOP Conference Series: Materials Science and Engineering, volume 750, page 012137. IOP Publishing, 2020.
  8. Stefan AD Popenici and Sharon Kerr. Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(1):1-13, 2017.
  9. Olaf Zawacki-Richter, Victoria I Marín, Melissa Bond, and Franziska Gouverneur. Systematic review of research on artificial intelligence applications in higher education-where are the educators? International Journal of Educational Technology in Higher Education, 16(1):1-27, 2019.
  10. Mostafa Zafari, Jalal Safari Bazargani, Abolghasem Sadeghi-Niaraki, and Soo-Mi Choi. Artificial intelligence applications in k-12 education: A systematic literature review. IEEE Access, 2022.
  11. Sayed Fayaz Ahmad, Mohd Khairil Rahmat, Muhammad Shujaat Mubarik, Muhammad Mansoor Alam, and Syed Irfan Hyder. Artificial intelligence and its role in education. Sustainability, 13(22):12902, 2021.
  12. Seungsu Paek and Namhyoung Kim. Analysis of worldwide research trends on the impact of artificial intelligence in education. Sustainability, 13(14):7941, 2021.
  13. Lanqin Zheng, Jiayu Niu, Lu Zhong, and Juliana Fosua Gyasi. The effectiveness of artificial intelligence on learning achievement and learning perception: A meta-analysis. Interactive Learning Environments, pages 1-15, 2021.
  14. Adam Gamoran and Eileen C Hannigan. Algebra for everyone? benefits of college-preparatory mathematics for students with diverse abilities in early secondary school. Educational Evaluation and Policy Analysis, 22(3):241-254, 2000.
  15. Robert Parris Moses, Charles E Cobb, et al. Radical equations: Math literacy and civil rights. Technical report, Beacon Press, 2002.
  16. Mohamed Zulhilmi bin Mohamed, Riyan Hidayat, Nurain Nabilah binti Suhaizi, Muhamad Khairul Hakim bin Mahmud, Siti Nurshafikah binti Baharuddin, et al. Artificial intelligence in mathematics education: A systematic literature review. International Electronic Journal of Mathematics Education, 17(3):em0694, 2022.
  17. Sunghwan Hwang. Examining the effects of artificial intelligence on elementary students' mathematics achieve- ment: A meta-analysis. Sustainability, 14(20):13185, 2022.
  18. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Available at SSRN 4389233, 2023.
  19. Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1):e45312, 2023.
  20. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Chris- tian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867, 2023. · arXiv
  21. OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  22. Xuan-Quy Dao, Ngoc-Bich Le, The-Duy Vo, Xuan-Dung Phan, Bac-Bien Ngo, Van-Tien Nguyen, Thi-My-Thanh Nguyen, and Hong-Phuoc Nguyen. Vnhsge: Vietnamese high school graduation examination dataset for large language models. arXiv preprint arXiv:2305.12199, 2023. · arXiv
  23. Ali Borji. A categorical archive of chatgpt failures. arXiv preprint arXiv:2302.03494, 2023. · arXiv
  24. Yousef Wardat, Mohammad A Tashtoush, Rommel AlAli, and Adeeb M Jarrah. Chatgpt: A revolutionary tool for teaching and learning mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19(7):em2286, 2023.
  25. Chung Kwan Lo. What is the impact of chatgpt on education? a rapid review of the literature. Education Sciences, 13(4):410, 2023.

Cited by

Sorted by citations

Metadata

Authors
Dao, X.-Q., & Le, N.-B.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2306.06331
arXiv
2306.06331
References
1
Type
Preprint
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review