CoChat
Sign Up
Cited by 50Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

Enough of the chit-chat: A comparative analysis of four AI chatbots for calculus and statistics

Open access
David Santandreu Calonge, Linda Smail, Firuz Kamalov·2023·Journal of Applied Learning & Teaching
DOI 10.37074/jalt.2023.6.2.22
Open this paper in CoChat — free accountOpen PDF

Abstract

This article presents a comparative analysis of four AI chatbots with potential utilization in the fields of mathematics education and statistics, namely ChatGPT, GPT-4, Bard, and LLaMA. Our objective is to evaluate and compare the features, functionalities, and potential applications of these platforms within the domains of calculus and statistics. By examining their strengths and limitations, this study aims to provide insights into the selection and implementation of AI chatbots in calculus and statistics to enhance student learning. The results of the comparative analysis reveal that, while not perfect, GPT-4 outperforms ChatGPT, Bard, and LLaMA as a learning tool in calculus and statistics. Findings also reveal that chatbots may have a positive transformational impact on higher education.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.37074/jalt.2023.6.2.22

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI helps non-experts analyze data, but it can be unpredictable for experts dealing with complex data analysis.

    “AI helps non-experts analyze data, but it can be unpredictable for experts dealing with complex data analysis.”

    From Artificial Intelligence in Detecting Statistical Errors: Implications for Authors, Reviewers, and Editors (2025) · cites this paper as [20]

    The 'Use cases in calculus and statistics' section says chatbots can guide students through data analysis, and the Summary says their responses vary greatly in accuracy. The paper does not establish a contrast between non-experts and experts or specifically assess experts doing complex data analysis. Page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. Turbo, and GPT-4 Mirroring math anxiety in high- school students. Big Data and Cognitive Computing, 7(3), 124 https://doi.org/10.3390/bdcc703012424 · DOI
  2. Carlander-Reuterfelt, D., Carrera, Á., Iglesias, C. A., Araque, Ó., Rada, J. F. S., & Muñoz, S. (2020). JAICOB: A data science chatbot. IEEE Access, 8, 180672-180680. https://doi.org/ 10.1109/ACCESS.2020.3024795. · DOI
  3. Castelvecchi, D. (2023). How will AI change mathematics? Rise of chatbots highlights discussion. Nature, 615(7950), 15-16. https://doi.org/10.1038/d41586-023-00487-2 · DOI
  4. Chaka, C. (2023). Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning and Teaching, 6(2), 1-11. Advanced online publication. https://doi.org/10.37074/jalt.2023.6.2.12 · DOI · Paper page
  5. Chen, Y., Jensen, S., Albert, L. J., Gupta, S., & Lee, T. (2023). Artificial intelligence (AI) student assistants in the classroom: Designing chatbots to support student success. Information Systems Frontiers, 25(1), 161-182. https://doi.org/10.1007/ s10796-022-10291-4 · DOI · Paper page
  6. Hultberg, P., Calonge, D. S., & Lee, A. E. S. (2018). Promoting long-lasting learning through instructional design. Journal of the Scholarship of Teaching and Learning, 18(3), 26-43. https://doi.org/10.14434/josotl.v18i3.23179 · DOI
  7. Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning-Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237-257. https://doi. org/10.1111/jcal.12610 · DOI · Paper page
  8. Hwang, G. J., & Chang, C. Y. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 1-14. https://doi.org/10.1080/10494820.2021 .1952615 · DOI · Paper page
  9. Ifelebuegu, A. (2023). Rethinking online assessment strategies: Authenticity versus AI chatbot intervention. Journal of Applied Learning and Teaching, 6(2), 1-8. Advanced online publication. https://doi.org/10.37074/jalt.2023.6.2.2 · DOI
  10. Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2023). Who answers it better? An in-depth analysis of ChatGPT and Stack overflow answers to software engineering questions. arXiv preprint arXiv:2308.02312. · arXiv
  11. Limna, P., Kraiwanit, T., Jangjarat, K., Klayklung, P., & Chocksathaporn, P. (2023). The use of ChatGPT in the digital era: Perspectives on chatbot implementation. Journal of Applied Learning and Teaching, 6(1), 64-74. https://doi. org/10.37074/jalt.2023.6.1.32 · DOI
  12. Liu, T., & Low, B. K. H. (2023). Goat: fine-tuned LLaMA outperforms GPT-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201. · arXiv
  13. Mogavi, R. H., Deng, C., Kim, J. J., Zhou, P., Kwon, Y. D., Metwally, A. H. S., ... & Hui, P. (2023). Exploring user perspectives on chatgpt: Applications, perceptions, and implications for AI- integrated education. arXiv preprint arXiv:2305.13114. · arXiv
  14. Neumann, A. T., Arndt, T., Kobis, L., Meissner, R., Martin, A., de Lange, P., Pengel, N., Klamma, R., & Wollersheim, H. W. (2021). Chatbots as a tool to scale mentoring processes: Individually supporting self-study in higher education. Frontiers in Artificial Intelligence, 4, 668220. https://doi. org/10.3389/frai.2021.668220 · DOI
  15. Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi. org/10.1016/j.caeai.2021.100033 · DOI · Paper page
  16. Popenici, S. (2023). The critique of AI as a foundation for judicious use in higher education. Journal of Applied Learning and Teaching, 6(2), 1-7. https://doi.org/10.37074/ jalt.2023.6.2.4 · DOI · Paper page
  17. Sánchez-Ruiz, L. M., Moll-López, S., Nuñez-Pérez, A., Moraño-Fernández, J. A., & Vega-Fleitas, E. (2023). ChatGPT challenges blended learning methodologies in engineering education: A case study in mathematics. Applied Sciences, 13(10), 6039. https://doi.org/10.3390/app13106039 · DOI
  18. Sandu, N., & Gide, E. (2019, September). Adoption of AI- Chatbots to enhance student learning experience in higher education in India. In 2019 18th International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1-5). IEEE. htpps://doi.org/10.1109/ ITHET46829.2019.8937382 · DOI
  19. Santandreu-Calonge, D., Medina-Aguerrebere, P., Hultberg, P., & Shah, M. A. (2023). Can ChatGPT improve communication in hospitals?. Profesional de la información, 32(2). https://doi.org/10.3145/epi.2023.mar. · DOI
  20. Shah, M., & Calonge, D. S. (2023). Refugees' experiences with online higher education: Impact and implications through the pandemic. Journal of Applied Learning and Teaching, 6(1), 209-221. https://doi.org/10.37074/jalt.2023.6.1.21 · DOI
  21. Shehata, S., Calonge, D. S., Purnell, P., & Thompson, M. (2023, July). Enhancing video-based learning using knowledge tracing: Personalizing students' learning experience with ORBITS. In Proceedings of the 18th workshop on innovative use of NLP for Building Educational Applications (BEA 2023) (pp. 100-107). https://doi.org/10.18653/v1/2023.bea-1. · DOI
  22. Singh Gill, S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., Fuller, S., Singh, M., Arora, P., Parlikad, A. K., Stankovski, V., Abraham, A., Ghosh, S. K., Lutfiyya, H., Kanhere, S. S., Bahsoon, R., Rana, O., Dustdar, S., Sakellariou, R., … Buyya, R. (2023). Transformative effects of ChatGPT on modern education: Emerging era of AI chatbots. arXiv e-prints, arXiv-2306. https://doi.org/10.1016/j.iotcps.2023.06.002 · DOI
  23. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  24. Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), em2286. https://doi. org/10.29333/ejmste/13272 · DOI
  25. Yang, S., & Evans, C. (2019, November). Opportunities and challenges in using AI chatbots in higher education. In Proceedings of the 2019 3rd international conference on education and e-learning (pp. 79-83). https://doi. org/10.1145/3371647.3371659 · DOI
  26. Zhang, S. J., Florin, S., Lee, A. N., Niknafs, E., Marginean, A., Wang, A., ... & Drori, I. (2023). Exploring the MIT mathematics and EECS curriculum using Large Language Models. arXiv preprint arXiv:2306.08997. · arXiv
  27. David Santandreu Calonge, Linda Smail and Firuz Kamalov.. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Cited by

Sorted by citations

Authors
Calonge, D. S., Smail, L., & Kamalov, F.
Affiliations
Canadian University of Dubai (AE)
ORCID
Venue
Journal of Applied Learning & Teaching
Publisher
Kaplan Business School, Australia
Published
2023-08-23
Volume
6
Issue
2
ISSN
2591-801X (ISSN-L)
DOI
10.37074/jalt.2023.6.2.22
OpenAlex
W4386053840
Cited by
50
References
21
Citations by year
2026: 9 · 2025: 20 · 2024: 17 · 2023: 4
FWCI
9.05
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
Journal of Applied Learning & Teaching (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
AI chatbots, large language models, ChatGPT, GPT-4, Google Bard, llamas, calculus teaching, statistics education, mathematics education, chatbot comparison, higher education, student learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review