CoChat
Sign Up
PreprintCited by 1Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

Evaluating Gemini in an arena for learning

Open access
LearnLM Team, Modi, Abhinit, Veerubhotla, Aditya Srikanth, Rysbek, Aliya, Huber, Andrea, Anand, Ankit, Bhoopchand, Avishkar, Wiltshire, Brett +29 more·2025·arXiv
DOI 10.48550/arxiv.2505.24477arXiv 2505.24477
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence (AI) is poised to transform education, but the research community lacks a robust, general benchmark to evaluate AI models for learning. To assess state-of-the-art support for educational use cases, we ran an "arena for learning" where educators and pedagogy experts conduct blind, head-to-head, multi-turn comparisons of leading AI models. In particular, $N = 189$ educators drew from their experience to role-play realistic learning use cases, interacting with two models sequentially, after which $N = 206$ experts judged which model better supported the user's learning goals. The arena evaluated a slate of state-of-the-art models: Gemini 2.5 Pro, Claude 3.7 Sonnet, GPT-4o, and OpenAI o3. Excluding ties, experts preferred Gemini 2.5 Pro in 73.2% of these match-ups -- ranking it first overall in the arena. Gemini 2.5 Pro also demonstrated markedly higher performance across key principles of good pedagogy. Altogether, these results position Gemini 2.5 Pro as a leading model for learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.24477

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Two lines of conceptual and empirical groundwork enabled this RCT: a generative AI model specifically fine-tuned for pedagogy, and an educational platform deeply rooted in learning science.

    “Two lines of conceptual and empirical groundwork enabled this RCT: first, a generative AI model specifically fine-tuned for pedagogy [16-18], and second, an educational platform deeply rooted in learning science [19].”

    From AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms (2025) · cites this paper as [18]

    The Introduction (p. 1) says the experimental LearnLM model’s pedagogical capabilities were integrated into Gemini 2.5 Pro and Flash, supporting the claim that the model was adapted for learning. It does not say the model was specifically fine-tuned for pedagogy. The separate claim about an educational platform is attached to citation [19], not this source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. [3]Ali Bissoondath. Artificially intelligent? Children's and parents' views on generative AI in education. https://web.archive.org/web/20241119022121/https://www.inte rnetmatters.org/hub/press-release/ai-research-warns-schools-unprepare d-artificial-intelligence/ , 2024. Accessed 15 May, 2025. https://web.archive.org/web/20241119022121/https://www.internetmatters.org/hub/press-release/ai-research-warns-schools-unprepared-artificial-intelligence/
  2. [5]Salman Khan. Brave new words: How AI will revolutionize education (and why that's a good thing) . Penguin, 2024.
  3. [6]Ethan Mollick. Co-intelligence: Living and working with AI . Penguin, 2024.
  4. [9]Irina Jurenka, Markus Kunesch, Kevin R. McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, et al. Towards responsible development of generative AI for education: An evaluation-driven approach. arXiv preprint arXiv:2407.12687 , 2024. · arXiv
  5. [10]LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla, Aliya Rysbek, Andrea Huber, Brett Wiltshire, Brian Veprek, Daniel Gillick, Daniel Kasenberg, Derek Ahmed, et al. LearnLM: Improving Gemini for learning. arXiv preprint arXiv:2412.16429 , 2024. · arXiv
  6. [12]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020. · arXiv
  7. [13]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. · arXiv
  8. [14]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021. · arXiv
  9. [15]Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249 , 2025. · arXiv
  10. [16]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9556-9567, 2024.
  11. [17]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling , 2024.
  12. [18]Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. arXiv preprint arXiv:2412.03304 , 2024. · arXiv
  13. [19]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018. · arXiv
  14. [22]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, et al. Chatbot Arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning , 2024.
  15. [23]Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika , 39(3/4):324-345, 1952. doi: 10.1093/biomet/39. 3-4.324. · DOI
  16. [24]Louis Deslauriers, Logan S. McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences 116(39) , 2019. doi: 10.1073/pnas.1821936116. · DOI · Paper page
  17. [26]Sean Kang, Kathleen McDermott, and Henry Roediger. Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology , 19:528-558, 07 2007. doi: 10.1080/09541440601056620. · DOI · Paper page
  18. [27]Adena M. Klem and James P. Connell. Relationships matter: Linking teacher support to student engagement and achievement. Journal of School Health , 74(7), 2004. doi: 10.1111/j.1746-156 1.2004.tb08283.x. · DOI
  19. [28]Debora L. Roorda, Helma M. Y. Koomen, Jantine L. Spilt, and Frans J. Oort. The influence of affective teacher-student relationships on students' school engagement and achievement: A meta-analytic approach. Review of Educational Research , 81(4):493-529, 2011. doi: 10.3102/ 0034654311421793.
  20. [29]Valentin Emslander, Doris Holzberger, Sverre Berg Ofstad, Antoine Fischbach, and Ronny Scherer. Teacher-student relationships and student outcomes: A systematic second-order meta-analytic review. Psychological Bulletin , 2025. doi: 10.1037/bul0000461. · DOI
  21. [30]Erik A. Ruzek, Christopher A. Hafen, Joseph P. Allen, Anne Gregory, Amori Yee Mikami, and Robert C. Pianta. How teacher emotional support motivates students: The mediating roles of perceived peer relatedness, autonomy support, and competence. Learning and Instruction , 42: 95-103, 2016. doi: 10.1016/j.learninstruc.2016.01.004. · DOI
  22. [31]Heather A. Davis. Conceptualizing the role and influence of student-teacher relationships on children's social and cognitive development. Educational Psychologist , 38(4):207-234, 2003. doi: 10.1207/S15326985EP3804\_2. · DOI
  23. [32]Lynda D. Stone, Charles Underwood, and Jacqueline Hotchkiss. The relational habitus: Intersubjective processes in learning settings. Human Development , 55(2):65-91, 2012. doi: 10.1159/000337150. · DOI
  24. [33]Hoda Fakour and Moslem Imani. Socratic wisdom in the age of AI: A comparative study of ChatGPT and human tutors in enhancing critical thinking skills. Frontiers in Education , 10, 01 2025. doi: 10.3389/feduc.2025.1528603. · DOI
  25. [34]Gregory Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, and Gregorio Ponti. AI tutoring outperforms active learning. Research Square , 2024. doi: 10.21203/rs.3.rs-4243877/v1. · DOI · Paper page
  26. [37]Owen Henkel, Hannah Horne-Robinson, Nessie Kozhakhmetova, and Amanda Lee. Effective and scalable math support: Experimental evidence on the impact of an AI-math tutor in Ghana. In International Conference on Artificial Intelligence in Education , pages 373-381. Springer, 2024.
  27. [38]Harsh Kumar, David M. Rothschild, Daniel G. Goldstein, and Jake M. Hofman. Math education with large language models: Peril or promise? Available at SSRN 4641653 , 2023.
  28. [39]Jin Wang and Wenxiang Fan. The effect of ChatGPT on students' learning performance, learning perception, and higher-order thinking: Insights from a meta-analysis. Humanities and Social Sciences Communications , 12(1):1-21, 2025. doi: 10.1057/s41599-025-04787-y. · DOI · Paper page
  29. [40]Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Ozge Kabakcı, and Rei Mariman. Generative AI can harm learning. Available at SSRN , 4895486, 2024.
  30. [41]Matthias Lehmann, Philipp B. Cornelius, and Fabian J Sting. AI meets the classroom: When does ChatGPT harm learning? Available at SSRN 4941259 , 2024.
  31. [43]Keith Sawyer ed. The Cambridge handbook of the learning sciences, 2nd ed. Cambridge University Press, 2014.
  32. [44]Richard E. Mayer. Multimedia learning, 2nd ed. Cambridge University Press, 2009.
  33. [45]National Academies of Sciences, Engineering, and Medicine. How people learn II: Learners, contexts, and cultures . The National Academies Press, 2018.
  34. [46]Pierre-Yves Oudeyer, Jacqueline Gottlieb, and Manuel Lopes. Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. Progress in Brain Research , 2016. doi: 10.1016/bs.pbr.2016.05.005. · DOI
  35. [47]Yana Weinstein, Megan Sumeracki, and Oliver Caviglioli. Understanding how we learn: A visual guide . Routledge, 2018.
  36. [48]Michelene T. H. Chi and Ruth Wylie. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist , 2014. doi: 10.1080/00461520.2014.965823. · DOI · Paper page
  37. [49]Kevin R. McKee. Human participants in AI research: Ethics and transparency in practice. IEEE Transactions on Technology and Society , 2024. doi: 10.1109/TTS.2024.3446183. · DOI
  38. [50]Ido Dagan, Dan Roth, Fabio Zanzotto, and Mark Sammons. Recognizing textual entailment: Models and applications . Springer Nature, 2022.

Cited by

Sorted by citations

Authors
LearnLM Team, Modi, A., Veerubhotla, A. S., Rysbek, A., Huber, A., Anand, A., Bhoopchand, A., Wiltshire, B., Gillick, D., Kasenberg, D., Sgouritsa, E., Elidan, G., Liu, H., Winnemoeller, H., Jurenka, I., Cohan, J., She, J., Wilkowski, J., Alarakyia, K., McKee, K. R., Singh, K., Wang, L., Kunesch, M., Pîslar, M., Efron, N., Mahmoudieh, P., Kamienny, P.-A., Wiltberger, S., Mohamed, S., Agarwal, S., Phal, S. M., Lee, S. J., Strinopoulos, T., Ko, W.-J., Gold-Zamir, Y., Haramaty, Y., & Assael, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.24477
arXiv
2505.24477
OpenAlex
W4414857680
Cited by
0
References
1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
Gemini 2.5 Pro, teaching principles, Claude 3.7 Sonnet, GPT-4o, OpenAI o3

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review