CoChat
Sign Up
PreprintCited by 24Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach

Open access
Jurenka, Irina, Kunesch, Markus, McKee, Kevin R., Gillick, Daniel, Zhu, Shaojian, Wiltberger, Sara, Phal, Shubham Milind, Hermann, Katherine +66 more·2024·arXiv
DOI 10.48550/arxiv.2407.12687arXiv 2407.12687
Open this paper in CoChat — free accountOpen PDF

Abstract

A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.12687

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LearnLM alleviates the issue of fine-tuning by integrating pedagogical instruction following tuning, but still heavily relies on synthetic data and a costly co-training process.

    “LearnLM (Jurenka et al., 2024; LearnLM Team, 2024) alleviates this issue by integrating pedagogical instruction following tuning, but still heavily relies on synthetic data and a costly co-training process where LearnLM is trained alongside a generic LLM.”

    From Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure (2025) · cites this paper as Jurenka et al., 2024

    The Additional information section (page unavailable) says LearnLM-Tutor was further supervised-fine-tuned from a post-trained Gemini checkpoint using a mixture of human-authored and synthetic pedagogical data. Section 10 (page unavailable) reports improved pedagogical performance from that fine-tuning. The paper does not establish that the model relies heavily on synthetic data or undergoes costly co-training alongside a generic LLM; it describes further fine-tuning of an existing checkpoint instead. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Two lines of conceptual and empirical groundwork enabled this RCT: a generative AI model specifically fine-tuned for pedagogy, and an educational platform deeply rooted in learning science.

    “Two lines of conceptual and empirical groundwork enabled this RCT: first, a generative AI model specifically fine-tuned for pedagogy [16-18], and second, an educational platform deeply rooted in learning science [19].”

    From AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms (2025) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 176

  1. Jeremy Roschelle, Elizabeth A. McLaughlin, and Kenneth R. Koedinger. Beyond benchmarks: Responsible AI in education needs learning sciences. Communications of the ACM, 68(12): 37-39, 2025.
  2. Ethan Mollick and Lilach Mollick. Assigning AI: Seven approaches for students, with prompts. arXiv preprint arXiv:2306.10052, 2023. · arXiv
  3. Eric A Hanushek and Ludger Woessmann. Education and economic growth. Economics of education, 60(67):1, 2010.
  4. Cecilia Ka Yuk Chan and Katherine KW Lee. The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and millennial generation teachers? Smart Learning Environments, 10(1):60, 2023.
  5. Stefan Bauschard and Sabba Quidwai. From insight to implementation: How to create your AI school guidance. SSRN, 2024.
  6. Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
  7. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  8. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. · arXiv
  9. Wayne Holmes and Kaśka Porayska-Pomsta. The Ethics of Artificial Intelligence in education: Practices, challenges, and debates. Taylor & Francis, 2022.
  10. Andy Nguyen, Ha Ngan Ngo, Yvonne Hong, Belle Dang, and Bich-Phuong Thi Nguyen. Ethical principles for artificial intelligence in education. Education and Information Technologies, 28 (4):4221-4241, 2023.
  11. René F Kizilcec. To advance ai use in education, focus on understanding educators. International Journal of Artificial Intelligence in Education, 34(1):12-19, 2024.
  12. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214-229, 2022.
  13. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. · arXiv
  14. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. · arXiv
  15. Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. The ethics of advanced AI assistants. arXiv preprint arXiv:2404.16244, 2024. · arXiv
  16. Kenneth R Koedinger, Julie L Booth, and David Klahr. Instructional complexity and the science to constrain it. Science, 342(6161):935-937, 2013.
  17. Sherry R Arnstein. A ladder of citizen participation. Journal of the American Institute of planners, 35(4):216-224, 1969.
  18. Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. Power to the people? Opportunities and challenges for participatory AI. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1-8, 2022.
  19. Niels Pinkwart. Another 25 years of AIED? Challenges and opportunities for intelligent educational technologies of the future. International journal of artificial intelligence in education, 26:771-783, 2016.
  20. Henry Sanoff. Community participation methods in design and planning. John Wiley & Sons, 1999.
  21. Holtzblatt Karen and Jones Sandra. Contextual inquiry: A participatory technique for system design. In Participatory design, pages 177-210. CRC Press, 2017.
  22. Chadia Abras, Diane Maloney-Krichmar, Jenny Preece, et al. User-centered design. Bainbridge, W. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications, 37(4): 445-456, 2004.
  23. Junlei Li and David Klahr. Cognitive research and elementary science instruction: From the laboratory, to the classroom, and back. Journal of Science Education and Technology, 14(2), 2005.
  24. David Klahr. What do we mean? On the importance of not abandoning scientific rigor when talking about science education. Proceedings of the National Academy of Sciences, 110 (supplement_3):14075-14080, 2013.
  25. Audrey Watters. Teaching machines: The history of personalized learning. MIT Press, 2023.
  26. Edward Fry. Teaching machine dichotomy: Skinner vs. Pressey. Psychological Reports, 6(1): 11-14, 1960.
  27. Jaime R Carbonell. AI in CAI: An artificial-intelligence approach to computer-assisted instruc- tion. IEEE transactions on man-machine systems, 11(4):190-202, 1970.
  28. John R Anderson, C Franklin Boyle, and Brian J Reiser. Intelligent tutoring systems. Science, 228(4698):456-462, 1985.
  29. Vincent Aleven, Bruce M McLaren, and Jonathan Sewall. Scaling up programming by demon- stration for intelligent tutoring systems development: An open-access web site for middle school mathematics learning. IEEE transactions on learning technologies, 2(2):64-78, 2009.
  30. John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167-207, 1995.
  31. A Mitrovic. Learning SQL with a computerised tutor. In 29th ACM SIGCSE Technical Symposium. Atlanta, 1998.
  32. Bruce M McLaren, Sung-Joo Lim, France Gagnon, David Yaron, and Kenneth R Koedinger. Studying the effects of personalized language and worked examples in the context of a web- based intelligent tutor. In Intelligent Tutoring Systems: 8th International Conference, ITS 2006, Jhongli, Taiwan, June 26-30, 2006. Proceedings 8, pages 318-328. Springer, 2006.
  33. Silvia Schiaffino, Patricio Garcia, and Analia Amandi. eTeacher: Providing personalized assistance to e-learning students. Computers & Education, 51(4):1744-1754, 2008.
  34. Aytürk Keleş, Rahim Ocak, Ali Keleş, and Aslan Gülcü. ZOSMAT: Web-based intelligent tutoring system for teaching-learning process. Expert Systems with Applications, 36(2):1229-1239, 2009.
  35. Bruce Cheung, Lucas Hui, J Zhang, and Siu-Ming Yiu. SmartTutor: An intelligent tutoring system in web-based adult education. Journal of Systems and Software, 68(1):11-25, 2003.
  36. Arthur C Graesser, Katja Wiemer-Hastings, Peter Wiemer-Hastings, Roger Kreuz, Tutoring Re- search Group, et al. AutoTutor: A simulation of a human tutor. Cognitive Systems Research, 1 (1):35-51, 1999.
  37. Erica Melis and Jörg Siekmann. ActiveMath: An intelligent tutoring system for mathematics. In International Conference on Artificial Intelligence and Soft Computing, pages 91-101. Springer, 2004.
  38. Arthur C Graesser, Kurt VanLehn, Carolyn P Rosé, Pamela W Jordan, and Derek Harter. Intelligent tutoring systems with conversational dialogue. AI magazine, 22(4):39-39, 2001.
  39. Benjamin Clément, Hélène Sauzéon, Didier Roy, and Pierre-Yves Oudeyer. Improved per- formances and motivation in intelligent tutoring systems: Combining machine learning and learner choice. arXiv preprint arXiv:2402.01669, 2024. · arXiv
  40. Adolphe Maxime, Marion Pech, Masataka Sawayama, Denis Maurel, Alexandra Delmas, Pierre- Yves Oudeyer, and Hélène Sauzeon. Exploring the potential of artificial intelligence in individ- ualized cognitive training: A systematic review. PsyArXiv, 2023. doi: 10.31234/osf.io/2wg59. URL osf.io/preprints/psyarxiv/2wg59. · DOI
  41. Cécile Mazon, Benjamin Clément, Didier Roy, Pierre-Yves Oudeyer, and Hélène Sauzéon. Pilot study of an intervention based on an intelligent tutoring system (ITS) for instructing mathematical skills of students with ASD and/or ID. Education and Information Technologies, 28(8):9325-9354, 2023.
  42. John F Pane, Daniel F McCaffrey, Mary Ellen Slaughter, Jennifer L Steele, and Gina S Ikemoto. An experiment to evaluate the efficacy of cognitive tutor geometry. Journal of Research on Educational Effectiveness, 3(3):254-281, 2010.
  43. Kurt VanLehn. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational psychologist, 46(4):197-221, 2011.
  44. Benjamin D Nye, Arthur C Graesser, and Xiangen Hu. AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4):427-469, 2014.
  45. Vincent Aleven, Bruce M Mclaren, Jonathan Sewall, and Kenneth R Koedinger. A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2):105-154, 2009.
  46. Kurt VanLehn, Pamela W Jordan, Carolyn P Rosé, Dumisizwe Bhembe, Michael Böttner, Andy Gaydos, Maxim Makatchev, Umarani Pappuswamy, Michael Ringenberg, Antonio Roque, et al. The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In International conference on intelligent tutoring systems, pages 158-167. Springer, 2002.
  47. Shu-Hsien Liao. Expert system methodologies and applications-a decade review from 1995 to 2004. Expert systems with applications, 28(1):93-103, 2005.
  48. Hyacinth S Nwana. Intelligent tutoring systems: An overview. Artificial Intelligence Review, 4 (4):251-277, 1990.
  49. S Mcroy and R Freedman. What is an intelligent tutoring system. Intelligence, 11(3):15-16, 2000.
  50. Roger Nkambou, Riichiro Mizoguchi, and Jacqueline Bourdeau. Advances in intelligent tutoring systems, volume 308. Springer Science & Business Media, 2010.
  51. Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020. · arXiv
  52. Irina Higgins, Antonia Creswell, and Sebastien Racaniere. Pay attention to what you need: Do structural priors still matter in the age of billion parameter models?, 2021. URL https: //neurips.cc/virtual/2021/tutorial/21891.
  53. Huw C Davies, Rebecca Eynon, and Cory Salveson. The mobilisation of AI in education: A Bourdieusean field analysis. Sociology, 55(3):539-560, 2021.
  54. Anthony Seldon, Oladimeji Abidoye, and Timothy Metcalf. The Fourth Education Revolution Reconsidered: Will Artificial Intelligence Enrich Or Diminish Humanity? Legend Press Ltd, 2020.
  55. Olaf Zawacki-Richter, Victoria I Marín, Melissa Bond, and Franziska Gouverneur. Systematic review of research on artificial intelligence applications in higher education-where are the educators? International Journal of Educational Technology in Higher Education, 16(1):1-27, 2019.
  56. James A Kulik and John D Fletcher. Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of educational research, 86(1):42-78, 2016.
  57. Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler. Are we there yet? -a systematic literature review on chatbots in education. Frontiers in artificial intelligence, 4:654924, 2021.
  58. Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2:100033, 2021.
  59. Arif Iqbal, Reinhard Oppermann, Ashok Patel, and Kinshuk. A classification of evaluation methods for intelligent tutoring systems. Software-Ergonomie'99: Design von Informationswelten, pages 169-181, 1999.
  60. Julika Siemer and Marios C Angelides. A comprehensive method for the evaluation of complete intelligent tutoring systems. Decision support systems, 22(1):85-102, 1998.
  61. Mary A Mark, Jim E Greer, et al. Evaluation methodologies for intelligent tutoring systems. Journal of Artificial Intelligence in Education, 4:129-129, 1993.
  62. Martha C Polson and J Jeffrey Richardson. Foundations of intelligent tutoring systems. Psychology Press, 2013.
  63. Tanya Nazaretsky, Mutlu Cukurova, and Giora Alexandron. An instrument for measuring teachers' trust in AI-based educational technology. In LAK22: 12th international learning analytics and knowledge conference, pages 56-66, 2022.
  64. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  65. Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019.
  66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  67. Justin Vasselli, Christopher Vasselli, Adam Nohejl, and Taro Watanabe. NAISTeacher: A prompt and rerank approach to generating teacher utterances in educational dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 772-784, 2023.
  68. Rania Abdelghani, Hélène Sauzéon, and Pierre-Yves Oudeyer. Generative AI in the classroom: Can students remain active learners? arXiv preprint arXiv:2310.03192, 2023. · arXiv
  69. Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. Evaluating language models for mathematics through interactions. arXiv preprint arXiv:2306.01694, 2023. · arXiv
  70. Changyoon Lee, Junho Myung, Jieun Han, Jiho Jin, and Alice Oh. Learning from teaching assistants to program with subgoals: Exploring the potential for AI teaching assistants. arXiv preprint arXiv:2309.10419, 2023. · arXiv
  71. Yu Li, Shang Qu, Jili Shen, Shangchao Min, and Zhou Yu. Curriculum-driven Edubot: A framework for developing language learning chatbots through synthesizing conversational data. arXiv preprint arXiv:2309.16804, 2023. · arXiv
  72. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. arXiv preprint arXiv:2305.14536, 2023. · arXiv
  73. Rose E Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. Step- by-step remediation of students' mathematical mistakes. arXiv preprint arXiv:2310.10648, 2023. · arXiv
  74. Blake Castleman and Mehmet Kerem Turkcan. Examining the influence of varied lev- els of domain knowledge base inclusion in GPT-based intelligent tutors. arXiv preprint arXiv:2309.12367, 2023. · arXiv
  75. Anaïs Tack and Chris Piech. The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. arXiv preprint arXiv:2205.07540, 2022. · arXiv
  76. Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. The BEA 2023 shared task on generating AI teacher responses in educational dialogues. arXiv preprint arXiv:2306.06941, 2023. · arXiv
  77. Yann Hicke, Abhishek Masand, Wentao Guo, and Tushaar Gangavarapu. Assessing the ef- ficacy of large language models in generating accurate teacher responses. arXiv preprint arXiv:2307.04274, 2023. · arXiv
  78. Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan, Tong Wang, Pauline Lucas, Hélène Sauzéon, and Pierre-Yves Oudeyer. GPT-3-driven pedagogical agents for training children's curious question-asking skills. International Journal of Artificial Intelligence in Education, pages 1-36, 2023.
  79. Harsh Kumar, David M Rothschild, Daniel G Goldstein, and Jake M Hofman. Math education with large language models: Peril or promise? Available at SSRN 4641653, 2023.
  80. Erfan Al-Hossami, Razvan Bunescu, Justin Smith, and Ryan Teehan. Can language models employ the Socratic method? Experiments with code debugging. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 53-59, 2024.
  81. Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, et al. Language models as science tutors. arXiv preprint arXiv:2402.11111, 2024. · arXiv
  82. Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J Malan. Teaching CS50 with AI: Leveraging generative artificial intelligence in computer science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 750-756, 2024.
  83. Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. Opportunities and challenges in neural dialog tutoring. arXiv preprint arXiv:2301.09919, 2023. · arXiv
  84. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. · arXiv
  85. Paul Denny, Sumit Gulwani, Neil T Heffernan, Tanja Käser, Steven Moore, Anna N Rafferty, and Adish Singla. Generative AI for education (GAIED): Advances, opportunities, and challenges. arXiv preprint arXiv:2402.01580, 2024. · arXiv
  86. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 technical report. arXiv preprint arXiv:2305.10403, 2023. · arXiv
  87. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
  88. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-589, 2021.
  89. Katherine Stasaski, Kimberly Kao, and Marti A Hearst. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-64, 2020.
  90. Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez- Paredes, Bill Byrne, and Paula Buttery. The teacher-student chatroom corpus. arXiv preprint arXiv:2011.07109, 2020. · arXiv
  91. Abhijit Suresh, Jennifer Jacobs, Margaret Perkoff, James H Martin, and Tamara Sumner. Fine-tuning transformers with additional context to classify discursive moves in mathematics classrooms. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications, 2022.
  92. Dorottya Demszky and Heather Hill. The NCTE transcripts: A dataset of elementary math classroom transcripts. arXiv preprint arXiv:2211.11772, 2022. · arXiv
  93. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  94. Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. Dialog inpainting: Turning documents into dialogs. In International conference on machine learning, pages 4558-4586. PMLR, 2022.
  95. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. · arXiv
  96. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurIPS, 2021.
  97. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. · arXiv
  98. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  99. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- ToxicityPrompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020. · arXiv
  100. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel R Bowman. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021. · arXiv
  101. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
  102. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675, 2019. · arXiv
  103. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
  104. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. Dialogue response ranking training with large-scale human feedback data. arXiv preprint arXiv:2009.06978, 2020. · arXiv
  105. Judith D Wilson. A Socratic approach to helping novice programmers debug programs. ACM SIGCSE Bulletin, 19(1):179-182, 1987.
  106. Alexis Baladón, Ignacio Sastre, Luis Chiruzzo, and Aiala Rosá. RETUYT-InCo at BEA 2023 shared task: Tuning open-source LLMs for generating teacher responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 756-765, 2023.
  107. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
  108. Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746, 2022. · arXiv
  109. Hua Shen and Tongshuang Wu. Parachute: Evaluating interactive human-LM co-writing systems. arXiv preprint arXiv:2303.06333, 2023. · arXiv
  110. Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. Measuring conversational uptake: A case study on student-teacher interactions. arXiv preprint arXiv:2106.03873, 2021. · arXiv
  111. Michelene TH Chi and Ruth Wylie. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational psychologist, 49(4):219-243, 2014.
  112. Kurt VanLehn, Stephanie Siler, Charles Murray, and William B Baggett. What makes a tutorial event effective? In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society. Citeseer, 1999.
  113. Yana Weinstein, Megan Sumeracki, and Oliver Caviglioli. Understanding How We Learn: A Visual Guide. Routledge, 2019.
  114. Richard E. Mayer. Multimedia learning. Cambridge University Press, 2022.
  115. Stanislas Dehaene. How we learn why brains learn better than any machine ... for now. Penguin Books, 2021.
  116. Emily R Lai. Metacognition: A literature review. Always learning: Pearson research report, 24: 1-40, 2011.
  117. John M Keller. Development and use of the ARCS model of instructional design. Journal of instructional development, 10(3):2-10, 1987.
  118. Erika A Patall, Harris Cooper, and Jorgianne Civey Robinson. The effects of choice on intrinsic motivation and related outcomes: A meta-analysis of research findings. Psychological bulletin, 134(2):270, 2008. [150] 2017. URL https://tech.ed.gov/files/2017/01/NETP17.pdf.
  119. Peter C Brown, Henry L Roediger III, and Mark A McDaniel. Make it stick: The science of successful learning. Harvard University Press, 2014.
  120. Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. Mea- suring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39):19251-19257, 2019.
  121. William Agnew, A Stevie Bergman, Jennifer Chien, Mark Díaz, Seliem El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R McKee. The illusion of artificial inclusion. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024.
  122. Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737-744. IEEE, 2020.
  123. George EP Box. Science and statistics. Journal of the American Statistical Association, 71(356): 791-799, 1976.
  124. Kevin R McKee. Human participants in AI research: Ethics and transparency in practice. arXiv preprint arXiv:2311.01254, 2023. · arXiv
  125. DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
  126. Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. CritiqueLLM: Scaling LLM-as-critic for effective and explainable evaluation of large language model generation. arXiv preprint arXiv:2311.18702, 2023. · arXiv
  127. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685, 2023. · arXiv
  128. Michael Quinn Patton. Qualitative research & evaluation methods: Integrating theory and practice. Sage Publications, 2014.
  129. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  130. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. · arXiv
  131. Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. GeomVerse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023. · arXiv
  132. Kristen DiCerbo. Implementation of AI tools in education at scale, 2023. URL https:// neurips.cc/virtual/2023/81332.
  133. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023. · arXiv
  134. Deborah G Johnson and Jameson M Wetmore. Technology and society: Building our sociotechnical future. MIT press, 2021.
  135. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023. · arXiv
  136. Nicholas Epley, Adam Waytz, and John T Cacioppo. On seeing human: A three-factor theory of anthropomorphism. Psychological review, 114(4):864, 2007.
  137. Katja Wagner, Frederic Nimmermann, and Hanna Schramm-Klein. Is it human? The role of anthropomorphism as a driver for the successful acceptance of digital voice assistants. In Hawaii International Conference on System Sciences, 2019.
  138. Abbe Don, Susan Brennan, Brenda Laurel, and Ben Shneiderman. Anthropomorphism: From ELIZA to Terminator 2. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 67-70, 1992.
  139. Arleen Salles, Kathinka Evers, and Michele Farisco. Anthropomorphism in AI. AJOB neuroscience, 11(2):88-95, 2020.
  140. Gavin Abercrombie, Amanda Cercas Curry, Tanvi Dinkar, and Zeerak Talat. Mirages: On anthropomorphism in dialogue systems. arXiv preprint arXiv:2305.09800, 2023. · arXiv
  141. Alexandra D Kaplan, Theresa T Kessler, J Christopher Brill, and Peter A Hancock. Trust in artificial intelligence: Meta-analytic findings. Human factors, 65(2):337-359, 2023.
  142. Markus Blut, Cheng Wang, Nancy V Wünderlich, and Christian Brock. Understanding anthro- pomorphism in service provision: A meta-analysis of physical robots, chatbots, and other AI. Journal of the Academy of Marketing Science, 49:632-658, 2021.
  143. Jenny Van Doorn, Martin Mende, Stephanie M Noble, John Hulland, Amy L Ostrom, Dhruv Grewal, and J Andrew Petersen. Domo arigato Mr. Roboto: Emergence of automated social presence in organizational frontlines and customers' service experiences. Journal of service research, 20(1):43-58, 2017.
  144. Ben Sheehan, Hyun Seung Jin, and Udo Gottlieb. Customer service chatbots: Anthropomor- phism and adoption. Journal of Business Research, 115:14-24, 2020.
  145. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220-229, 2019.
  146. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. · arXiv
  147. Norman Jouppi, Doe Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63:67-78, 06 2020. doi: 10.1145/3360307. · DOI
  148. Norman P. Jouppi, George Kurian, Sheng Li, Peter C. Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiaoping Zhou, Zongwei Zhou, and David A. Patterson. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023. URL https://api.semanticscholar.org/ CorpusID:257921908.
  149. Virginia Braun and Victoria Clarke. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77-101, 2006.
  150. Greg Thompson and Ian Cook. The logic of data-sense: Thinking through learning personali- sation. In The education assemblage, pages 81-95. Routledge, 2020.
  151. Lev S Vygotsky. Mind in society: The development of higher psychological processes, volume 86. Harvard university press, 1978.
  152. Vincent Aleven, Ido Roll, Bruce M McLaren, and Kenneth R Koedinger. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26:205-223, 2016.
  153. Sidney D'Mello and Art Graesser. Dynamics of affective states during complex learning. Learning and Instruction, 22(2):145-157, 2012.
  154. Ran Zhao, Alexandros Papangelis, and Justine Cassell. Towards a dyadic computational model of rapport management for human-virtual agent interaction. In Intelligent Virtual Agents: 14th International Conference, IVA 2014, Boston, MA, USA, August 27-29, 2014. Proceedings 14, pages 514-527. Springer, 2014.
  155. Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1): 973-1018, 2023.
  156. Carole R Beal, Ivon M Arroyo, Paul R Cohen, and Beverly P Woolf. Evaluation of AnimalWatch: An intelligent tutoring system for arithmetic and fractions. Journal of Interactive Online Learning, 9(1), 2010.
  157. Janice D Gobert, Raha Moussavi, Haiying Li, Michael Sao Pedro, and Rachel Dickler. Real-time scaffolding of students' online data interpretation during inquiry with Inq-ITS using educational data mining. Cyber-physical laboratories in engineering and science education, pages 191-217, 2018.
  158. Michael Mendicino, Leena Razzaq, and Neil T Heffernan. A comparison of traditional homework to computer-supported homework. Journal of Research on Technology in Education, 41(3): 331-359, 2009.
  159. Kurt VanLehn, Collin Lynch, Kay Schulze, Joel A Shapiro, Robert Shelby, Linwood Taylor, Don Treacy, Anders Weinstein, and Mary Wintersgill. The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence in Education, 15(3):147-204, 2005.
  160. Wenting Ma, Olusola O Adesope, John C Nesbit, and Qing Liu. Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of educational psychology, 106(4):901, 2014.
  161. Arthur C Graesser, Natalie K Person, and Joseph P Magliano. Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied cognitive psychology, 9(6):495-522, 1995.
  162. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023. · arXiv
  163. George Loewenstein. The psychology of curiosity: A review and reinterpretation. Psychological bulletin, 116(1):75, 1994.
  164. Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794, 2023. · arXiv
  165. Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024. · arXiv
  166. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. · arXiv
  167. Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological bulletin, 52(4):281, 1955.
  168. Lee Anna Clark and David Watson. Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 1995.
  169. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. FaithDial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473-1490, 2022.
  170. Inigo Casanueva, Ivan Vulić, Georgios P Spithourakis, and Paweł Budzianowski. NLU++: A multi-label, slot-rich, generalisable dataset for natural language understanding in task-oriented dialogue. arXiv preprint arXiv:2204.13021, 2022. · arXiv
  171. Eyal Peer, David Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer. Data quality of platforms and panels for online behavioral research. Behavior research methods, pages 1-20, 2021.
  172. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. · arXiv
  173. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  174. Curran As- sociates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  175. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022. URL https://api.semanticscholar.org/CorpusID: 246411621. · arXiv
  176. Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020. · arXiv

Cited by

Sorted by citations

Authors
Jurenka, I., Kunesch, M., McKee, K. R., Gillick, D., Zhu, S., Wiltberger, S., Phal, S. M., Hermann, K., Kasenberg, D., Bhoopchand, A., Anand, A., Pîslar, M., Chan, S., Wang, L., She, J., Mahmoudieh, P., Rysbek, A., Ko, W.-J., Huber, A., Wiltshire, B., Elidan, G., Rabin, R., Rubinovitz, J., Pitaru, A., McAllister, M., Wilkowski, J., Choi, D., Engelberg, R., Hackmon, L., Levin, A., Griffin, R., Sears, M., Bar, F., Mesar, M., Jabbour, M., Chaudhry, A., Cohan, J., Thiagarajan, S., Levine, N., Brown, B., Gorur, D., Grant, S., Hashimshoni, R., Weidinger, L., Hu, J., Chen, D., Dolecki, K., Akbulut, C., Bileschi, M., Culp, L., Dong, W.-X., Marchal, N., Van Deman, K., Misra, H. B., Duah, M., Ambar, M., Caciularu, A., Lefdal, S., Summerfield, C., An, J., Kamienny, P.-A., Mohdi, A., Strinopoulous, T., Hale, A., Anderson, W., Cobo, L. C., Efron, N., Ananda, M., Mohamed, S., Heymans, M., Ghahramani, Z., Matias, Y., Gomes, B., & Ibrahim, L.
Affiliations
  • Google (United States) (US)
  • Arizona State University (US)
  • Lund University (SE)
  • University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.12687
arXiv
2407.12687
OpenAlex
W4402345999
Cited by
24
References
0
Citations by year
2026: 2 · 2025: 14 · 2024: 8
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Arizona State University
  • DeepMind
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
generative AI in education, educational assessment, teaching evaluation, AI tutors, Gemini, prompt tuning, educational benchmarks, responsible AI development, educational equity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review