CoChat
Sign Up
PreprintCited by 96Intelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in EducationSpeech and dialogue systems

The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues

Open access
Tack, Anaïs, Piech, Chris·2022·arXiv
DOI 10.48550/arxiv.2205.07540arXiv 2205.07540
Open this paper in CoChat — free accountOpen PDF

Abstract

How can we test whether state-of-the-art generative models, such as Blender and GPT-3, are good AI teachers, capable of replying to a student in an educational dialogue? Designing an AI teacher test is challenging: although evaluation methods are much-needed, there is no off-the-shelf solution to measuring pedagogical ability. This paper reports on a first attempt at an AI teacher test. We built a solution around the insight that you can run conversational agents in parallel to human teachers in real-world dialogues, simulate how different agents would respond to a student, and compare these counterpart responses in terms of three abilities: speak like a teacher, understand a student, help a student. Our method builds on the reliability of comparative judgments in education and uses a probabilistic model and Bayesian sampling to infer estimates of pedagogical ability. We find that, even though conversational agents (Blender in particular) perform well on conversational uptake, they are quantifiably worse than real teachers on several pedagogical dimensions, especially with regard to helpfulness (Blender: Δ ability = -0.75; GPT-3: Δ ability = -0.93).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2205.07540

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current LLMs lack crucial pedagogical skills.

    “However, current LLMs are trained primarily to be helpful assistants and lack crucial pedagogical skills.”

    From Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure (2025) · cites this paper as Tack and Piech, 2022

    The introduction (p. 1) says Blender and GPT-3 lag behind human teachers in pedagogical ability, and Table 3 (p. 5) reports lower ratings for those models on speaking like a teacher, understanding the student, and helping the student. This supports a pedagogical-skills gap, but the study tests two models; it does not establish that current LLMs generally lack those skills. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • It is becoming increasingly difficult to distinguish whether a text is machine- or human-generated, presenting an additional major challenge to teachers and educators.

    “It is becoming increasingly difficult to distinguish whether a text is machine- or human-generated, presenting an additional major challenge to teachers and educators [58, 59, 60, 61].”

    From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [58]

    Section 2 (p. 2) describes comparing human and AI teacher replies for pedagogical ability, not determining whether a text was written by a human or a machine. Searches for difficulty distinguishing human- from machine-generated text and for AI-text detection as a challenge for educators found no passage addressing the claim. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 8

  1. [4][4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. -
  2. [5][5] A. Caines, H. Yannakoudakis, H. Edmondson, H. Allen, P. P´ erez-Paredes, B. Byrne, and P. Buttery. The teacher-student chatroom corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning , pages 10-20, Gothenburg, Sweden, Nov. 2020. LiU Electronic Press. -
  3. [10][10] S. Heldsinger and S. Humphry. Using the Method of Pairwise Comparison to Obtain Reliable Teacher Assessments. Australian Educational Researcher , 37(2): 1-19, Aug. 2010. ISSN 0311-6999. -
  4. [11][11] M. D. Homan and A. Gelman. The No-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. The Journal of Machine Learning Research , 15 (1):1593-1623, Jan. 2014. ISSN 1532-4435. -
  5. [12][12] M. Lesterhuis, S. Verhavert, L. Coertjens, V. Donche, and S. De Maeyer. Comparative judgement as a promising alternative to score competences. In E. Cano, G. Ion, and J. Keengwe, editors, Innovative Practices for Higher Education Assessment and Measurement: , Advances in Higher Education and Professional Development. IGI Global, 2017. ISBN 978-1-5225-0531-0 9781-5225-0532-7. doi: 10.4018/978-1-5225-0531-0. - · DOI
  6. [13][13] A. Miller, W. Feng, D. Batra, A. Bordes, A. Fisch, J. Lu, D. Parikh, and J. Weston. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 79-84, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/v1/ D17-2014. - · DOI
  7. [15][15] K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. In Advances in Neural Information Processing Systems 34 Pre-Proceedings (NeurIPS 2021) , pages 1-35, Nov. 2021. -
  8. [17][17] S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, K. Shuster, E. M. Smith, Y.L. Boureau, and J. Weston. Recipes for building an open-domain chatbot. 2020. doi: 10.48550/ARXIV. 2004.13637. - · DOI

Cited by

Sorted by citations

Related

Authors
Tack, A., & Piech, C.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2205.07540
arXiv
2205.07540
OpenAlex
W4280504808
Cited by
36
References
0
Citations by year
2025: 8 · 2024: 19 · 2023: 9
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
AI teachers, pedagogical abilities, generative language models, Blender, GPT-3, chatbots, educational dialogue, Bayesian sampling, comparative judgment, helpfulness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review