Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction
Abstract
Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B) identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model's suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT's responses may lack novelty or insightful perspectives regarding teaching improvement.
“However, ChatGPT's responses may lack novelty or insightful perspectives regarding teaching improvement [210].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [210]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996 · cited 3× · see citing works
“Transcripts are derived from video recordings, which were scored by expert raters using two instruments at the time of the NCTE data collection: the Classroom Assessment Scoring System (CLASS) (Pianta et al., 2008) and Mathematical Quality Instruction (MQI) (Hill et al., 2008) instruments.”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 22
- Clement Adelman and Roy Walker. 2003. A guide to classroom observation. Routledge.
- L. Alrajhi, A. Alamri, F. D. Pereira, and A. I. Cristea. 2021. Urgency analysis of learners' comments: An automated intervention priority model for mooc. In International Conference on Intelligent Tutoring Sys- tems, pages 148-160.
- S. Aslan, N. Alyuz, C. Tanriover, S. E. Mete, E. Okur, S. K. D'Mello, and A. Arslan Esme. 2019. Investigat- ing the impact of a real-time. In multimodal student engagement analytics technology in authentic class- rooms, pages 1-12. of the 2019 CHI conference on human factors in computing systems.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Ad- vances in neural information processing systems, 30.
- Dorottya Demszky and Heather Hill. 2022. The NCTE Transcripts: A dataset of elementary math classroom transcripts. arXiv preprint arXiv:2211.11772. · arXiv
- Laura M Desimone and Katie Pak. 2017. Instructional coaching as high-quality professional development. Theory into practice, 56(1):3-12.
- P. J. Donnelly, N. Blanchard, A. M. Olney, S. Kelly, M. Nystrand, and S. K. D'Mello. 2017. Words mat- ter: Automatic detection of teacher questions in live classroom discourse using linguistics, acoustics and context. 218-227. Proceedings of the Seventh Inter- national Learning Analytics & Knowledge Confer- ence on -LAK '17.
- Pihel Hunt, Äli Leijen, and Marieke van der Schaaf. 2021. Automated feedback is nice and human pres- ence makes it better: Teachers' perceptions of feed- back by means of an e-portfolio enhanced with learn- ing analytics. Education Sciences, 11(6):278.
- E. Jensen, M. Dale, P. J. Donnelly, C. Stone, S. Kelly, A. Godley, and S. K. D'Mello. 2020. Toward au- tomated feedback on teacher discourse to enhance teacher learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38. · DOI · Paper page
- S. Kelly, A. M. Olney, P. Donnelly, M. Nystrand, and S. K. D'Mello. 2018. Automatically measuring ques- tion authenticity in real-world classrooms. Educa- tional Researcher, 47:7. · DOI · Paper page
- M. A. Kraft, D. Blazar, and D. Hogan. 2018. The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4):547-588. · DOI · Paper page
- Matthew A Kraft and David Blazar. 2018. Taking teacher coaching to scale: Can personalized training become standard practice? Education Next, 18(4):68- 75.
- Taja Kuzman, Igor Mozetic, and Nikola Ljubešic. 2023. Chatgpt: Beginning of an end of manual linguistic data annotation? use case of automatic genre identifi- cation. arXiv e-prints, pages arXiv-2303.
- Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852. · arXiv
- Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell- Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147. · arXiv
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted question-answering with human feedback.
- Yen-Ning Su, Chia-Cheng Hsu, Hsin-Chin Chen, Kuo- Kuang Huang, and Yueh-Min Huang. 2014. Develop- ing a sensor-based learning concentration detection system. Engineering Computations, 31(2):216-230.
- A. Suresh, J. Jacobs, V. Lai, C. Tan, W. Ward, J. H. Martin, and T. Sumner. 2021. Using transformers to provide teachers with personalized feedback on their classroom discourse: The talkmoves application. arxiv. Preprint.
- Earl E Thomas, David L Bell, Maureen Spelman, and Jennifer Briody. 2015. The growth of instructional coaching partner conversations in a prek-3rd grade teacher professional development experience. Jour- nal of Adult Education, 44(2):1-6.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv
- Ted Wragg. 2011. An introduction to classroom obser- vation (Classic edition). Routledge. Prompt for identifying highlights and missed opportunity on CLPC Consider the following classroom transcript. Transcript: {transcript} Please do the following.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Impacts of generative artificial intelligence on the future of labor market: a systematic review
Salari, N., Beiromvand, M., Hosseinian-Far, A., Habibi, J., Babajani, F., & Mohammadi, M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100652 · Cited by 49 · see citing works
“…songs, reports, forms, scripts, guides, creating codeless automated programs for businesses, and enhancing relationships between stakeholders and businesses through dialogue and service (Basic et al., 2023; Floridi & Chiriatti, 2020; Singh & Singh, 2023; Wang & Demszky, 2023; Zhang & Li, 2021).”
Tan, M., & Subramonyam, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642592 · Cited by 40 · see citing works
A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Huang, Y., Tang, K., Chen, M., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2404.15777 · Cited by 14 · see citing works
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024) · arXiv · doi:10.48550/arxiv.2410.03017 · Cited by 6 · see citing works
“LMs may be able to provide real-time guidance for novices at scale, but there are several challenges to enabling this.”
Self-citation
- Authors
- Wang, R. E., & Demszky, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2306.03090
- References
- 1
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
