GPTeach: Interactive TA Training with GPT-based Students
Open accessPublished version of a preprint
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Interactive and realistic teacher training is hard to scale. This is a key issue for learning at scale, as inadequate preparation can negatively impact both students and teachers. What if we could make the teacher training experience more engaging and, as a downstream effect, reduce the potential for harm that teachers-in-training could inflict on students? We present GPTeach, an interactive chat-based teacher training tool that allows novice teachers to practice with simulated students. We performed two studies to evaluate GPTeach: one think-aloud study and one A/B test between our tool and a baseline. Participants took the role of a teaching assistant conducting office hours with two GPT-simulated students. We found that our tool provides the opportunity for teachers to get valuable teaching practice without the pressures of affecting real students, allowing them to iterate their responses both during and across sessions. Additionally, participants enjoyed flexibility in tailoring their responses according to the varied personas, needs, and learning goals. In this paper, we provide quantitative results and qualitative observations to inform future work in this area. We conclude with a discussion of actionable design ideas for such systems, as well as other ways to use this tool for evaluating teachers and students. GPTeach has recently been deployed into the teacher training component of an online course with over 800 novice teachers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Mooney, C., & Duval, R. (1993) · doi:10.4135/9781412983532 · Cited by 893
Recipes for building an open-domain chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Shuster, K., Smith, E. M., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.13637 · Cited by 784
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · cited 1×
“Moreover, recent work has shown that with dedicated prompting techniques, LLMs can be successfully used to simulate human sub-populations [2].”
Social Simulacra: Creating Populated Prototypes for Social Computing Systems
Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., & Bernstein, M. S. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545616 · Cited by 212
PromptMaker: Prompt-based Prototyping with Large Language Models
Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503564 · Cited by 157
GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor
Jiang, E., Toh, E., Molina, A., Donsbach, A., Cai, C. J., & Terry, M. (2021) · Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3474349.3480209 · Cited by 27
Research works printed but not linked: 23
- Arora, S., Narayan, A., Chen, M. F., Orr, L. J., Guha, N., Bhatia, K., Chami, I., Sala, F., and Ré, C. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441 (2022). · arXiv
- 10.64152/10125/73488 · DOI · Paper page
- Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Kohd, P. W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang, P. On the Opportunities and Risks of Foundation Models. Tech. rep., Aug. 2021.
- 10.1016/j.compedu.2021.104158 · DOI · Paper page
- Cotton, D. R., Cotton, P. A., and Shipway, J. R. Chatting and cheating: Ensuring academic integrity in the era of chatgpt. Preprint. https://doi. org/10.35542/osf. io/mrz8h (2023). · doi:10.35542/osf
- Demszky, D., Liu, J., Hill, H. C., Jurafsky, D., and Piech, C. Can automated feedback improve teachers' uptake of student ideas? evidence from a randomized controlled trial in a large-scale online course. edworkingpaper no. 21--483. Annenberg Institute for School Reform at Brown University (2021).
- 10.18653/v1/2021.acl-long.130 · DOI · Paper page
- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Minneapolis, Minnesota, June 2019), Association for Computational Linguistics, pp. 4171--4186.
- Dzikovska, M. O., Moore, J. D., Steinhauser, N., Campbell, G., Farrow, E., and Callaway, C. B. Beetle ii: a system for tutoring and computational linguistics experimentation. In Proceedings of the ACL 2010 System Demonstrations (2010), pp. 13--18.
- Goe, L., Bell, C., and Little, O. Approaches to Evaluating Teacher Effectiveness: A Research Synthesis. National Comprehensive Center for Teacher Quality, June 2008.
- Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., and Louwerse, M. M. Autotutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers 36, 2 (2004), 180--192.
- Jääskeläinen, R. Think-aloud protocol. Handbook of translation studies 1 (2010), 371--374.
- 10.1145/3408877.3432533 · DOI
- 10.18653/v1/D17-2014 · DOI · Paper page
- 10.1080/13803610500392236 · DOI · Paper page
- 10.1007/s40593-014-0029-5 · DOI · Paper page
- OpenAI. GPT-4 technical report, 2023.
- 10.1145/3408877.3432562 · DOI
- 10.1145/3386527.3405915 · DOI
- 10.1145/3290605.3300587 · DOI · Paper page
- Ruan, S., Nie, A., Steenbergen, W., He, J., Zhang, J., Guo, M., Liu, Y., Nguyen, K. D.,Wang, C. Y., Ying, R., et al. Reinforcement learning tutor better supported lower performers in a math task. arXiv preprint arXiv:2304.04933 (2023). · arXiv
- Tack, A., and Piech, C. The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. In Proceedings of the 15th International Conference on Educational Data Mining (Durham, United Kingdom, July 2022), A. Mitrovic and N. Bosch, Eds., International Educational Data Mining Society, pp. 522--529.
- Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., and Drachsler, H. AreWe There Yet? - A Systematic Literature Review on Chatbots in Education. Frontiers in Artificial Intelligence 4 (July 2021), 654924
Cited by
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
Data-Driven Artificial Intelligence in Education: A Comprehensive Review
Ahmad, K., Iqbal, W., El-Hassan, A., Qadir, J., Benhaddou, D., Ayyash, M., & Al-Fuqaha, A. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3314610 · Cited by 347
History, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109
“These developments expand the scope and potential of automated code generation, setting a new standard in software engineering that offers both opportunities and challenges for the industry.”
Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education
Jin, H., Lee, S., Shin, H., & Kim, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642349 · Cited by 99
“While the development cost and skill barrier has limited teachable agents to few learning activities in the past, recent research showed LLMs can simulate virtual students and coaches through natural language prompting [37, 51, 76].”
InfluentialDai, W., Tsai, Y.-S., Lin, J., Aldino, A., Jin, H., Li, T., Gašević, D., & Chen, G. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100299 · Cited by 93
Metadata
- Authors
- Markel, J. M., Opferman, S. G., Landay, J. A., & Piech, C.
- Venue
- Proceedings of the Tenth ACM Conference on Learning @ Scale
- Publisher
- ACM
- Published
- 2023-07-20
- Pages
- 226-236
- OpenAlex
- W4384659608
- Cited by
- 147
- References
- 12
- FWCI
- 27.60
- Type
- conference paper
- Language
- en
- Open access
- bronze
- Funders
- Human Centered Artificial Intelligence (203116)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.