Automatically Measuring Question Authenticity in Real-World Classrooms
Abstract
Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human-coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human-coded and computer-coded authenticity at the classroom level were sufficiently high ( r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment.
“e previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [37]
In 2018, one empirical study was published and it focused the deep learning technology on the modelling of scoring-based data.
“In 2018, one empirical study [37] was published and it focused the deep learning technology on the modelling of scoring-based data.”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [37]
Cites
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496 · see citing works
Searle, J. R. (1969) · doi:10.1017/cbo9781139173438 · Cited by 11,718 · see citing works
Handbook of research on teaching
Wittrock, M. C. (1986) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 10,630 · see citing works
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026683 · Cited by 8,134 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,150 · see citing works
Galar, M., Fernández, A., Barrenechea, E., Bustince, H., & Herrera, F. (2011) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2161285 · Cited by 2,820 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Elaborating a model of teacher professional growth
Clarke, D., & Hollingsworth, H. (2002) · Teaching and Teacher Education · doi:10.1016/s0742-051x(02)00053-7 · Cited by 2,228 · see citing works
(2008) · Cited by 1,400 · see citing works
MacNeilley, L. H. (1998) · Language · doi:10.2307/417942 · Cited by 1,237 · see citing works
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E. E., Bates, R. A., Jurafsky, D. S., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. W. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,112 · see citing works
Student engagement and achievement in American secondary schools
<vcard>BEGIN:vCard\VERSION:3.0\FN:B. Bradford Brown\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Adam Gamoran\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Susie D. Lamborn\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Karen Seashore-Louis\ORG:University of Minnesota\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Nina S. Mounts\ORG:Northern Illinois University\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Martin Nystrand\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:BetsAnn Smith\ORG:University of Minnesota\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Gregory A. Smith\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Laurence Steinberg\ORG:Temple University\ADR;TYPE=WORK, P., & <vcard>BEGIN:vCard\VERSION:3.0\FN:Gary G. Wehlage\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P. (1993) · Choice Reviews Online · doi:10.5860/choice.30-3945 · Cited by 1,109 · see citing works
10 CHALLENGING PROBLEMS IN DATA MINING RESEARCH
Yang, Q., & Wu, X. (2006) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622006002258 · Cited by 939 · see citing works
Question Asking During Tutoring
Graesser, A. C., & Person, N. K. (1994) · American Educational Research Journal · doi:10.3102/00028312031001104 · Cited by 900 · see citing works
Teacher learning in the context of a video club
Sherin, M. G., & Han, S. Y. (2004) · Teaching and Teacher Education · doi:10.1016/j.tate.2003.08.001 · Cited by 708 · see citing works
Proceedings of the 7th International Conference on Educational Data Mining.
Stamper, J. C., Pardos, Z. A., Mavrikis, M., & McLaren, B. M. (2014) · Educational Data Mining · Cited by 667 · see citing works
Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011) · Educational Researcher · doi:10.3102/0013189x11413260 · Cited by 635 · see citing works
Turner, J. C., Midgley, C., Meyer, D. K., Gheen, M. H., Anderman, E. M., Kang, Y., & Patrick, H. (2002) · Journal of Educational Psychology · doi:10.1037/0022-0663.94.1.88 · Cited by 628 · see citing works
Authentic Pedagogy and Student Performance
Newmann, F. M., Marks, H. M., & Gamoran, A. (1996) · American Journal of Education · doi:10.1086/444136 · Cited by 591 · see citing works
Language, the Learner and the School
Scott, J. L., Barnes, D., Britton, J., Rosen, H., & E., L. A. T. (1973) · The English Journal · doi:10.2307/813888 · Cited by 499 · see citing works
The Framework for Teaching Evaluation Instrument
Danielson, C. (2011) · Cited by 476 · see citing works
The Process of Question Answering
Lehnert, W. G. (2022) · Routledge eBooks · doi:10.4324/9781003316817 · Cited by 451 · see citing works
Using Model Trees for Classification
Frank, E., Wang, Y., Inglis, S. J., Holmes, G., & Witten, I. H. (1998) · Machine Learning · doi:10.1023/a:1007421302149 · Cited by 390 · see citing works
Patterns of Discourse in Two Kinds of Literature Discussion
Chinn, C. A., Anderson, R. C., & Waggoner, M. A. (2001) · Reading Research Quarterly · doi:10.1598/rrq.36.4.3 · Cited by 355 · see citing works
Research on the Role of Classroom Discourse As It Affects Reading Comprehension
Nystrand, M. (2006) · Research in the Teaching of English · doi:10.58680/rte20065107 · Cited by 351 · see citing works
Self-citationKersting, N., Givvin, K. B., Thompson, B. J., Santagata, R., & Stigler, J. (2012) · American Educational Research Journal · doi:10.3102/0002831212437853 · Cited by 291 · see citing works
Scaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART
Graesser, A. C., McNamara, D. S., & VanLehn, K. (2005) · Educational Psychologist · doi:10.1207/s15326985ep4004_4 · Cited by 285 · see citing works
Osborne, J., Simon, S., Christodoulou, A., Howell‐Richardson, C., & Richardson, K. (2013) · Journal of Research in Science Teaching · doi:10.1002/tea.21073 · Cited by 268 · see citing works
Classroom Discourse Analysis: A Tool for Critical Reflection
Rymes, B. (2009) · Works - Scholarship, Research, & Creative Expression (Swarthmore College) · Cited by 213 · see citing works
Innovations in Educational Psychology: Perspectives on Learning, Teaching, and Human Development
Preiss, D., & Sternberg, R. J. (2010) · Cited by 194 · see citing works
Recent Advances in Conversational Intelligent Tutoring Systems
Rus, V., D’Mello, S. K., Hu, X., & Graesser, A. C. (2013) · AI Magazine · doi:10.1609/aimag.v34i3.2485 · Cited by 189 · see citing works
Self-citationThe Role of MTurk in Education Research: Advantages, Issues, and Future Directions
Follmer, D. J., Sperling, R. A., & Suen, H. K. (2017) · Educational Researcher · doi:10.3102/0013189x17725519 · Cited by 184 · see citing works
Integrating Literacy and Science in Biology
Greenleaf, C. L., Litman, C., Hanson, T., Rosen, R., Boscardin, C., Herman, J. L., Schneider, S., Madden, S., & Jones, B. (2010) · American Educational Research Journal · doi:10.3102/0002831210384839 · Cited by 167 · see citing works
Bringing reading research to life
(2010) · Choice Reviews Online · doi:10.5860/choice.48-0406 · Cited by 164 · see citing works
Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom.
Roscoe, R. D., & McNamara, D. S. (2013) · Journal of Educational Psychology · doi:10.1037/a0032340 · Cited by 144 · see citing works
Questions and Information Systems
Eileen, T. L., Lauer, T. W., Peacock, E., Graesser, A. C., & White, J. S. (2013) · Psychology Press eBooks · doi:10.4324/9780203763148 · Cited by 138 · see citing works
Mechanisms That Generate Questions
Lauer, T. W., Peacock, E., & Graesser, A. C. (2013) · Psychology Press eBooks · doi:10.4324/9780203763148-12 · Cited by 130 · see citing works
Rosé, C. P., & Ferschke, O. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0107-y · Cited by 129 · see citing works
Inspiring Dialogue: Talking to Learn in the English Classroom
Juzwik, M. M. (2013) · Medical Entomology and Zoology · Cited by 119 · see citing works
Race, social class, and student engagement in middle school English classrooms
Kelly, S. (2007) · Social Science Research · doi:10.1016/j.ssresearch.2007.08.003 · Cited by 115 · see citing works
Self-citationAutomatic classification of activities in classroom discourse
Wang, Z., Pan, X., Miller, K., & Cortina, K. (2014) · Comput. Educ. · doi:10.1016/j.compedu.2014.05.010 · Cited by 102 · see citing works
Inquiry-based English instruction: engaging students in life and literature
(2002) · Choice Reviews Online · doi:10.5860/choice.39-4101 · Cited by 89 · see citing works
Psychology of classroom learning: an encyclopedia
(2009) · Choice Reviews Online · doi:10.5860/choice.47-0053 · Cited by 86 · see citing works
Classroom discourse and the distribution of student engagement
Kelly, S. (2007) · Social Psychology of Education · doi:10.1007/s11218-007-9024-0 · Cited by 85 · see citing works
Self-citationTwo Practical Rhetorical Structure Theory Parsers
Surdeanu, M., Hicks, T., & Valenzuela-Escárcega, M. A. (2015) · doi:10.3115/v1/n15-3001 · Cited by 85 · see citing works
English Teacher Candidates Developing Dialogically Organized Instructional Practices
Caughlan, S., Juzwik, M. M., Borsheim-Black, C., Kelly, S., & Fine, J. G. (2013) · Research in the Teaching of English · doi:10.58680/rte201322711 · Cited by 71 · see citing works
Self-citationMining Collaborative Patterns in Tutorial Dialogues
D’Mello, S. K., Olney, A. M., & Person, N. K. (2010) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554737 · Cited by 65 · see citing works
Self-citationD’Mello, S. K., Olney, A. M., Blanchard, N., Samei, B., Sun, X., Ward, B., & Kelly, S. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830602 · Cited by 65 · see citing works
Self-citationAutomatic Teacher Modeling from Live Classroom Audio
Donnelly, P., Blanchard, N., Samei, B., Olney, A. M., Sun, X., Ward, B., Kelly, S., Nystran, M., & D’Mello, S. K. (2016) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/2930238.2930250 · Cited by 64 · see citing works
Self-citationUtterance classification in AutoTutor
Olney, A. M., Louwerse, M. M., Mathews, E., Marineau, J., Mitchell, H. H., & Graesser, A. C. (2003) · doi:10.3115/1118894.1118895 · Cited by 54 · see citing works
Self-citationBetter Feedback for Better Teaching: A Practical Guide to Improving Classroom Observations.
Archer, J., Cantrell, S., Holtzman, S. L., Joe, J. N., Tocci, C. M., & Wood, J. (2016) · Cited by 53 · see citing works
Toward Spoken Human–Computer Tutorial Dialogues
D’Mello, S. K., Graesser, A. C., & King, B. G. (2010) · Human-Computer Interaction · doi:10.1080/07370024.2010.499850 · Cited by 50 · see citing works
Self-citationReal talk in elementary classrooms: effective oral language practice
(2011) · Choice Reviews Online · doi:10.5860/choice.49-2204 · Cited by 47 · see citing works
Talking, listening, and learning in elementary classrooms
Morine‐Dershimer, G. (1985) · Cited by 45 · see citing works
Developing a collaborative model of research using digital video
Armstrong, V. G., & Curran, S. (2006) · Computers & Education · doi:10.1016/j.compedu.2005.11.015 · Cited by 43 · see citing works
(2011) · Cited by 42 · see citing works
Self-citationInsights From Teachers Who Analyzed Transcripts of Their Own Classroom Discussions
Kucan, L. (2007) · The Reading Teacher · doi:10.1598/rt.61.3.3 · Cited by 41 · see citing works
Talking to Learn: The Promise and Challenge of Dialogic Teaching
Resnick, L. Β., & Schantz, F. (2015) · doi:10.3102/978-0-935302-43-1_34 · Cited by 38 · see citing works
Roskos, K. A., Boehlen, S., & Walker, B. J. (2000) · The Elementary School Journal · doi:10.1086/499641 · Cited by 36 · see citing works
City, E. A. (2014) · Educational leadership · Cited by 29 · see citing works
The narrative waltz: The role of flexibility in writing proficiency.
Allen, L. K., Snow, E. L., & McNamara, D. S. (2016) · Journal of Educational Psychology · doi:10.1037/edu0000109 · Cited by 26 · see citing works
Tutorial Dialog in Natural Language
Olney, A. M., Graesser, A. C., & Person, N. K. (2010) · Studies in computational intelligence · doi:10.1007/978-3-642-14363-2_9 · Cited by 23 · see citing works
Self-citationMaking themost from multiple microphones in meeting recognition
Stolcke, A. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947477 · Cited by 22 · see citing works
Kucan, L., Palincsar, A. S., Khasnabis, D., & Chang, C.-I. (2008) · Teaching and Teacher Education · doi:10.1016/j.tate.2008.09.003 · Cited by 21 · see citing works
Domain Independent Assessment of Dialogic Properties of Classroom Discourse
Samei, B., Olney, A. M., Kelly, S., Nystrand, M., D’Mello, S. K., Blanchard, N., Sun, X., Glaus, M., & Graesser, A. C. (2014) · Cited by 21 · see citing works
Self-citationMeasuring Instruction for Teacher Learning
Stein, M. K., & Matsumura, L. C. (2009) · Sage eBooks · doi:10.4135/9781483329857.n11 · Cited by 18 · see citing works
Learning Through Extended Talk
Thompson, P. (2008) · Language and Education · doi:10.1080/09500780802152507 · Cited by 16 · see citing works
Adler, M., & Rougle, E. (2005) · Cited by 16 · see citing works
Samei, B., Olney, A. M., Kelly, S., Nystrand, M., D’Mello, S. K., Blanchard, N., & Graesser, A. C. (2015) · Educational Data Mining · Cited by 14 · see citing works
Self-citationThe Hollywood teachers' perspective on authority
Kelly, S., & Caughlan, S. (2011) · Pedagogies An International Journal · doi:10.1080/1554480x.2011.532086 · Cited by 13 · see citing works
Self-citationRosemary, C. A., Freppon, P. A., & Kinnucan-Welsch, K. (2002) · Cited by 9 · see citing works
Effective Classroom Talk Is Reading Comprehension Instruction
McKeown, M. G., & Beck, I. L. (2015) · doi:10.3102/978-0-935302-43-1_4 · Cited by 9 · see citing works
Assessing the Dialogic Properties of Classroom Discourse: Proportion Models for Imbalanced Classes.
Olney, A. M., Samei, B., Donnelly, P., & D’Mello, S. K. (2017) · Educational Data Mining · Cited by 9 · see citing works
Self-citationKelly, S., Zhang, Y., Northrop, L., VanDerHeide, J., Dunn, M. B., & Caughlan, S. (2018) · Teacher education quarterly (Claremont, Calif.) · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
5 of the 80 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Review of Artificial Intelligence (AI) in Education from 2010 to 2020
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021) · Complexity · doi:10.1155/2021/8812542 · Cited by 1,364 · see citing works
“…diagnosis [19]. previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
BackgroundThe promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984 · see citing works
Coding classroom dialogue: Methodological considerations for researchers
Hennessy, S., Howe, C. J., Mercer, N., & Vrikki, M. (2020) · Learning Culture and Social Interaction · doi:10.1016/j.lcsi.2020.100404 · Cited by 135 · see citing works
Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. Β. (2020) · Journal of the Learning Sciences · doi:10.1080/10508406.2020.1783269 · Cited by 116 · see citing works
Demszky, D., Liu, J., Hill, H., Jurafsky, D., & Piech, C. (2023) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737231169270 · Cited by 141 · see citing works
Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning
Jensen, E., Dale, M., Donnelly, P. J., Stone, C., Kelly, S., GODLEY, A. J., & D’Mello, S. K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376418 · Cited by 99 · see citing works
Self-citationScalability, Sustainability, and Ethicality of Multimodal Learning Analytics
Yan, L., Zhao, L., Gasevic, D., & Martinez-Maldonado, R. (2022) · LAK22: 12th International Learning Analytics and Knowledge Conference · doi:10.1145/3506860.3506862 · Cited by 83 · see citing works
Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., & Sumner, T. (2022) · Teaching and Teacher Education : An International Journal of Research and Studies · doi:10.1016/j.tate.2022.103631 · Cited by 84 · see citing works
Artificial intelligence in classroom discourse: A systematic review of the past decade
Wang, D., Yang, T., & Chen, G. (2023) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102275 · Cited by 84 · see citing works
D’Mello, S. K., Southwell, R. V., & Gregg, J. M. (2020) · Discourse Processes · doi:10.1080/0163853x.2020.1739600 · Cited by 52 · see citing works
Self-citationTechnologies for automated analysis of co-located, real-life, physical learning spaces
Chua, Y. H. V., Dauwels, J., & Tan, S. C. (2019) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3303772.3303811 · Cited by 46 · see citing works
METS: Multimodal Learning Analytics of Embodied Teamwork Learning
Zhao, L., Swiecki, Z., Gašević, D., Yan, L., Dix, S., Jaggard, H., Wotherspoon, R., Osborne, A., Li, X., Alfredo, R., et al. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576076 · Cited by 45 · see citing works
A Deep Transfer Learning Approach to Modeling Teacher Discourse in the Classroom
Jensen, E., Pugh, S. L., & D’Mello, S. K. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448168 · Cited by 44 · see citing works
Self-citationDemszky, D., Liu, J., Hill, H. C., Sanghi, S., & Chung, A. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105183 · Cited by 49 · see citing works
Chen, Z. (2024) · Human Resource Development Review · doi:10.1177/15344843241273316 · Cited by 41 · see citing works
Ramakrishnan, A., Zylich, B., Ottmar, E., LoCasale-Crouch, J., & Whitehill, J. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3059209 · Cited by 40 · see citing works
Kelly, S., Bringe, R., Aucejo, E. M., & Fruehwirth, J. C. (2020) · Education Policy Analysis Archives · doi:10.14507/epaa.28.5012 · Cited by 39 · see citing works
Self-citationWang, R., & Demszky, D. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.53 · Cited by 37 · see citing works
Automated feedback on discourse moves: teachers’ perceived utility of a professional learning tool
Jacobs, J., Scornavacco, K., Clevenger, C., Suresh, A., & Sumner, T. (2024) · Educational Technology Research and Development · doi:10.1007/s11423-023-10338-6 · Cited by 35 · see citing works
Toward the automated analysis of teacher talk in secondary ELA classrooms
Dale, M., GODLEY, A. J., Capello, S. A., Donnelly, P. J., D’Mello, S. K., & Kelly, S. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103584 · Cited by 39 · see citing works
Self-citationMeasuring Teaching Practices at Scale: A Novel Application of Text-as-Data Methods
Liu, J., & Cohen, J. (2021) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211009267 · Cited by 34 · see citing works
The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts
Demszky, D., & Hill, H. C. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.44 · Cited by 31 · see citing works
Computationally Identifying Funneling and Focusing Questions in Classroom Discourse
Alic, S., Demszky, D., Mancenido, Z., Liu, J., Hill, H., & Jurafsky, D. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.27 · Cited by 54 · see citing works
Wang, R., & Demszky, D. (2023) · doi:10.35542/osf.io/5vrby · Cited by 50 · see citing works
Text as Data Methods for Education Research
Fesler, L., Dee, T. S., Baker, R. B., & Evans, B. J. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1634168 · Cited by 27 · see citing works
Demszky, D., & Liu, J. (2023) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3573051.3593379 · Cited by 27 · see citing works
Sun, M., Liu, J., Zhu, J., & LeClair, Z. (2019) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373719869318 · Cited by 24 · see citing works
Automatic classification of activities in classroom videos
Foster, J. K., Korban, M., Youngs, P. A., Watson, G. S., & Acton, S. T. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100207 · Cited by 24 · see citing works
Zhao, L., Gašević, D., Swiecki, Z., Li, Y., Lin, J., Sha, L., Yan, L., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13476 · Cited by 24 · see citing works
Zhao, L., Yan, L., Gašević, D., Dix, S., Jaggard, H., Wotherspoon, R., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2022) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3506860.3506935 · Cited by 23 · see citing works
Reshaping the Arc of Quantitative Educational Research: It’s Time to Broaden Our Paradigm
Singer, J. D. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1658835 · Cited by 23 · see citing works
Anglin, K. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1654576 · Cited by 22 · see citing works
Schlotterbeck, D., Uribe, P., Araya, R., Jiménez, A., & Caballero, D. (2021) · International Conference on Learning Analytics and Knowledge · doi:10.1145/3448139.3448152 · Cited by 19 · see citing works
Wang, D., & Chen, G. (2024) · IEEE Transactions on Education · doi:10.1109/te.2024.3421606 · Cited by 20 · see citing works
Nazaretsky, T., Mikeska, J. N., & Klebanov, B. B. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576067 · Cited by 21 · see citing works
Zhao, L., Echeverría, V., Swiecki, Z., Yan, L., Alfredo, R., Li, X., Gašević, D., & Martínez‐Maldonado, R. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636855 · Cited by 19 · see citing works
Human-tutor Coaching Technology (HTCT): Automated Discourse Analytics in a Coached Tutoring Model
Booth, B. M., Jacobs, J., Bush, J. B., Milne, B., Fischaber, T., & D’Mello, S. K. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636937 · Cited by 17 · see citing works
Self-citationNorthrop, L., & Kelly, S. (2018) · Reading Research Quarterly · doi:10.1002/rrq.237 · Cited by 17 · see citing works
Self-citationWang, D., & Chen, G. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13604 · Cited by 18 · see citing works
Zhao, L., Tan, Y., Gašević, D., Shaffer, D. W., Yan, L., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_20 · Cited by 16 · see citing works
Fütterer, T., Goldberg, P., Bühler, B., Sikimić, V., Trautwein, U., Gerjets, P. H., Stürmer, K., & Kasneci, E. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100483 · Cited by 28 · see citing works
TARTA: Teacher Activity Recognizer from Transcriptions and Audio
Schlotterbeck, D., Uribe, P., Jiménez, A., Araya, R., Moris, J. V. D. M., & Caballero, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78292-4_30 · Cited by 16 · see citing works
Authority and authenticity in teachers’ questions about literature in three contexts
Levine, S. R., Hauser, M. E., & Smith, M. W. (2022) · English Teaching Practice & Critique · doi:10.1108/etpc-03-2021-0021 · Cited by 18 · see citing works
Shin, J., Balyan, R., Banawan, M., Arner, T., Leite, W. L., & McNamara, D. S. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104897 · Cited by 13 · see citing works
Kelly, S., Guner, G., Hunkins, N., & D’Mello, S. K. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00417-x · Cited by 14 · see citing works
Self-citationDiffering Views of Equity: How Prospective Educators Perceive Their Role in Closing Achievement Gaps
Penner, E. K., Rochmes, J., Liu, J., Solanki, S. M., & Loeb, S. (2019) · RSF The Russell Sage Foundation Journal of the Social Sciences · doi:10.7758/rsf.2019.5.3.06 · Cited by 14 · see citing works
Abandoning questions with unpredictable answers
Gaspard, C., & Gainsburg, J. (2019) · Journal of Mathematics Teacher Education · doi:10.1007/s10857-019-09440-5 · Cited by 13 · see citing works
Corporate investment in artificial intelligence: The role of GDP, ICT exports, and patents
Rahman, M. S., Golder, U., & Ghosh, P. (2024) · Journal of Economics and Management · doi:10.22367/jem.2024.46.21 · Cited by 12 · see citing works
Anglin, K., Wong, V. C., & Boguslav, A. (2021) · AERA Open · doi:10.1177/23328584211028615 · Cited by 12 · see citing works
Simaremare, J. A., & Thesalonika, E. (2021) · Jurnal Tunas Bangsa · doi:10.46244/tunasbangsa.v8i2.1642 · Cited by 12 · see citing works
A Methodological Review of Machine Learning in Applied Linguistics
Lin, Z. (2020) · English Language Teaching · doi:10.5539/elt.v14n1p74 · Cited by 12 · see citing works
Are perfect transcripts necessary when we analyze classroom dialogue using AIoT?
Wang, D., & Chen, G. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101105 · Cited by 11 · see citing works
Schlotterbeck, D., Jiménez, A., Araya, R., Caballero, D., Uribe, P., & Moris, J. V. D. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11644-5_22 · Cited by 11 · see citing works
Lo, C. K., & Chen, G. (2021) · Sustainability · doi:10.3390/su13158610 · Cited by 10 · see citing works
Habib, F. A. B. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13271-0 · Cited by 10 · see citing works
Sert, O. (2025) · Journal of Research on Technology in Education · doi:10.1080/15391523.2025.2504355 · Cited by 10 · see citing works
Audio Features in Education: A Systematic Review of Computational Applications and Research Gaps
Pardo, F., Cánovas, Ó., & Clemente, F. J. G. (2025) · Applied Sciences · doi:10.3390/app15126911 · Cited by 10 · see citing works
Din Kültürü ve Ahlak Bilgisi Derslerinde Yapay Zekâ Destekli Modern Öğretim Yaklaşımları
ARICAN, S. (2024) · Dinbilimleri Akademik Araştırma Dergisi · doi:10.33415/daad.1591945 · Cited by 9 · see citing works
Altamirano, M., Uribe, P., Schlotterbeck, D., Jiménez, A., Araya, R., Moris, J. V. D. M., & Caballero, D. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.09.078 · Cited by 9 · see citing works
Ezema, K., Chandler, C., Southwell, R. V., Cholendiran, N., & D’Mello, S. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714059 · Cited by 9 · see citing works
Self-citationQuestioning Practice and Classroom Interaction
Darong, H. C., Niman, E. M., Menggo, S., & Beda, R. (2021) · Tell Teaching of English Language and Literature Journal · doi:10.30651/tell.v9i1.5905 · Cited by 8 · see citing works
Kelly, S., & Abruzzo, E. (2021) · Educational Researcher · doi:10.3102/0013189x20982255 · Cited by 7 · see citing works
Self-citationQuestioning practice in EFL classrrom interactions: from type to syntactical form
Darong, H. C. (2021) · International Journal of Education and Learning · doi:10.31763/ijele.v3i2.261 · Cited by 7 · see citing works
Utilization of Artificial Intelligence in Education: A Perspective on Learning Strategies
Yalçınalp, S., Türkoğlu, H., Koç, S. E., & Ersoy, H. (2024) · Artificial intelligence · doi:10.5772/intechopen.1005086 · Cited by 12 · see citing works
Pengembangan Media Bantu Latihan Bertahan Pada Bola Voli Tahun 2019
Sagala, R. S., & Daulay, D. E. (2020) · Journal Coaching Education Sports · doi:10.31599/jces.v1i2.371 · Cited by 6 · see citing works
Wang, D., Shan, D., Zheng, Y., & Chen, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_53 · Cited by 6 · see citing works
The silent scream of digital future: Exploring AI awareness and quiet quitting in Turkish teachers
Can, H. C., & Alas, D. K. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105589 · Cited by 5 · see citing works
Abdala, J. (2024) · SN Social Sciences · doi:10.1007/s43545-024-01024-2 · Cited by 5 · see citing works
Yapay Zekanın Eğitimdeki Etkisinin İncelenmesi Üzerine Bir Alan Araştırması
Korkmaz, M. S., & Polat, S. (2023) · International Conference on Recent and Innovative Results in Engineering and Technology (ICRIRET) · doi:10.59287/icriret.1371 · Cited by 5 · see citing works
Handa, K., Clapper, M., Boyle, J., Wang, R., Yang, D., Yeager, D. S., & Demszky, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.549 · Cited by 11 · see citing works
The Uses of Machine Learning (ML) in Teaching and Learning English Language: A Methodical Review.
AlHarbi, A. A. (2022) · المجلة التربوية لکلية التربية بسوهاج · doi:10.21608/edusohag.2022.212355 · Cited by 5 · see citing works
Cook, C. P., Olney, A. M., Kelly, S., & D’Mello, S. K. (2018) · Educational Data Mining · Cited by 5 · see citing works
Self-citationAnglin, K. (2024) · AERA Open · doi:10.1177/23328584241303495 · Cited by 4 · see citing works
Dialogische Klassengesprächsführung im Geschichtsunterricht
Zimmermann, M. (2022) · Wochenschau Verlag eBooks · doi:10.46499/1920 · Cited by 4 · see citing works
Automated Assessment of Classroom Interaction Based on Verbal Dynamics: A Deep Learning Approach
Singh, S., Singh, L., & Satsangee, N. (2025) · SN Computer Science · doi:10.1007/s42979-025-03770-3 · Cited by 7 · see citing works
Agnosticism in instructional observation systems
Kelly, S. (2023) · Education Policy Analysis Archives · doi:10.14507/epaa.31.7493 · Cited by 3 · see citing works
Self-citationClassroom Data Visualization: Tracking Individuals During Group-Centered Instruction
Makowski, M. B., & Lubienski, S. T. (2023) · Educational Researcher · doi:10.3102/0013189x231158374 · Cited by 3 · see citing works
Towards Automatic Analysis of Science Classroom Talk
Lehesvuori, S., Schlotterbeck, D., Jiménez, A., Caballero, D., Araya, R., & Hämäläinen, R. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/978-1-6684-6932-3.ch006 · Cited by 3 · see citing works
Kelly, S., Zhong, Y., Guner, G., Celik, C., & Litman, D. J. (2025) · Journal of School Choice · doi:10.1080/15582159.2025.2510599 · Cited by 2 · see citing works
Self-citationDiscovering technology-aided possibilities for automatic analysis of science teacher questions
Lehesvuori, S., Urrutia, F., Heilala, V., Araya, R., & Hämäläinen, R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10545-3 · Cited by 2 · see citing works
Xue, H. (2023) · International Conference on Education Technology and Computer · doi:10.1145/3629296.3629366 · Cited by 2 · see citing works
Classification of Instructional Activities in Classroom Videos Using Neural Networks
Foster, J. K., Korban, M., Youngs, P. A., Watson, G. S., & Acton, S. T. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198882077.003.0020 · Cited by 2 · see citing works
Towards an Audio-based CNN for Classroom Observation on a Smartwatch
Zualkernan, I. A., & Khan, M. S. (2020) · IEEE/ITU International Conference on Artificial Intelligence for Good (AI4G) · doi:10.1109/ai4g50087.2020.9311083 · Cited by 2 · see citing works
An Automatic Teacher Activity Recognition System based on Speech Transcriptions
Uribe, P., Schlotterbeck, D., Jiménez, A., Araya, R., Caballero, D., & Moris, J. V. D. M. (2021) · Latin American Conference on Learning Technologies (LACLO) · doi:10.1109/laclo54177.2021.00029 · Cited by 2 · see citing works
Hamuni, Idrus, M. I., & Aswati (2024) · International Journal of Religion · doi:10.61707/bwj2vb11 · Cited by 1 · see citing works
Suresh, A., Penuel, W. R., Jacobs, J., Raza, A., Martin, J. H., & Sumner, T. (2024) · doi:10.1093/oso/9780198882077.003.0017 · Cited by 1 · see citing works
Shin, J., Pinto, W. N., & Wang, B. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02711-5 · Cited by 1 · see citing works
Thảo, L. T., & Pham, T. T. (2025) · Language Testing in Asia · doi:10.1186/s40468-025-00399-0 · Cited by 1 · see citing works
Öztürk, H. İ., & Aksoy, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1195-1.ch003 · Cited by 1 · see citing works
Yin, X., & Chen, F. N. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1807137 · Cited by 1 · see citing works
Zhao, L., Echeverría, V., Gašević, D., Yan, L., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105655 · Cited by 1 · see citing works
Contingency, Convergence, and Divergence of Teachers’ Questions In Classroom Interactions
Darong, H. C. (2022) · Indonesian Journal of EFL and Linguistics · doi:10.21462/ijefl.v7i2.508 · Cited by 1 · see citing works
Salas, F. S., & Guzmán, S. C. (2021) · Estudios pedagógicos · doi:10.4067/s0718-07052021000200007 · Cited by 1 · see citing works
Analysis of Collaborative Argumentation in Text-based Classroom Discussions
Lugini, L. (2021) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 1 · see citing works
Sinclair, J. (2020) · TSpace (University of Toronto) · Cited by 2 · see citing works
Measuring Conversational Uptake: A Case Study on Student-Teacher InteractionsPreprint
Demszky, D., Liu, J., Mancenido, Z., Cohen, J., Hill, H. C., Jurafsky, D., & Hashimoto, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03873 · Cited by 1 · see citing works
Zhang, Y. (2019) · D-Scholarship@Pitt (University of Pittsburgh)
الدراسات المقارنة فى المناهج: دراسة تحلیلیة مقارنة، وإطار مقترح
الخيال, ن. ح. ع. ا. (2019) · مجلة کلیة التربیة فى العلوم التربویة · doi:10.21608/jfees.2019.79151
Duckworth, A., & CharacterLab (2020) · Character Lab Tips · doi:10.53776/tips-ask-dont-tell
PENINGKATAN PRESTASI BELAJAR FISIKA MATERI SUHU DAN KALOR MENGGUNAKAN MODEL PEMBELAJARAN SUPER
Yuliani, M. (2023) · Jurnal Pendidikan DEWANTARA Media Komunikasi Kreasi dan Inovasi Ilmiah Pendidikan · doi:10.55933/jpd.v9i1.481
- Authors
- Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K.
- Affiliations
- University of Pittsburgh (US)
- University of Memphis (US)
- California State University, Chico (US)
- University of Wisconsin–Madison (US)
- University of Colorado Boulder (US)
- Venue
- Educational Researcher
- Published
- 2018-06-29
- Volume
- 47
- Issue
- 7
- Pages
- 451-464
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2809787282
- Cited by
- 114
- References
- 80
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 21 · 2023: 19 · 2022: 10 · 2021: 18 · 2020: 10 · 2019: 9 · 2018: 2
- FWCI
- 12.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Funders
- Institute of Education Sciences (R305A130030)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- teacher questions, classroom discourse, teacher-student relationship, human coding, classroom observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review