Cited by 115Online Learning and AnalyticsComputational and Text Analysis MethodsEvaluation of Teaching Practices

The DOI matches its Crossref record; 75 of the 80 references it cites were found in a registry or an index.

Automatically Measuring Question Authenticity in Real-World Classrooms

Kelly, Sean, Olney, Andrew M., Donnelly, Patrick, Nystrand, Martin, D’Mello, Sidney K.·2018·Educational Researcher

Open this paper in CoChat — free account

Abstract

Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human-coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human-coded and computer-coded authenticity at the classroom level were sufficiently high ( r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.

Integrity signals

Registry

Crossref · 10.3102/0013189x18785613

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

How To Do Things With Words
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496
Speech Acts
Searle, J. R. (1969) · doi:10.1017/cbo9781139173438 · Cited by 11,718
Handbook of research on teaching
Wittrock, M. C. (1986) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 10,630
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026683 · Cited by 8,271
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches
Galar, M., Fernández, A., Barrenechea, E., Bustince, H., & Herrera, F. (2011) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2161285 · Cited by 2,877
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Elaborating a model of teacher professional growth
Clarke, D., & Hollingsworth, H. (2002) · Teaching and Teacher Education · doi:10.1016/s0742-051x(02)00053-7 · Cited by 2,228
Untitled work
(2008) · Cited by 1,400

Show all 80 references

Research works printed but not linked: 17

Burke J., 2010, What’s the big idea? Question-driven units to motivate reading, writing, and thinking
Ford M., 2008, The LENA language environment analysis system
Gamoran A., 2003, Stability and change in American education: Structure, process, and outcomes, 109
Grossier P., 1964, How to use the fine art of questioning
Hamilton L., 2012, Assessing teacher quality: Understanding teacher effects on instruction and achievement, 49
Hardt M., 2016, Advances in neural information processing systems, 3315
Inter-University Consortium for Political and Social Research (ICPSR), 2014, Measures of effective teaching: Observation measures report
Kelly S., 2018, Teacher Education Quarterly, 45, 57
Measures of Effective Teaching Project, 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom
Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom, 30
Olney A. M., 2017, Design recommendations for intelligent tutoring systems: Assessment, 5, 261
Olney A. M., 2017, Proceedings of the 10th international conference on educational data mining, 162
Rosemary C. A., 2002, 51, 368
Samei B., 2015, Proceedings of the 8th international conference on educational data mining (EDM 2015), 444
Wang Z., 2013, Unterrichtswissenschaft, 4, 290
Wei R. C., 2010, Teacher assessment and the quest for teacher quality, 691

Sorted by citations

A Review of Artificial Intelligence (AI) in Education from 2010 to 2020
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021) · Complexity · doi:10.1155/2021/8812542 · Cited by 1,364
“…diagnosis [19]. previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the eﬀective interaction of the learning environment [37].”
Background
The promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984
Coding classroom dialogue: Methodological considerations for researchers
Hennessy, S., Howe, C. J., Mercer, N., & Vrikki, M. (2020) · Learning Culture and Social Interaction · doi:10.1016/j.lcsi.2020.100404 · Cited by 154
Efficacy of video-based teacher professional development for increasing classroom discourse and student learning
Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. Β. (2020) · Journal of the Learning Sciences · doi:10.1080/10508406.2020.1783269 · Cited by 130
Can Automated Feedback Improve Teachers’ Uptake of Student Ideas? Evidence From a Randomized Controlled Trial in a Large-Scale Online Course
Demszky, D., Liu, J., Hill, H. C., Jurafsky, D., & Piech, C. (2023) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737231169270 · Cited by 147
Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning
Jensen, E., Dale, M., Donnelly, P. J., Stone, C., Kelly, S., GODLEY, A. J., & D’Mello, S. K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376418 · Cited by 108
Self-citation
Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics
Yan, L., Zhao, L., Gasevic, D., & Martinez-Maldonado, R. (2022) · LAK22: 12th International Learning Analytics and Knowledge Conference · doi:10.1145/3506860.3506862 · Cited by 83
Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change
Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., & Sumner, T. (2022) · Teaching and Teacher Education : An International Journal of Research and Studies · doi:10.1016/j.tate.2022.103631 · Cited by 84
Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction
Wang, R. E., & Demszky, D. (2023) · arXiv · doi:10.48550/arxiv.2306.03090 · Cited by 74
Artificial intelligence in classroom discourse: A systematic review of the past decade
Wang, D., Yang, T., & Chen, G. (2023) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102275 · Cited by 96
Machine-Learned Computational Models Can Enhance the Study of Text and Discourse: A Case Study Using Eye Tracking to Model Reading Comprehension
D’Mello, S. K., Southwell, R. V., & Gregg, J. M. (2020) · Discourse Processes · doi:10.1080/0163853x.2020.1739600 · Cited by 55
Self-citation
Technologies for automated analysis of co-located, real-life, physical learning spaces
Chua, Y. H. V., Dauwels, J., & Tan, S. C. (2019) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3303772.3303811 · Cited by 46
METS: Multimodal Learning Analytics of Embodied Teamwork Learning
Zhao, L., Swiecki, Z., Gašević, D., Yan, L., Dix, S., Jaggard, H., Wotherspoon, R., Osborne, A., Li, X., Alfredo, R., et al. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576076 · Cited by 45
A Deep Transfer Learning Approach to Modeling Teacher Discourse in the Classroom
Jensen, E., Pugh, S. L., & D’Mello, S. K. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448168 · Cited by 46
Self-citation
Automated feedback improves teachers’ questioning quality in brick-and-mortar classrooms: Opportunities for further enhancement
Demszky, D., Liu, J., Hill, H. C., Sanghi, S., & Chung, A. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105183 · Cited by 52
Responsible AI in Organizational Training: Applications, Implications, and Recommendations for Future Development
Chen, Z. (2024) · Human Resource Development Review · doi:10.1177/15344843241273316 · Cited by 41
Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate
Ramakrishnan, A., Zylich, B., Ottmar, E., LoCasale-Crouch, J., & Whitehill, J. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3059209 · Cited by 40
Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations
Kelly, S., Bringe, R., Aucejo, E. M., & Fruehwirth, J. C. (2020) · Education Policy Analysis Archives · doi:10.14507/epaa.28.5012 · Cited by 41
Self-citation
Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction
Wang, R., & Demszky, D. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.53 · Cited by 41
Automated feedback on discourse moves: teachers’ perceived utility of a professional learning tool
Jacobs, J., Scornavacco, K., Clevenger, C., Suresh, A., & Sumner, T. (2024) · Educational Technology Research and Development · doi:10.1007/s11423-023-10338-6 · Cited by 35
Toward the automated analysis of teacher talk in secondary ELA classrooms
Dale, M., GODLEY, A. J., Capello, S. A., Donnelly, P. J., D’Mello, S. K., & Kelly, S. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103584 · Cited by 46
Self-citation
Measuring Teaching Practices at Scale: A Novel Application of Text-as-Data Methods
Liu, J., & Cohen, J. (2021) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211009267 · Cited by 37
The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts
Demszky, D., & Hill, H. C. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.44 · Cited by 63
Computationally Identifying Funneling and Focusing Questions in Classroom Discourse
Alic, S., Demszky, D., Mancenido, Z., Liu, J., Hill, H., & Jurafsky, D. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.27 · Cited by 54
Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom InstructionPreprint
Wang, R., & Demszky, D. (2023) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/5vrby · Cited by 74
Text as Data Methods for Education Research
Fesler, L., Dee, T. S., Baker, R. B., & Evans, B. J. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1634168 · Cited by 31
M-Powering Teachers: Natural Language Processing Powered Feedback Improves 1:1 Instruction and Student Outcomes
Demszky, D., & Liu, J. (2023) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3573051.3593379 · Cited by 27
Using a Text-as-Data Approach to Understand Reform Processes: A Deep Exploration of School Improvement Strategies
Sun, M., Liu, J., Zhu, J., & LeClair, Z. (2019) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373719869318 · Cited by 24
Automatic classification of activities in classroom videos
Foster, J. K., Korban, M., Youngs, P. A., Watson, G. S., & Acton, S. T. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100207 · Cited by 30
Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analytics
Zhao, L., Gašević, D., Swiecki, Z., Li, Y., Lin, J., Sha, L., Yan, L., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13476 · Cited by 29
Modelling Co-located Team Communication from Voice Detection and Positioning Data in Healthcare Simulation
Zhao, L., Yan, L., Gašević, D., Dix, S., Jaggard, H., Wotherspoon, R., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2022) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3506860.3506935 · Cited by 26
Reshaping the Arc of Quantitative Educational Research: It’s Time to Broaden Our Paradigm
Singer, J. D. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1658835 · Cited by 24
Gather-Narrow-Extract: A Framework for Studying Local Policy Variation Using Web-Scraping and Natural Language Processing
Anglin, K. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1654576 · Cited by 22
What Classroom Audio Tells About Teaching: A Cost-effective Approach for Detection of Teaching Practices Using Spectral Audio Features
Schlotterbeck, D., Uribe, P., Araya, R., Jiménez, A., & Caballero, D. (2021) · International Conference on Learning Analytics and Knowledge · doi:10.1145/3448139.3448152 · Cited by 19
Making AI Accessible for STEM Teachers: Using Explainable AI for Unpacking Classroom Discourse Analysis
Wang, D., & Chen, G. (2024) · IEEE Transactions on Education · doi:10.1109/te.2024.3421606 · Cited by 21
Empowering Teacher Learning with AI: Automated Evaluation of Teacher Attention to Student Ideas during Argumentation-focused Discussion
Nazaretsky, T., Mikeska, J. N., & Klebanov, B. B. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576067 · Cited by 24
Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team Learning
Zhao, L., Echeverría, V., Swiecki, Z., Yan, L., Alfredo, R., Li, X., Gašević, D., & Martínez‐Maldonado, R. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636855 · Cited by 21
Human-tutor Coaching Technology (HTCT): Automated Discourse Analytics in a Coached Tutoring Model
Booth, B. M., Jacobs, J., Bush, J. B., Milne, B., Fischaber, T., & D’Mello, S. K. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636937 · Cited by 18
Self-citation
Who Gets to Read What? Tracking, Instructional Practices, and Text Complexity for Middle School Struggling Readers
Northrop, L., & Kelly, S. (2018) · Reading Research Quarterly · doi:10.1002/rrq.237 · Cited by 21
Self-citation
Evaluating the use of BERT and Llama to analyse classroom dialogue for teachers' learning of dialogic pedagogy
Wang, D., & Chen, G. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13604 · Cited by 18
Analysing Verbal Communication in Embodied Team Learning Using Multimodal Data and Ordered Network Analysis
Zhao, L., Tan, Y., Gašević, D., Shaffer, D. W., Yan, L., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_20 · Cited by 16
Artificial intelligence in classroom management: A systematic review on educational purposes, technical implementations, and ethical considerations
Fütterer, T., Goldberg, P., Bühler, B., Sikimić, V., Trautwein, U., Gerjets, P. H., Stürmer, K., & Kasneci, E. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100483 · Cited by 38
TARTA: Teacher Activity Recognizer from Transcriptions and Audio
Schlotterbeck, D., Uribe, P., Jiménez, A., Araya, R., Moris, J. V. D. M., & Caballero, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78292-4_30 · Cited by 16
Authority and authenticity in teachers’ questions about literature in three contexts
Levine, S. R., Hauser, M. E., & Smith, M. W. (2022) · English Teaching Practice & Critique · doi:10.1108/etpc-03-2021-0021 · Cited by 23
Pedagogical discourse markers in online algebra learning: Unraveling instructor's communication using natural language processing
Shin, J., Balyan, R., Banawan, M., Arner, T., Leite, W. L., & McNamara, D. S. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104897 · Cited by 14
High School English Teachers Reflect on Their Talk: A Study of Response to Automated Feedback with the Teacher Talk Tool
Kelly, S., Guner, G., Hunkins, N., & D’Mello, S. K. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00417-x · Cited by 15
Self-citation
Differing Views of Equity: How Prospective Educators Perceive Their Role in Closing Achievement Gaps
Penner, E. K., Rochmes, J., Liu, J., Solanki, S. M., & Loeb, S. (2019) · RSF The Russell Sage Foundation Journal of the Social Sciences · doi:10.7758/rsf.2019.5.3.06 · Cited by 14
Abandoning questions with unpredictable answers
Gaspard, C., & Gainsburg, J. (2019) · Journal of Mathematics Teacher Education · doi:10.1007/s10857-019-09440-5 · Cited by 13
Corporate investment in artificial intelligence: The role of GDP, ICT exports, and patents
Rahman, M. S., Golder, U., & Ghosh, P. (2024) · Journal of Economics and Management · doi:10.22367/jem.2024.46.21 · Cited by 13
A Natural Language Processing Approach to Measuring Treatment Adherence and Consistency Using Semantic Similarity
Anglin, K., Wong, V. C., & Boguslav, A. (2021) · AERA Open · doi:10.1177/23328584211028615 · Cited by 14
PENERAPAN METODE COOPERATIF LEARNING TIPE JIGSAW UNTUK MENINGKATKAN MOTIVASI DAN HASIL BELAJAR MAHASISWA
Simaremare, J. A., & Thesalonika, E. (2021) · Jurnal Tunas Bangsa · doi:10.46244/tunasbangsa.v8i2.1642 · Cited by 12
A Methodological Review of Machine Learning in Applied Linguistics
Lin, Z. (2020) · English Language Teaching · doi:10.5539/elt.v14n1p74 · Cited by 12
Are perfect transcripts necessary when we analyze classroom dialogue using AIoT?
Wang, D., & Chen, G. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101105 · Cited by 13
“Teacher, Can You Say It Again?" Improving Automatic Speech Recognition Performance over Classroom Environments with Limited Data
Schlotterbeck, D., Jiménez, A., Araya, R., Caballero, D., Uribe, P., & Moris, J. V. D. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11644-5_22 · Cited by 11
Improving Experienced Mathematics Teachers’ Classroom Talk: A Visual Learning Analytics Approach to Professional Development
Lo, C. K., & Chen, G. (2021) · Sustainability · doi:10.3390/su13158610 · Cited by 10
Unveiling the role of educators attitudes & intention toward artificial intelligence in teaching: A multi-dimensional analysis
Habib, F. A. B. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13271-0 · Cited by 11
Partnering with AI in teacher education? Using an automatic question detection tool to reflect on classroom interaction
Sert, O. (2025) · Journal of Research on Technology in Education · doi:10.1080/15391523.2025.2504355 · Cited by 10
Audio Features in Education: A Systematic Review of Computational Applications and Research Gaps
Pardo, F., Cánovas, Ó., & Clemente, F. J. G. (2025) · Applied Sciences · doi:10.3390/app15126911 · Cited by 10
Din Kültürü ve Ahlak Bilgisi Derslerinde Yapay Zekâ Destekli Modern Öğretim Yaklaşımları
ARICAN, S. (2024) · Dinbilimleri Akademik Araştırma Dergisi · doi:10.33415/daad.1591945 · Cited by 9
Unsupervised characterization of lessons according to temporal patterns of teacher talk via topic modeling
Altamirano, M., Uribe, P., Schlotterbeck, D., Jiménez, A., Araya, R., Moris, J. V. D. M., & Caballero, D. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.09.078 · Cited by 9
“It feels like we're not meeting the criteria”: Examining and Mitigating the Cascading Effects of Bias in Automatic Speech Recognition in Spoken Language Interfaces.
Ezema, K., Chandler, C., Southwell, R. V., Cholendiran, N., & D’Mello, S. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714059 · Cited by 10
Self-citation
Questioning Practice and Classroom Interaction
Darong, H. C., Niman, E. M., Menggo, S., & Beda, R. (2021) · Tell Teaching of English Language and Literature Journal · doi:10.30651/tell.v9i1.5905 · Cited by 8
Using Lesson-Specific Teacher Reports of Student Engagement to Investigate Innovations in Curriculum and Instruction
Kelly, S., & Abruzzo, E. (2021) · Educational Researcher · doi:10.3102/0013189x20982255 · Cited by 7
Self-citation
Questioning practice in EFL classrrom interactions: from type to syntactical form
Darong, H. C. (2021) · International Journal of Education and Learning · doi:10.31763/ijele.v3i2.261 · Cited by 7
Utilization of Artificial Intelligence in Education: A Perspective on Learning Strategies
Yalçınalp, S., Türkoğlu, H., Koç, S. E., & Ersoy, H. (2024) · Artificial intelligence · doi:10.5772/intechopen.1005086 · Cited by 12
Pengembangan Media Bantu Latihan Bertahan Pada Bola Voli Tahun 2019
Sagala, R. S., & Daulay, D. E. (2020) · Journal Coaching Education Sports · doi:10.31599/jces.v1i2.371 · Cited by 6
Teacher Talk Moves in K12 Mathematics Lessons: Automatic Identification, Prediction Explanation, and Characteristic Exploration
Wang, D., Shan, D., Zheng, Y., & Chen, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_53 · Cited by 6
The silent scream of digital future: Exploring AI awareness and quiet quitting in Turkish teachers
Can, H. C., & Alas, D. K. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105589 · Cited by 8
Teachers’ questions and learners’ responses in teaching and learning English as a foreign language in Tanzania’s secondary schools
Abdala, J. (2024) · SN Social Sciences · doi:10.1007/s43545-024-01024-2 · Cited by 5
Yapay Zekanın Eğitimdeki Etkisinin İncelenmesi Üzerine Bir Alan Araştırması
Korkmaz, M. S., & Polat, S. (2023) · International Conference on Recent and Innovative Results in Engineering and Technology (ICRIRET) · doi:10.59287/icriret.1371 · Cited by 5
“Mistakes Help Us Grow”: Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms
Handa, K., Clapper, M., Boyle, J., Wang, R., Yang, D., Yeager, D. S., & Demszky, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.549 · Cited by 13
The Uses of Machine Learning (ML) in Teaching and Learning English Language: A Methodical Review.
AlHarbi, A. A. (2022) · المجلة التربوية لکلية التربية بسوهاج · doi:10.21608/edusohag.2022.212355 · Cited by 5
An Open Vocabulary Approach for Estimating Teacher Use of Authentic Questions in Classroom Discourse.
Cook, C. P., Olney, A. M., Kelly, S., & D’Mello, S. K. (2018) · Educational Data Mining · Cited by 5
Self-citation
Addressing Threats to Validity in Supervised Machine Learning: A Framework and Best Practices for Education Researchers
Anglin, K. (2024) · AERA Open · doi:10.1177/23328584241303495 · Cited by 8
Dialogische Klassengesprächsführung im Geschichtsunterricht
Zimmermann, M. (2022) · Wochenschau Verlag eBooks · doi:10.46499/1920 · Cited by 5
Automated Assessment of Classroom Interaction Based on Verbal Dynamics: A Deep Learning Approach
Singh, S., Singh, L., & Satsangee, N. (2025) · SN Computer Science · doi:10.1007/s42979-025-03770-3 · Cited by 7
Agnosticism in instructional observation systems
Kelly, S. (2023) · Education Policy Analysis Archives · doi:10.14507/epaa.31.7493 · Cited by 4
Self-citation
Classroom Data Visualization: Tracking Individuals During Group-Centered Instruction
Makowski, M. B., & Lubienski, S. T. (2023) · Educational Researcher · doi:10.3102/0013189x231158374 · Cited by 3
Towards Automatic Analysis of Science Classroom Talk
Lehesvuori, S., Schlotterbeck, D., Jiménez, A., Caballero, D., Araya, R., & Hämäläinen, R. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/978-1-6684-6932-3.ch006 · Cited by 3
Changing Value Dimensions in Higher Education? Evidence from Faculty Job Postings in Education, Business, and the Social Sciences
Kelly, S., Zhong, Y., Guner, G., Celik, C., & Litman, D. J. (2025) · Journal of School Choice · doi:10.1080/15582159.2025.2510599 · Cited by 2
Self-citation
Discovering technology-aided possibilities for automatic analysis of science teacher questions
Lehesvuori, S., Urrutia, F., Heilala, V., Araya, R., & Hämäläinen, R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10545-3 · Cited by 6
Development History and Frontier Trends of Natural Language Processing Technology in Education: A Knowledge Graph-Based Visualization Analysis
Xue, H. (2023) · International Conference on Education Technology and Computer · doi:10.1145/3629296.3629366 · Cited by 2
Classification of Instructional Activities in Classroom Videos Using Neural Networks
Foster, J. K., Korban, M., Youngs, P. A., Watson, G. S., & Acton, S. T. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198882077.003.0020 · Cited by 2
Towards an Audio-based CNN for Classroom Observation on a Smartwatch
Zualkernan, I. A., & Khan, M. S. (2020) · IEEE/ITU International Conference on Artificial Intelligence for Good (AI4G) · doi:10.1109/ai4g50087.2020.9311083 · Cited by 2
An Automatic Teacher Activity Recognition System based on Speech Transcriptions
Uribe, P., Schlotterbeck, D., Jiménez, A., Araya, R., Caballero, D., & Moris, J. V. D. M. (2021) · Latin American Conference on Learning Technologies (LACLO) · doi:10.1109/laclo54177.2021.00029 · Cited by 2
The Influence of Cooperative Learning Models and Formative Test Types on Students' Civic Education Learning Outcomes
Hamuni, Idrus, M. I., & Aswati (2024) · International Journal of Religion · doi:10.61707/bwj2vb11 · Cited by 1
Using AI Tools to Provide Teachers with Fully Automated, Personalized Feedback on Their Classroom Discourse Patterns
Suresh, A., Penuel, W. R., Jacobs, J., Raza, A., Martin, J. H., & Sumner, T. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198882077.003.0017 · Cited by 1
Investigating the effect of transformer encoder architecture to improve the reliability of classroom observation ratings on high-inference discourse
Shin, J., Pinto, W. N., & Wang, B. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02711-5 · Cited by 2
Writing-With assessment: dialogic reflection as a bridge between teacher feedback and student learning
Thảo, L. T., & Pham, T. T. (2025) · Language Testing in Asia · doi:10.1186/s40468-025-00399-0 · Cited by 2
Tailored Learning at Scale
Öztürk, H. İ., & Aksoy, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1195-1.ch003 · Cited by 1
Reshaping the resilient educational ecosystem in the intelligent era: a case study of the CLIL course of international trade practice
Yin, X., & Chen, F. N. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1807137 · Cited by 1
Automated multimodal visual analytics of social and epistemic dimensions in team communication for reflective practice in healthcare simulation
Zhao, L., Echeverría, V., Gašević, D., Yan, L., Alfredo, R., Li, X., & Martínez‐Maldonado, R. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105655 · Cited by 1
Contingency, Convergence, and Divergence of Teachers’ Questions In Classroom Interactions
Darong, H. C. (2022) · Indonesian Journal of EFL and Linguistics · doi:10.21462/ijefl.v7i2.508 · Cited by 1
Caracterización de la enseñanza de matemática en educación superior técnico profesional a través del estudio de las preguntas hechas por docentes en la sala de clases
Salas, F. S., & Guzmán, S. C. (2021) · Estudios pedagógicos · doi:10.4067/s0718-07052021000200007 · Cited by 1
Analysis of Collaborative Argumentation in Text-based Classroom Discussions
Lugini, L. (2021) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 1
Using Machine Learning to Predict Children’s Reading Comprehension from Lexical and Syntactic Features Extracted from Spoken and Written Language
Sinclair, J. (2020) · TSpace (University of Toronto) · Cited by 2
Measuring Conversational Uptake: A Case Study on Student-Teacher InteractionsPreprint
Demszky, D., Liu, J., Mancenido, Z., Cohen, J., Hill, H. C., Jurafsky, D., & Hashimoto, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03873 · Cited by 1
ENGAGEMENT, INSTRUCTION, AND PROFESSIONAL DEVELOPMENT: INSIGHTS FROM INTERNATIONAL LARGE-SCALE SURVEY AND ASSESSMENT STUDIES
Zhang, Y. (2019) · D-Scholarship@Pitt (University of Pittsburgh)
الدراسات المقارنة فى المناهج: دراسة تحلیلیة مقارنة، وإطار مقترح
الخيال, ن. ح. ع. ا. (2019) · مجلة کلیة التربیة فى العلوم التربویة · doi:10.21608/jfees.2019.79151
Ask, Don’t Tell
Duckworth, A., & CharacterLab (2020) · Character Lab Tips · doi:10.53776/tips-ask-dont-tell

Authors: Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K.
Affiliations: University of Pittsburgh (US)
University of Memphis (US)
California State University, Chico (US)
University of Wisconsin–Madison (US)
University of Colorado Boulder (US)
ORCID: Kelly, Sean
Olney, Andrew M.
Donnelly, Patrick
D’Mello, Sidney K.
Venue: Educational Researcher
Publisher: American Educational Research Association (AERA)
Published: 2018-06-29
Volume: 47
Issue: 7
Pages: 451-464
ISSN: 0013-189X (ISSN-L)
1935-102X
DOI: 10.3102/0013189x18785613
OpenAlex: W2809787282
Cited by: 114
References: 80
Citations by year: 2026: 8 · 2025: 17 · 2024: 21 · 2023: 19 · 2022: 10 · 2021: 18 · 2020: 10 · 2019: 9 · 2018: 2
FWCI: 12.32
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Educational Researcher (publishedVersion)
Funders: Institute of Education Sciences (R305A130030)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Social Sciences › Education › Evaluation of Teaching Practices
Keywords: teacher questions, classroom discourse, teacher-student relationship, human coding, classroom observation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Automatically Measuring Question Authenticity in Real-World Classrooms

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks