The DOI matches its Crossref record; 75 of the 80 references it cites were found in a registry or an index.
Automatically Measuring Question Authenticity in Real-World Classrooms
Abstract
Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human-coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human-coded and computer-coded authenticity at the classroom level were sufficiently high ( r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“e previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [37]
In the citing paper
“In 2018, one empirical study [37] was published and it focused the deep learning technology on the modelling of scoring-based data.”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [37]
Citations
Cites
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496
Searle, J. R. (1969) · doi:10.1017/cbo9781139173438 · Cited by 11,718
Handbook of research on teaching
Wittrock, M. C. (1986) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 10,630
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026683 · Cited by 8,271
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Galar, M., Fernández, A., Barrenechea, E., Bustince, H., & Herrera, F. (2011) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2161285 · Cited by 2,877
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Elaborating a model of teacher professional growth
Clarke, D., & Hollingsworth, H. (2002) · Teaching and Teacher Education · doi:10.1016/s0742-051x(02)00053-7 · Cited by 2,228
(2008) · Cited by 1,400
MacNeilley, L. H. (1998) · Language · doi:10.2307/417942 · Cited by 1,238
Student engagement and achievement in American secondary schools
<vcard>BEGIN:vCard\VERSION:3.0\FN:B. Bradford Brown\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Adam Gamoran\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Susie D. Lamborn\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Karen Seashore-Louis\ORG:University of Minnesota\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Nina S. Mounts\ORG:Northern Illinois University\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Martin Nystrand\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:BetsAnn Smith\ORG:University of Minnesota\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Gregory A. Smith\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P., <vcard>BEGIN:vCard\VERSION:3.0\FN:Laurence Steinberg\ORG:Temple University\ADR;TYPE=WORK, P., & <vcard>BEGIN:vCard\VERSION:3.0\FN:Gary G. Wehlage\ORG:University of Wisconsin-Madison\ADR;TYPE=WORK, P. (1993) · Choice Reviews Online · doi:10.5860/choice.30-3945 · Cited by 1,112
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E. E., Bates, R. A., Jurafsky, D. S., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. W. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,112
10 CHALLENGING PROBLEMS IN DATA MINING RESEARCH
Yang, Q., & Wu, X. (2006) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622006002258 · Cited by 956
Question Asking During Tutoring
Graesser, A. C., & Person, N. K. (1994) · American Educational Research Journal · doi:10.3102/00028312031001104 · Cited by 900
Teacher learning in the context of a video club
Sherin, M. G., & Han, S. Y. (2004) · Teaching and Teacher Education · doi:10.1016/j.tate.2003.08.001 · Cited by 742
Proceedings of the 7th International Conference on Educational Data Mining.
Stamper, J. C., Pardos, Z. A., Mavrikis, M., & McLaren, B. M. (2014) · Educational Data Mining · Cited by 667
Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011) · Educational Researcher · doi:10.3102/0013189x11413260 · Cited by 635
Turner, J. C., Midgley, C., Meyer, D. K., Gheen, M. H., Anderman, E. M., Kang, Y., & Patrick, H. (2002) · Journal of Educational Psychology · doi:10.1037/0022-0663.94.1.88 · Cited by 628
Authentic Pedagogy and Student Performance
Newmann, F. M., Marks, H. M., & Gamoran, A. (1996) · American Journal of Education · doi:10.1086/444136 · Cited by 606
Language, the Learner and the School
Scott, J. L., Barnes, D., Britton, J., Rosen, H., & E., L. A. T. (1973) · The English Journal · doi:10.2307/813888 · Cited by 499
The Framework for Teaching Evaluation Instrument
Danielson, C. (2011) · Cited by 476
The Process of Question Answering
Lehnert, W. G. (2022) · Routledge eBooks · doi:10.4324/9781003316817 · Cited by 451
Kersting, N. B., Givvin, K. B., Thompson, B. J., Santagata, R., & Stigler, J. W. (2012) · American Educational Research Journal · doi:10.3102/0002831212437853 · Cited by 393
Using Model Trees for Classification
Frank, E., Wang, Y., Inglis, S. J., Holmes, G., & Witten, I. H. (1998) · Machine Learning · doi:10.1023/a:1007421302149 · Cited by 389
Patterns of Discourse in Two Kinds of Literature Discussion
Chinn, C. A., Anderson, R. C., & Waggoner, M. A. (2001) · Reading Research Quarterly · doi:10.1598/rrq.36.4.3 · Cited by 355
Research on the Role of Classroom Discourse As It Affects Reading Comprehension
Nystrand, M. (2006) · Research in the Teaching of English · doi:10.58680/rte20065107 · Cited by 351
Self-citationScaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART
Graesser, A. C., McNamara, D. S., & VanLehn, K. (2005) · Educational Psychologist · doi:10.1207/s15326985ep4004_4 · Cited by 285
Osborne, J., Simon, S., Christodoulou, A., Howell‐Richardson, C., & Richardson, K. (2013) · Journal of Research in Science Teaching · doi:10.1002/tea.21073 · Cited by 268
Classroom Discourse Analysis: A Tool for Critical Reflection
Rymes, B. (2009) · Works - Scholarship, Research, & Creative Expression (Swarthmore College) · Cited by 213
Recent Advances in Conversational Intelligent Tutoring Systems
Rus, V., D’Mello, S. K., Hu, X., & Graesser, A. C. (2013) · AI Magazine · doi:10.1609/aimag.v34i3.2485 · Cited by 199
Self-citationInnovations in Educational Psychology: Perspectives on Learning, Teaching, and Human Development
Preiss, D., & Sternberg, R. J. (2010) · Cited by 194
Integrating Literacy and Science in Biology
Greenleaf, C. L., Litman, C., Hanson, T., Rosen, R., Boscardin, C., Herman, J. L., Schneider, S., Madden, S., & Jones, B. (2010) · American Educational Research Journal · doi:10.3102/0002831210384839 · Cited by 186
The Role of MTurk in Education Research: Advantages, Issues, and Future Directions
Follmer, D. J., Sperling, R. A., & Suen, H. K. (2017) · Educational Researcher · doi:10.3102/0013189x17725519 · Cited by 184
Bringing reading research to life
(2010) · Choice Reviews Online · doi:10.5860/choice.48-0406 · Cited by 164
Rosé, C. P., & Ferschke, O. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0107-y · Cited by 145
Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom.
Roscoe, R. D., & McNamara, D. S. (2013) · Journal of Educational Psychology · doi:10.1037/a0032340 · Cited by 144
Questions and Information Systems
Eileen, T. L., Lauer, T. W., Peacock, E., Graesser, A. C., & White, J. S. (2013) · Psychology Press eBooks · doi:10.4324/9780203763148 · Cited by 138
Mechanisms That Generate Questions
Lauer, T. W., Peacock, E., & Graesser, A. C. (2013) · Psychology Press eBooks · doi:10.4324/9780203763148-12 · Cited by 128
Race, social class, and student engagement in middle school English classrooms
Kelly, S. (2007) · Social Science Research · doi:10.1016/j.ssresearch.2007.08.003 · Cited by 125
Self-citationInspiring Dialogue: Talking to Learn in the English Classroom
Juzwik, M. M. (2013) · Medical Entomology and Zoology · Cited by 118
Automatic classification of activities in classroom discourse
Wang, Z., Pan, X., Miller, K. F., & Cortina, K. S. (2014) · Computers & Education · doi:10.1016/j.compedu.2014.05.010 · Cited by 107
Inquiry-based English instruction: engaging students in life and literature
(2002) · Choice Reviews Online · doi:10.5860/choice.39-4101 · Cited by 89
Classroom discourse and the distribution of student engagement
Kelly, S. (2007) · Social Psychology of Education · doi:10.1007/s11218-007-9024-0 · Cited by 89
Self-citationPsychology of classroom learning: an encyclopedia
(2009) · Choice Reviews Online · doi:10.5860/choice.47-0053 · Cited by 86
Two Practical Rhetorical Structure Theory Parsers
Surdeanu, M., Hicks, T., & Valenzuela-Escárcega, M. A. (2015) · doi:10.3115/v1/n15-3001 · Cited by 85
English Teacher Candidates Developing Dialogically Organized Instructional Practices
Caughlan, S., Juzwik, M. M., Borsheim-Black, C., Kelly, S., & Fine, J. G. (2013) · Research in the Teaching of English · doi:10.58680/rte201322711 · Cited by 72
Self-citationD’Mello, S. K., Olney, A. M., Blanchard, N., Samei, B., Sun, X., Ward, B., & Kelly, S. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830602 · Cited by 72
Self-citationMining Collaborative Patterns in Tutorial Dialogues
D’Mello, S. K., Olney, A. M., & Person, N. K. (2010) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554737 · Cited by 65
Self-citationAutomatic Teacher Modeling from Live Classroom Audio
Donnelly, P., Blanchard, N., Samei, B., Olney, A. M., Sun, X., Ward, B., Kelly, S., Nystran, M., & D’Mello, S. K. (2016) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/2930238.2930250 · Cited by 64
Self-citationUtterance classification in AutoTutor
Olney, A. M., Louwerse, M. M., Mathews, E., Marineau, J., Mitchell, H. H., & Graesser, A. C. (2003) · doi:10.3115/1118894.1118895 · Cited by 54
Self-citationBetter Feedback for Better Teaching: A Practical Guide to Improving Classroom Observations.
Archer, J., Cantrell, S., Holtzman, S. L., Joe, J. N., Tocci, C. M., & Wood, J. (2016) · Cited by 53
Toward Spoken Human–Computer Tutorial Dialogues
D’Mello, S. K., Graesser, A. C., & King, B. G. (2010) · Human-Computer Interaction · doi:10.1080/07370024.2010.499850 · Cited by 50
Self-citationReal talk in elementary classrooms: effective oral language practice
(2011) · Choice Reviews Online · doi:10.5860/choice.49-2204 · Cited by 47
Talking, listening, and learning in elementary classrooms
Morine‐Dershimer, G. (1985) · Cited by 45
Developing a collaborative model of research using digital video
Armstrong, V. G., & Curran, S. (2006) · Computers & Education · doi:10.1016/j.compedu.2005.11.015 · Cited by 43
(2011) · Cited by 42
Self-citationInsights From Teachers Who Analyzed Transcripts of Their Own Classroom Discussions
Kucan, L. (2007) · The Reading Teacher · doi:10.1598/rt.61.3.3 · Cited by 41
Talking to Learn: The Promise and Challenge of Dialogic Teaching
Resnick, L. Β., & Schantz, F. (2015) · doi:10.3102/978-0-935302-43-1_34 · Cited by 38
Roskos, K. A., Boehlen, S., & Walker, B. J. (2000) · The Elementary School Journal · doi:10.1086/499641 · Cited by 36
City, E. A. (2014) · Educational leadership · Cited by 29
The narrative waltz: The role of flexibility in writing proficiency.
Allen, L. K., Snow, E. L., & McNamara, D. S. (2016) · Journal of Educational Psychology · doi:10.1037/edu0000109 · Cited by 26
Tutorial Dialog in Natural Language
Olney, A. M., Graesser, A. C., & Person, N. K. (2010) · Studies in computational intelligence · doi:10.1007/978-3-642-14363-2_9 · Cited by 23
Self-citationMaking themost from multiple microphones in meeting recognition
Stolcke, A. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947477 · Cited by 22
Kucan, L., Palincsar, A. S., Khasnabis, D., & Chang, C.-I. (2008) · Teaching and Teacher Education · doi:10.1016/j.tate.2008.09.003 · Cited by 21
Domain Independent Assessment of Dialogic Properties of Classroom Discourse
Samei, B., Olney, A. M., Kelly, S., Nystrand, M., D’Mello, S. K., Blanchard, N., Sun, X., Glaus, M., & Graesser, A. C. (2014) · Cited by 21
Self-citationMeasuring Instruction for Teacher Learning
Stein, M. K., & Matsumura, L. C. (2009) · Sage eBooks · doi:10.4135/9781483329857.n11 · Cited by 18
Learning Through Extended Talk
Thompson, P. (2008) · Language and Education · doi:10.1080/09500780802152507 · Cited by 16
Adler, M., & Rougle, E. (2005) · Cited by 16
Samei, B., Olney, A. M., Kelly, S., Nystrand, M., D’Mello, S. K., Blanchard, N., & Graesser, A. C. (2015) · Educational Data Mining · Cited by 14
Self-citationThe Hollywood teachers' perspective on authority
Kelly, S., & Caughlan, S. (2011) · Pedagogies An International Journal · doi:10.1080/1554480x.2011.532086 · Cited by 13
Self-citationRosemary, C. A., Freppon, P. A., & Kinnucan-Welsch, K. (2002) · Cited by 9
Effective Classroom Talk Is Reading Comprehension Instruction
McKeown, M. G., & Beck, I. L. (2015) · doi:10.3102/978-0-935302-43-1_4 · Cited by 9
Assessing the Dialogic Properties of Classroom Discourse: Proportion Models for Imbalanced Classes.
Olney, A. M., Samei, B., Donnelly, P., & D’Mello, S. K. (2017) · Educational Data Mining · Cited by 9
Self-citationKelly, S., Zhang, Y., Northrop, L., VanDerHeide, J., Dunn, M. B., & Caughlan, S. (2018) · Teacher education quarterly (Claremont, Calif.) · Cited by 4
Self-citation
Research works printed but not linked: 17
- Burke J., 2010, What’s the big idea? Question-driven units to motivate reading, writing, and thinking
- Ford M., 2008, The LENA language environment analysis system
- Gamoran A., 2003, Stability and change in American education: Structure, process, and outcomes, 109
- Grossier P., 1964, How to use the fine art of questioning
- Hamilton L., 2012, Assessing teacher quality: Understanding teacher effects on instruction and achievement, 49
- Hardt M., 2016, Advances in neural information processing systems, 3315
- Inter-University Consortium for Political and Social Research (ICPSR), 2014, Measures of effective teaching: Observation measures report
- Kelly S., 2018, Teacher Education Quarterly, 45, 57
- Measures of Effective Teaching Project, 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
- Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom
- Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom, 30
- Olney A. M., 2017, Design recommendations for intelligent tutoring systems: Assessment, 5, 261
- Olney A. M., 2017, Proceedings of the 10th international conference on educational data mining, 162
- Rosemary C. A., 2002, 51, 368
- Samei B., 2015, Proceedings of the 8th international conference on educational data mining (EDM 2015), 444
- Wang Z., 2013, Unterrichtswissenschaft, 4, 290
- Wei R. C., 2010, Teacher assessment and the quest for teacher quality, 691
Cited by
Sorted by citations
A Review of Artificial Intelligence (AI) in Education from 2010 to 2020
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021) · Complexity · doi:10.1155/2021/8812542 · Cited by 1,364
“…diagnosis [19]. previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
BackgroundThe promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984
Coding classroom dialogue: Methodological considerations for researchers
Hennessy, S., Howe, C. J., Mercer, N., & Vrikki, M. (2020) · Learning Culture and Social Interaction · doi:10.1016/j.lcsi.2020.100404 · Cited by 154
Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. Β. (2020) · Journal of the Learning Sciences · doi:10.1080/10508406.2020.1783269 · Cited by 130
Demszky, D., Liu, J., Hill, H. C., Jurafsky, D., & Piech, C. (2023) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737231169270 · Cited by 147
Metadata
- Authors
- Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K.
- Affiliations
- University of Pittsburgh (US)
- University of Memphis (US)
- California State University, Chico (US)
- University of Wisconsin–Madison (US)
- University of Colorado Boulder (US)
- Venue
- Educational Researcher
- Published
- 2018-06-29
- Volume
- 47
- Issue
- 7
- Pages
- 451-464
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2809787282
- Cited by
- 114
- References
- 80
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 21 · 2023: 19 · 2022: 10 · 2021: 18 · 2020: 10 · 2019: 9 · 2018: 2
- FWCI
- 12.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Funders
- Institute of Education Sciences (R305A130030)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- teacher questions, classroom discourse, teacher-student relationship, human coding, classroom observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
