The DOI matches its Crossref record; 75 of the 80 references it cites were found in a registry or an index.
Automatically Measuring Question Authenticity in Real-World Classrooms
Abstract
Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human-coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human-coded and computer-coded authenticity at the classroom level were sufficiently high ( r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“e previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [37]
In the citing paper
“In 2018, one empirical study [37] was published and it focused the deep learning technology on the modelling of scoring-based data.”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [37]
Citations
Cites
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496
Searle, J. R. (1969) · doi:10.1017/cbo9781139173438 · Cited by 11,718
Handbook of research on teaching
Wittrock, M. C. (1986) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 10,630
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026683 · Cited by 8,271
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Galar, M., Fernández, A., Barrenechea, E., Bustince, H., & Herrera, F. (2011) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2161285 · Cited by 2,877
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Elaborating a model of teacher professional growth
Clarke, D., & Hollingsworth, H. (2002) · Teaching and Teacher Education · doi:10.1016/s0742-051x(02)00053-7 · Cited by 2,228
(2008) · Cited by 1,400
Research works printed but not linked: 17
- Burke J., 2010, What’s the big idea? Question-driven units to motivate reading, writing, and thinking
- Ford M., 2008, The LENA language environment analysis system
- Gamoran A., 2003, Stability and change in American education: Structure, process, and outcomes, 109
- Grossier P., 1964, How to use the fine art of questioning
- Hamilton L., 2012, Assessing teacher quality: Understanding teacher effects on instruction and achievement, 49
- Hardt M., 2016, Advances in neural information processing systems, 3315
- Inter-University Consortium for Political and Social Research (ICPSR), 2014, Measures of effective teaching: Observation measures report
- Kelly S., 2018, Teacher Education Quarterly, 45, 57
- Measures of Effective Teaching Project, 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
- Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom
- Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom, 30
- Olney A. M., 2017, Design recommendations for intelligent tutoring systems: Assessment, 5, 261
- Olney A. M., 2017, Proceedings of the 10th international conference on educational data mining, 162
- Rosemary C. A., 2002, 51, 368
- Samei B., 2015, Proceedings of the 8th international conference on educational data mining (EDM 2015), 444
- Wang Z., 2013, Unterrichtswissenschaft, 4, 290
- Wei R. C., 2010, Teacher assessment and the quest for teacher quality, 691
Cited by
Sorted by citations
A Review of Artificial Intelligence (AI) in Education from 2010 to 2020
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021) · Complexity · doi:10.1155/2021/8812542 · Cited by 1,364
“…diagnosis [19]. previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
BackgroundThe promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984
Coding classroom dialogue: Methodological considerations for researchers
Hennessy, S., Howe, C. J., Mercer, N., & Vrikki, M. (2020) · Learning Culture and Social Interaction · doi:10.1016/j.lcsi.2020.100404 · Cited by 154
Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. Β. (2020) · Journal of the Learning Sciences · doi:10.1080/10508406.2020.1783269 · Cited by 130
Demszky, D., Liu, J., Hill, H. C., Jurafsky, D., & Piech, C. (2023) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737231169270 · Cited by 147
Metadata
- Authors
- Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K.
- Affiliations
- University of Pittsburgh (US)
- University of Memphis (US)
- California State University, Chico (US)
- University of Wisconsin–Madison (US)
- University of Colorado Boulder (US)
- Venue
- Educational Researcher
- Published
- 2018-06-29
- Volume
- 47
- Issue
- 7
- Pages
- 451-464
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2809787282
- Cited by
- 114
- References
- 80
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 21 · 2023: 19 · 2022: 10 · 2021: 18 · 2020: 10 · 2019: 9 · 2018: 2
- FWCI
- 12.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Funders
- Institute of Education Sciences (R305A130030)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- teacher questions, classroom discourse, teacher-student relationship, human coding, classroom observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
