Cited by 114Online Learning and AnalyticsComputational and Text Analysis MethodsEvaluation of Teaching Practices

Automatically Measuring Question Authenticity in Real-World Classrooms

Kelly, Sean, Olney, Andrew M., Donnelly, Patrick, Nystrand, Martin, D’Mello, Sidney K.·2018·Educational Researcher

Abstract

Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human-coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human-coded and computer-coded authenticity at the classroom level were sufficiently high ( r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.

Integrity signals

Registry

Crossref · 10.3102/0013189x18785613

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment.

    “e previous study combined speech recognition, natural language processing, and machine learning to measure the quality of classroom talk, in which new forms of interaction were created to provoke thoughts and further shape the effective interaction of the learning environment [37].”
  • In 2018, one empirical study was published and it focused the deep learning technology on the modelling of scoring-based data.

    “In 2018, one empirical study [37] was published and it focused the deep learning technology on the modelling of scoring-based data.”

Cites

Flagged first, then by citations

5 of the 80 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K.
Affiliations
  • University of Pittsburgh (US)
  • University of Memphis (US)
  • California State University, Chico (US)
  • University of Wisconsin–Madison (US)
  • University of Colorado Boulder (US)
Venue
Educational Researcher
Published
2018-06-29
Volume
47
Issue
7
Pages
451-464
ISSN
  • 0013-189X (ISSN-L)
  • 1935-102X
OpenAlex
W2809787282
Cited by
114
References
80
Citations by year
2026: 8 · 2025: 17 · 2024: 21 · 2023: 19 · 2022: 10 · 2021: 18 · 2020: 10 · 2019: 9 · 2018: 2
FWCI
12.32
Type
journal article
Language
en
Open access
closed
Locations
Educational Researcher (publishedVersion)
Funders
Institute of Education Sciences (R305A130030)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Education › Evaluation of Teaching Practices
Keywords
teacher questions, classroom discourse, teacher-student relationship, human coding, classroom observation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review