CoChat
Sign Up
Cited by 115Online Learning and AnalyticsComputational and Text Analysis MethodsEvaluation of Teaching Practices

The DOI matches its Crossref record; 75 of the 80 references it cites were found in a registry or an index.

Automatically Measuring Question Authenticity in Real-World Classrooms

Kelly, Sean, Olney, Andrew M., Donnelly, Patrick, Nystrand, Martin, D’Mello, Sidney K.·2018·Educational Researcher
DOI 10.3102/0013189x18785613
Open this paper in CoChat — free account

Abstract

Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human-coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human-coded and computer-coded authenticity at the classroom level were sufficiently high ( r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.

Overview

Integrity signals

Registry

Crossref · 10.3102/0013189x18785613

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 80 references

Research works printed but not linked: 17

  1. Burke J., 2010, What’s the big idea? Question-driven units to motivate reading, writing, and thinking
  2. Ford M., 2008, The LENA language environment analysis system
  3. Gamoran A., 2003, Stability and change in American education: Structure, process, and outcomes, 109
  4. Grossier P., 1964, How to use the fine art of questioning
  5. Hamilton L., 2012, Assessing teacher quality: Understanding teacher effects on instruction and achievement, 49
  6. Hardt M., 2016, Advances in neural information processing systems, 3315
  7. Inter-University Consortium for Political and Social Research (ICPSR), 2014, Measures of effective teaching: Observation measures report
  8. Kelly S., 2018, Teacher Education Quarterly, 45, 57
  9. Measures of Effective Teaching Project, 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
  10. Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom
  11. Nystrand M., 1997, Opening dialogue: Understanding the dynamics of language and learning in the English classroom, 30
  12. Olney A. M., 2017, Design recommendations for intelligent tutoring systems: Assessment, 5, 261
  13. Olney A. M., 2017, Proceedings of the 10th international conference on educational data mining, 162
  14. Rosemary C. A., 2002, 51, 368
  15. Samei B., 2015, Proceedings of the 8th international conference on educational data mining (EDM 2015), 444
  16. Wang Z., 2013, Unterrichtswissenschaft, 4, 290
  17. Wei R. C., 2010, Teacher assessment and the quest for teacher quality, 691

Cited by

Sorted by citations

Items 101–115 of 115

Related

See all

Metadata

Authors
Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & D’Mello, S. K.
Affiliations
  • University of Pittsburgh (US)
  • University of Memphis (US)
  • California State University, Chico (US)
  • University of Wisconsin–Madison (US)
  • University of Colorado Boulder (US)
ORCID
Venue
Educational Researcher
Publisher
American Educational Research Association (AERA)
Published
2018-06-29
Volume
47
Issue
7
Pages
451-464
ISSN
  • 0013-189X (ISSN-L)
  • 1935-102X
DOI
10.3102/0013189x18785613
OpenAlex
W2809787282
Cited by
114
References
80
Citations by year
2026: 8 · 2025: 17 · 2024: 21 · 2023: 19 · 2022: 10 · 2021: 18 · 2020: 10 · 2019: 9 · 2018: 2
FWCI
12.32
Type
journal article
Language
en
Open access
closed
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
Educational Researcher (publishedVersion)
Funders
Institute of Education Sciences (R305A130030)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Education › Evaluation of Teaching Practices
Keywords
teacher questions, classroom discourse, teacher-student relationship, human coding, classroom observation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review