The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental study that manipulated the length of observation and order of presentation of 40-minute videotaped lessons from secondary grade classrooms. Results indicate that two 20-minute observation segments presented in random order produce the most desirable effect on score reliability and validity. This suggests that 20-minute occasions may be sufficient time for a rater to observe true characteristics of teaching quality assessed by the measure used in the study, and randomizing the order in which segments were rated may reduce construct irrelevant variance arising from carry over effects and rater drift.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Memory: A Contribution to Experimental Psychology
Ebbinghaus, H. (2013) · Annals of Neurosciences · doi:10.5214/ans.0972.7531.200408 · Cited by 2,827
Memory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996 · cited 4×
“For example, two content-specific measures are the Mathematical Quality of Instruction (MQI; Hill et al., 2008) instrument that assesses five domains of teaching mathematics to students in kindergarten through 8th grade classrooms, and the Protocol for Language Arts Teaching Mashburn et al.”
BackgroundInfluentialAn Interaction-Based Approach to Enhancing Secondary School Instruction and Student Achievement
Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011) · Science · doi:10.1126/science.1207998 · Cited by 590
Self-citationHill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012) · Educational Researcher · doi:10.3102/0013189x12437203 · Cited by 331 · cited 1×
“Arguments for this design instead of a crossed design may be found in Hill, Charalambous, and Kraft (2012) and Erlich and Borich (1979).”
BackgroundAn Argument Approach to Observation Protocol Validity
Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012) · Educational Assessment · doi:10.1080/10627197.2012.715014 · Cited by 236
Self-citation
Cited by
Reimsbach, D., Hahn, R., & Gürtürk, A. (2018) · European Accounting Review · doi:10.1080/09638180.2016.1273787 · Cited by 304
Closing the Racial Discipline Gap in Classrooms by Changing Teacher Practice
Gregory, A., Hafen, C. A., Ruzek, E., Mikami, A. Y., Allen, J. P., & Pianta, R. C. (2016) · School Psychology Review · doi:10.17105/spr45-2.171-191 · Cited by 272
Self-citationSchiefele, U., & Schaffner, E. (2015) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2015.06.005 · Cited by 232
Allen, J. P., Hafen, C. A., Gregory, A. C., Mikami, A. Y., & Pianta, R. (2015) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2015.1017680 · Cited by 92
Self-citationMaking Inferences About Teacher Observation Scores Over Time
Briggs, D. C., & Alzen, J. L. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164419826237 · Cited by 14
Metadata
- Authors
- Mashburn, A. J., Meyer, J. P., Allen, J. P., & Pianta, R. C.
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2013-12-26
- Volume
- 74
- Issue
- 3
- Pages
- 400-422
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- OpenAlex
- W2066901843
- Cited by
- 65
- References
- 6
- FWCI
- 17.56
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality. https://app.cochat.ai/papers/10.1177/0013164413515882
- The paper
- Mashburn, A. J., Meyer, J. P., Allen, J. P., & Pianta, R. C. (2014). The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality. Educational and Psychological Measurement. https://doi.org/10.1177/0013164413515882
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.