Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.

Making Inferences About Teacher Observation Scores Over Time

Open access
Briggs, Derek C., Alzen, Jessica L. · 2019 · Educational and Psychological Measurement

Abstract

Observation protocol scores are commonly used as status measures to support inferences about teacher practices. When multiple observations are collected for the same teacher over the course of a year, some portion of a teacher’s score on each occasion may be attributable to the rater, lesson, and the time of year of the observation. All three of these are facets that can threaten the generalizability of teacher scores, but the role of time is easiest to overlook. A generalizability theory framework is used in this study to illustrate the concept of a hidden facet of measurement. When there are many temporally spaced observation occasions, it may be possible to support inferences about the growth in teaching practices over time as an alternative (or complement) to making inferences about status at a single point in time. This study uses longitudinal observation scores from the Measures of Effective Teaching project to estimate the reliability of teacher-level growth parameters for designs that vary in the number and spacing of observation occasions over a 2-year span. On the basis of a subsample of teachers scored using the Danielson Framework for Teaching, we show that at least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficients of .39.

Overview

CoChat Summary

TL;DR

The paper uses generalizability theory and longitudinal Measures of Effective Teaching (MET) observation data to show that the occasion/time of an observation is a hidden facet that can bias reliability estimates for teacher observation scores. Using Danielson Framework for Teaching scores, it finds that at least eight observations over 2 years are needed before teacher growth distinctions reach a reliability coefficient of .39.

Key findings

  • At least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficient of .39, based on a subsample of teachers scored with the Danielson Framework for Teaching.
  • Occasion/time is a hidden facet in observation protocol scores; ignoring it can bias generalizability/reliability estimates, and if occasion variance is mistakenly attributed to true teacher differences, reliability may be biased upward.
  • Casabianca et al. (2015) found time trends accounted for a fairly small proportion of observed score variance but could still introduce significant bias: teaching practice scores showed a small linear decline over time, while rater judgments showed a “burn-in” trend with higher initial scores that rapidly adjusted downward and stabilized; the rater time trend impact was two to three times as large as the practice time trend.
  • Ho and Kane’s (2013) MET supplementary study predicted generalizability coefficients of .59 for two lessons each scored by a different rater and .69 for four lessons with three unique raters, but occasions were fully confounded with lessons, so these may be overestimated due to hidden occasion facets.

Methods

  • The study uses a generalizability theory framework to conceptualize occasion as a hidden facet and to compare reliability of score levels versus score growth.
  • Data come from the MET project: longitudinal observation scores for a subsample of teachers scored using the Danielson Framework for Teaching, with up to 16 observations of unique lessons per teacher over two academic school years.
  • The main analysis fits a two-level hierarchical linear model (HLM) with observation scores repeated at level 1 and teachers at level 2, estimating teacher-level growth parameters for designs varying in number and spacing of observation occasions over a 2-year span.

Limitations

  • The paper states that the MET data are not ideal for disentangling variability in score levels attributable to occasions from variability attributable to lessons and raters at a single point in time, though they provide a proof of concept for estimating score growth.
  • The paper states that outside Casabianca et al. (2015), no study of which the authors are aware has attempted to disentangle lesson variance from occasion variance, and the empirical size of hidden occasion bias remains mostly unexplored.
  • Evident from the supplied text: the article is truncated after the introduction, so the final results, discussion, and any limitations stated there are not available for summary.

Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/0013164419826237

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations

3 more claims

Create a free account to see all 18 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 8
  • It may state two separate claims as one. 2
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 46 references

Research works printed but not linked: 5

  1. Cantrell S., 2013, Ensuring fair and reliable measures of effective teaching: Culminating findings from the MET project’s three-year study
  2. Ho A. D., 2013, The reliability of classroom observations by school personnel
  3. Kane T. J., 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
  4. Lord F. M., 1968, Statistical theories of mental test scores: Some la-tent trait models and their use in inferring an examinee’s ability
  5. Pianta R. C., 2006, Classroom assessment scoring system: Manual K-3 version

Cited by

See all 14

Metadata

Authors
Briggs, D. C., & Alzen, J. L.
Affiliations
University of Colorado Boulder (US)
Corresponding author
Briggs, Derek C.
Venue
Educational and Psychological Measurement
Published
2019-01-30
Volume
79
Issue
4
Pages
636-664
ISSN
  • 0013-1644 (ISSN-L)
  • 1552-3888
OpenAlex
W2913142427
Cited by
14
References
46
Citations by year
2026: 4 · 2025: 2 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 2
FWCI
0.72
Type
journal article
Language
en
Open access
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Educational and Psychological Measurement (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Education › Teacher Education and Assessments
  • Social Sciences › Education › Educational Assessment and Improvement
Keywords
teacher professional development, longitudinal observation, generalizability theory, Measures of Effective Teaching, rater effects, time effects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Making Inferences About Teacher Observation Scores Over Time. https://app.cochat.ai/papers/10.1177/0013164419826237
The paper
Briggs, D. C., & Alzen, J. L. (2019). Making Inferences About Teacher Observation Scores Over Time. Educational and Psychological Measurement. https://doi.org/10.1177/0013164419826237

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.