The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Making Inferences About Teacher Observation Scores Over Time
Open accessAbstract
Observation protocol scores are commonly used as status measures to support inferences about teacher practices. When multiple observations are collected for the same teacher over the course of a year, some portion of a teacher’s score on each occasion may be attributable to the rater, lesson, and the time of year of the observation. All three of these are facets that can threaten the generalizability of teacher scores, but the role of time is easiest to overlook. A generalizability theory framework is used in this study to illustrate the concept of a hidden facet of measurement. When there are many temporally spaced observation occasions, it may be possible to support inferences about the growth in teaching practices over time as an alternative (or complement) to making inferences about status at a single point in time. This study uses longitudinal observation scores from the Measures of Effective Teaching project to estimate the reliability of teacher-level growth parameters for designs that vary in the number and spacing of observation occasions over a 2-year span. On the basis of a subsample of teachers scored using the Danielson Framework for Teaching, we show that at least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficients of .39.
Overview
CoChat Summary
TL;DR
The paper uses generalizability theory and longitudinal Measures of Effective Teaching (MET) observation data to show that the occasion/time of an observation is a hidden facet that can bias reliability estimates for teacher observation scores. Using Danielson Framework for Teaching scores, it finds that at least eight observations over 2 years are needed before teacher growth distinctions reach a reliability coefficient of .39.
Key findings
- At least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficient of .39, based on a subsample of teachers scored with the Danielson Framework for Teaching.
- Occasion/time is a hidden facet in observation protocol scores; ignoring it can bias generalizability/reliability estimates, and if occasion variance is mistakenly attributed to true teacher differences, reliability may be biased upward.
- Casabianca et al. (2015) found time trends accounted for a fairly small proportion of observed score variance but could still introduce significant bias: teaching practice scores showed a small linear decline over time, while rater judgments showed a “burn-in” trend with higher initial scores that rapidly adjusted downward and stabilized; the rater time trend impact was two to three times as large as the practice time trend.
- Ho and Kane’s (2013) MET supplementary study predicted generalizability coefficients of .59 for two lessons each scored by a different rater and .69 for four lessons with three unique raters, but occasions were fully confounded with lessons, so these may be overestimated due to hidden occasion facets.
Methods
- The study uses a generalizability theory framework to conceptualize occasion as a hidden facet and to compare reliability of score levels versus score growth.
- Data come from the MET project: longitudinal observation scores for a subsample of teachers scored using the Danielson Framework for Teaching, with up to 16 observations of unique lessons per teacher over two academic school years.
- The main analysis fits a two-level hierarchical linear model (HLM) with observation scores repeated at level 1 and teachers at level 2, estimating teacher-level growth parameters for designs varying in number and spacing of observation occasions over a 2-year span.
Limitations
- The paper states that the MET data are not ideal for disentangling variability in score levels attributable to occasions from variability attributable to lessons and raters at a single point in time, though they provide a proof of concept for estimating score growth.
- The paper states that outside Casabianca et al. (2015), no study of which the authors are aware has attempted to disentangle lesson variance from occasion variance, and the empirical size of hidden occasion bias remains mostly unexplored.
- Evident from the supplied text: the article is truncated after the introduction, so the final results, discussion, and any limitations stated there are not available for summary.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the basis of a subsample of teachers scored using the Danielson Framework for Teaching, we show that at least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficients of .39.”
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, in all of these studies, it can be argued that there is an important ''hidden'' facet to the measurement procedure involved in the scoring of a classroom observation. This facet is the occasion or, more specifically, the temporal location of the observation within the academic school year.”
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When multiple observations are collected for the same teacher over the course of a year, some portion of a teacher's score on each occasion may be attributable to the rater, lesson, and the time of year of the observation.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, the occasion of measurement may also introduce bias directly into teacher observation scores. This can happen if teaching practices are, in fact, getting better or worse over time.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If some teachers happen, by chance, to be more ''on'' or ''off'' on some days, weeks, or months of the year irrespective of the lesson they happen to be teaching, it will result in variability in scores that will be mistakenly attributed to true differences among teachers.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 18 claims and every source verdict.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 2
- The quote only partly states this claim. 1
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
Review of research in education
Shulman, L. S., Mowry, S., & Berliner, D. C. (1973) · F.E. Peacock eBooks · Cited by 7,754
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2633 · Cited by 1,717
Humphreys, L. G., & Lindquist, E. F. (1952) · The American Journal of Psychology · doi:10.2307/1418848 · Cited by 1,502
Pianta, R. C., & Hamre, B. K. (2009) · Educational Researcher · doi:10.3102/0013189x09332374 · Cited by 1,331
Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2593 · Cited by 1,241
A growth curve approach to the measurement of change.
Rogosa, D., Brandt, D., & Zimowski, M. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.3.726 · Cited by 1,081
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957
Research works printed but not linked: 5
- Cantrell S., 2013, Ensuring fair and reliable measures of effective teaching: Culminating findings from the MET project’s three-year study
- Ho A. D., 2013, The reliability of classroom observations by school personnel
- Kane T. J., 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
- Lord F. M., 1968, Statistical theories of mental test scores: Some la-tent trait models and their use in inferring an examinee’s ability
- Pianta R. C., 2006, Classroom assessment scoring system: Manual K-3 version
Cited by
Kure, A. E., Brevik, L. M., & Blikstad‐Balas, M. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12745 · Cited by 65
Weston, T. J., Hayward, C. N., & Laursen, S. L. (2021) · American Journal of Evaluation · doi:10.1177/1098214020931941 · Cited by 20
“As noted, this is a common practice in other Gstudies that examine the number of occasions classes have been observed (Briggs & Alzen, 2019; Hill et al., 2012).”
ResultMcDaniel, H. L., Braun, S. S., Bottiani, J. H., Lucia, D. D., Tolan, P. H., & Bradshaw, C. P. (2022) · School Psychology Review · doi:10.1080/2372966x.2022.2100275 · Cited by 22
“For example, Briggs and Alzen ( 2019 ) determined that it would be necessary to observe teachers eight times over the course of two years to reliably capture change in teacher practices over time.”
BackgroundTürken, S., Oppedal, B., Ali, W. A., & Adem, H. A. (2024) · Identity · doi:10.1080/15283488.2024.2373476 · Cited by 13
“To further increase the validity of the reported change in teacher competence, a direct measure such as classroom observations could be beneficial (e.g., Briggs & Alzen, 2019).”
Different Methods for Assessing Preservice Teachers’ Instruction: Why Measures Matter
Boguslav, A., & Cohen, J. (2023) · Journal of Teacher Education · doi:10.1177/00224871231200279 · Cited by 15
“This means that our estimates serve as an upper bound, as the true proportions would be lower if any unobserved factors influence scores (Briggs & Alzen, 2019).”
BackgroundMethodResultInfluential
Metadata
- Authors
- Briggs, D. C., & Alzen, J. L.
- Affiliations
- University of Colorado Boulder (US)
- Corresponding author
- Briggs, Derek C.
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2019-01-30
- Volume
- 79
- Issue
- 4
- Pages
- 636-664
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- PMC
- PMC7328239
- OpenAlex
- W2913142427
- Semantic Scholar
- 2fad97910a7842b4b6dd07c40e45ff2ef1f2c50d
- Cited by
- 14
- References
- 46
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 2
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational and Psychological Measurement (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Teacher Education and Assessments
- Social Sciences › Education › Educational Assessment and Improvement
- Keywords
- teacher professional development, longitudinal observation, generalizability theory, Measures of Effective Teaching, rater effects, time effects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Making Inferences About Teacher Observation Scores Over Time. https://app.cochat.ai/papers/10.1177/0013164419826237
- The paper
- Briggs, D. C., & Alzen, J. L. (2019). Making Inferences About Teacher Observation Scores Over Time. Educational and Psychological Measurement. https://doi.org/10.1177/0013164419826237
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.