Making Inferences About Teacher Observation Scores Over Time
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Abstract
Observation protocol scores are commonly used as status measures to support inferences about teacher practices. When multiple observations are collected for the same teacher over the course of a year, some portion of a teacher’s score on each occasion may be attributable to the rater, lesson, and the time of year of the observation. All three of these are facets that can threaten the generalizability of teacher scores, but the role of time is easiest to overlook. A generalizability theory framework is used in this study to illustrate the concept of a hidden facet of measurement. When there are many temporally spaced observation occasions, it may be possible to support inferences about the growth in teaching practices over time as an alternative (or complement) to making inferences about status at a single point in time. This study uses longitudinal observation scores from the Measures of Effective Teaching project to estimate the reliability of teacher-level growth parameters for designs that vary in the number and spacing of observation occasions over a 2-year span. On the basis of a subsample of teachers scored using the Danielson Framework for Teaching, we show that at least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficients of .39.
Overview
CoChat Summary
TL;DR
The paper uses generalizability theory and longitudinal Measures of Effective Teaching (MET) observation data to show that the occasion/time of an observation is a hidden facet that can bias reliability estimates for teacher observation scores. Using Danielson Framework for Teaching scores, it finds that at least eight observations over 2 years are needed before teacher growth distinctions reach a reliability coefficient of .39.
Key findings
- At least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficient of .39, based on a subsample of teachers scored with the Danielson Framework for Teaching.
- Occasion/time is a hidden facet in observation protocol scores; ignoring it can bias generalizability/reliability estimates, and if occasion variance is mistakenly attributed to true teacher differences, reliability may be biased upward.
- Casabianca et al. (2015) found time trends accounted for a fairly small proportion of observed score variance but could still introduce significant bias: teaching practice scores showed a small linear decline over time, while rater judgments showed a “burn-in” trend with higher initial scores that rapidly adjusted downward and stabilized; the rater time trend impact was two to three times as large as the practice time trend.
- Ho and Kane’s (2013) MET supplementary study predicted generalizability coefficients of .59 for two lessons each scored by a different rater and .69 for four lessons with three unique raters, but occasions were fully confounded with lessons, so these may be overestimated due to hidden occasion facets.
Methods
- The study uses a generalizability theory framework to conceptualize occasion as a hidden facet and to compare reliability of score levels versus score growth.
- Data come from the MET project: longitudinal observation scores for a subsample of teachers scored using the Danielson Framework for Teaching, with up to 16 observations of unique lessons per teacher over two academic school years.
- The main analysis fits a two-level hierarchical linear model (HLM) with observation scores repeated at level 1 and teachers at level 2, estimating teacher-level growth parameters for designs varying in number and spacing of observation occasions over a 2-year span.
Limitations
- The paper states that the MET data are not ideal for disentangling variability in score levels attributable to occasions from variability attributable to lessons and raters at a single point in time, though they provide a proof of concept for estimating score growth.
- The paper states that outside Casabianca et al. (2015), no study of which the authors are aware has attempted to disentangle lesson variance from occasion variance, and the empirical size of hidden occasion bias remains mostly unexplored.
- Evident from the supplied text: the article is truncated after the introduction, so the final results, discussion, and any limitations stated there are not available for summary.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the basis of a subsample of teachers scored using the Danielson Framework for Teaching, we show that at least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficients of .39.”
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, in all of these studies, it can be argued that there is an important ''hidden'' facet to the measurement procedure involved in the scoring of a classroom observation. This facet is the occasion or, more specifically, the temporal location of the observation within the academic school year.”
What the sources say
- Not checked — no full text of the sourcecites Bell et al., 2012 matched to “An Argument Approach to Observation Protocol Validity” (2012) source
- Not matchedcites Casabianca, McCaffrey, et al., 2013
More than one paper fits this citation.
- Not checked — no full text of the sourcecites Charalambous, Kyriakides, Tsangaridou, & Kyriakides, 2017 matched to “Exploring the reliability of generic and content-specific instructional aspects in physical education lessons” (2017) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When multiple observations are collected for the same teacher over the course of a year, some portion of a teacher's score on each occasion may be attributable to the rater, lesson, and the time of year of the observation.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, the occasion of measurement may also introduce bias directly into teacher observation scores. This can happen if teaching practices are, in fact, getting better or worse over time.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If some teachers happen, by chance, to be more ''on'' or ''off'' on some days, weeks, or months of the year irrespective of the lesson they happen to be teaching, it will result in variability in scores that will be mistakenly attributed to true differences among teachers.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 18 claims and every source verdict.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 2
- The quote only partly states this claim. 1
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
Review of research in education
Shulman, L. S., Mowry, S., & Berliner, D. C. (1973) · F.E. Peacock eBooks · Cited by 7,754
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
Merrifield, P. R., Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1974) · American Educational Research Journal · doi:10.2307/1162145 · Cited by 1,819
Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2633 · Cited by 1,717
Humphreys, L. G., & Lindquist, E. F. (1952) · The American Journal of Psychology · doi:10.2307/1418848 · Cited by 1,502
Pianta, R. C., & Hamre, B. K. (2009) · Educational Researcher · doi:10.3102/0013189x09332374 · Cited by 1,331
Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014) · American Economic Review · doi:10.1257/aer.104.9.2593 · Cited by 1,241
A growth curve approach to the measurement of change.
Rogosa, D., Brandt, D., & Zimowski, M. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.3.726 · Cited by 1,081
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957
Generalizations about Using Value-Added Measures of Teacher Quality
Hanushek, E. A., & Rivkin, S. G. (2010) · American Economic Review · doi:10.1257/aer.100.2.267 · Cited by 793
Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement*
Rothstein, J. (2010) · Quarterly Journal of Economics · doi:10.1162/qjec.2010.125.1.175 · Cited by 735
Approaches to Evaluating Teacher Effectiveness: A Research Synthesis.
Goe, L., Bell, C. A., & Little, O. M. (2008) · Cited by 673
Teacher and Teaching Effects on Students’ Attitudes and Behaviors
Blazar, D., & Kraft, M. A. (2017) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373716670260 · Cited by 591
Sanders, W. L., & Horn, S. P. (1998) · Journal of Personnel Evaluation in Education · doi:10.1023/a:1008067210518 · Cited by 590
The art and science of teaching: a comprehensive framework for effective instruction
(2008) · Choice Reviews Online · doi:10.5860/choice.45-6316 · Cited by 589
Chapter 9: Questions and Answers in the Measurement of Change
Willett, J. B. (1988) · Review of Research in Education · doi:10.3102/0091732x015001345 · Cited by 565
Evaluating Value-Added Models for Teacher Accountability
McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003) · PsycEXTRA Dataset · doi:10.1037/e658712010-001 · Cited by 541
The Framework for Teaching Evaluation Instrument
Danielson, C. (2011) · Cited by 476
Making Content Comprehensible for English Learners: The SIOP Model
Poole, A. (2004) · TESL reporter · Cited by 460
Questions and Answers in the Measurement of Change
Willett, J. B. (1988) · Review of Research in Education · doi:10.2307/1167368 · Cited by 454
The Intertemporal Variability of Teacher Effect Estimates
McCaffrey, D. F., Sass, T. R., Lockwood, J. R., & Mihaly, K. (2009) · Education Finance and Policy · doi:10.1162/edfp.2009.4.4.572 · Cited by 331
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012) · Educational Researcher · doi:10.3102/0013189x12437203 · Cited by 331
One lesson is all you need? Stability of instructional quality across lessons
Praetorius, A.-K., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E. (2014) · Learning and Instruction · doi:10.1016/j.learninstruc.2013.12.002 · Cited by 313
Grossman, P., Loeb, S., Cohen, J., & Wyckoff, J. (2013) · American Journal of Education · doi:10.1086/669901 · Cited by 271
An Argument Approach to Observation Protocol Validity
Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012) · Educational Assessment · doi:10.1080/10627197.2012.715014 · Cited by 236
Building a More Complete Understanding of Teacher Evaluation Using Classroom Observations
Cohen, J., & Goldhaber, D. (2016) · Educational Researcher · doi:10.3102/0013189x16659442 · Cited by 201
Ballou, D., & Springer, M. G. (2015) · Educational Researcher · doi:10.3102/0013189x15574904 · Cited by 186
Classroom Assessment Scoring System (CLASS). Manual K-3.
Pianta, R. C., Paro, K. M. L., & Hamre, B. K. (2008) · doi:10.14507/er.v0.984 · Cited by 177
Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure?
Millman, J. (1997) · Medical Entomology and Zoology · Cited by 130
Is it Just a Bad Class? Assessing the Long‐term Stability of Estimated Teacher Performance
Goldhaber, D., & Hansen, M. (2013) · Economica · doi:10.1111/ecca.12002 · Cited by 111
Trends in Classroom Observation Scores
Casabianca, J. M., Lockwood, J. R., & McCaffrey, D. F. (2015) · Educational and Psychological Measurement · doi:10.1177/0013164414539163 · Cited by 105
Harris, D. N. (2009) · Education Finance and Policy · doi:10.1162/edfp.2009.4.4.319 · Cited by 100
Goldhaber, D. (2015) · Educational Researcher · doi:10.3102/0013189x15574905 · Cited by 96
Performance Incentives: Their Growing Impact on American K-12 Education
Springer, M. G. (2009) · Project Muse (Johns Hopkins University) · Cited by 90
Effect of Observation Mode on Measures of Secondary Mathematics Teaching
Casabianca, J. M., McCaffrey, D. F., Gitomer, D. H., Bell, C. A., Hamre, B. K., & Pianta, R. C. (2013) · Educational and Psychological Measurement · doi:10.1177/0013164413486987 · Cited by 80
Value-Added Modeling and Educational Accountability: Are We Answering the Real Questions?
Everson, K. C. (2017) · Review of Educational Research · doi:10.3102/0034654316637199 · Cited by 68
Mashburn, A. J., Meyer, J. P., Allen, J. P., & Pianta, R. C. (2014) · Educational and Psychological Measurement · doi:10.1177/0013164413515882 · Cited by 65
Hoffman, J. V., Sailors, M., Duffy, G. G., et al. (2004) · Journal of Literacy Research · doi:10.1207/s15548430jlr3603_3 · Cited by 47
Bacher-Hicks, A., Chin, M. J., Kane, T. J., & Staiger, D. O. (2017) · National Bureau of Economic Research · doi:10.3386/w23478 · Cited by 44
Stacy, B. W., Guarino, C. M., Reckase, M. D., & Wooldridge, J. M. (2013) · Economics of Education Review · doi:10.1016/j.econedurev.2018.04.001 · Cited by 44
Lord, F. M. (1956) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1956.tb00058.x · Cited by 32
Getting Value Out of Value-Added
National Research Council (2010) · National Academies Press eBooks · doi:10.17226/12820 · Cited by 27
Teacher and Teaching Effects on Students' Academic Behaviors and MindsetsPreprint
Blazar, D., & Kraft, M. A. (2015) · RePEc: Research Papers in Economics · Cited by 11
Charalambous, C. Y., Kyriakides, E., Tsangaridou, N., & Kyriakides, L. (2017) · School Effectiveness and School Improvement · doi:10.1080/09243453.2017.1311929 · Cited by 11
Measures of Effective Teaching: 1 - Study Information
Bill And Melinda Gates Foundation (2013) · ICPSR Data Holdings · doi:10.3886/icpsr34771 · Cited by 6
Multifacet Universes of Generalization and D Study Designs
Brennan, R. L. (2001) · Springer eBooks · doi:10.1007/978-1-4757-3456-0_4 · Cited by 3
Measures of Effective Teaching: 1 - Study Information
Bill And Melinda Gates Foundation (2013) · ICPSR Data Holdings · doi:10.3886/icpsr34771.v2 · Cited by 3
Research works printed but not linked: 5
- Cantrell S., 2013, Ensuring fair and reliable measures of effective teaching: Culminating findings from the MET project’s three-year study
- Ho A. D., 2013, The reliability of classroom observations by school personnel
- Kane T. J., 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
- Lord F. M., 1968, Statistical theories of mental test scores: Some la-tent trait models and their use in inferring an examinee’s ability
- Pianta R. C., 2006, Classroom assessment scoring system: Manual K-3 version
Cited by
Kure, A. E., Brevik, L. M., & Blikstad‐Balas, M. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12745 · Cited by 66
Weston, T. J., Hayward, C. N., & Laursen, S. L. (2021) · American Journal of Evaluation · doi:10.1177/1098214020931941 · Cited by 20
“As noted, this is a common practice in other Gstudies that examine the number of occasions classes have been observed (Briggs & Alzen, 2019; Hill et al., 2012).”
ResultMcDaniel, H. L., Braun, S. S., Bottiani, J. H., Lucia, D. D., Tolan, P. H., & Bradshaw, C. P. (2022) · School Psychology Review · doi:10.1080/2372966x.2022.2100275 · Cited by 22
“For example, Briggs and Alzen ( 2019 ) determined that it would be necessary to observe teachers eight times over the course of two years to reliably capture change in teacher practices over time.”
BackgroundTürken, S., Oppedal, B., Ali, W. A., & Adem, H. A. (2024) · Identity · doi:10.1080/15283488.2024.2373476 · Cited by 13
“To further increase the validity of the reported change in teacher competence, a direct measure such as classroom observations could be beneficial (e.g., Briggs & Alzen, 2019).”
Different Methods for Assessing Preservice Teachers’ Instruction: Why Measures Matter
Boguslav, A., & Cohen, J. (2023) · Journal of Teacher Education · doi:10.1177/00224871231200279 · Cited by 15
“This means that our estimates serve as an upper bound, as the true proportions would be lower if any unobserved factors influence scores (Briggs & Alzen, 2019).”
BackgroundMethodResultInfluential
Metadata
- Authors
- Briggs, D. C., & Alzen, J. L.
- Affiliations
- University of Colorado Boulder (US)
- Corresponding author
- Briggs, Derek C.
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2019-01-30
- Volume
- 79
- Issue
- 4
- Pages
- 636-664
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- PMC
- PMC7328239
- OpenAlex
- W2913142427
- Semantic Scholar
- 2fad97910a7842b4b6dd07c40e45ff2ef1f2c50d
- Cited by
- 14
- References
- 46
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 2
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational and Psychological Measurement (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Teacher Education and Assessments
- Social Sciences › Education › Educational Assessment and Improvement
- Keywords
- teacher professional development, longitudinal observation, generalizability theory, Measures of Effective Teaching, rater effects, time effects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.