The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Trends in Classroom Observation Scores
Open accessAbstract
Observations and ratings of classroom teaching and interactions collected over time are susceptible to trends in both the quality of instruction and rater behavior. These trends have potential implications for inferences about teaching and for study design. We use scores on the Classroom Assessment Scoring System-Secondary (CLASS-S) protocol from 458 middle school teachers over a 2-year period to study changes over time in (a) the average quality of teaching for the population of teachers, (b) the average severity of the population of raters, and (c) the severity of individual raters. To obtain these estimates and assess them in the context of other factors that contribute to the variability in scores, we develop an augmented G study model that is broadly applicable for modeling sources of variability in classroom observation ratings data collected over time. In our data, we found that trends in teaching quality were small. Rater drift was very large during raters' initial days of observation and persisted throughout nearly 2 years of scoring. Raters did not converge to a common level of severity; using our model we estimate that variability among raters actually increases over the course of the study. Variance decompositions based on the model find that trends are a modest source of variance relative to overall rater effects, rater errors on specific lessons, and residual error. The discussion provides possible explanations for trends and rater divergence as well as implicat…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Casabianca et al. (2015), the impact of the time trend on raters tended to be two to three times as large as the general impact of time on the practices being rated.
“Though both trends were significant, the impact of the time trend on raters tended to be two to three times as large as the general impact of time on the practices being rated.”
From Making Inferences About Teacher Observation Scores Over Time (2019) · cites this paper as Casabianca, Lockwood, and McCaffrey (2015)
Casabianca et al. (2015) found that main effect time trends are responsible for a fairly small proportion of observed score variance but can still introduce a significant source of bias.
“Casabianca et al. (2015) find that while these main effect trends are responsible for a fairly small proportion of observed score variance, they can still introduce a significant source of bias.”
From Making Inferences About Teacher Observation Scores Over Time (2019) · cites this paper as Casabianca, Lockwood, and McCaffrey (2015)
Casabianca et al. (2015) found that teaching practice scores showed evidence of a small linear decline over time.
“Somewhat counterintuitively, teaching practice scores showed evidence of a small linear decline over time.”
From Making Inferences About Teacher Observation Scores Over Time (2019) · cites this paper as Casabianca, Lockwood, and McCaffrey (2015)
Citations
Cites
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996 · cited 15×
“For Instructional Support, the rater trend accounted for more variance than the teacher or the lesson, but less than rater variance, the remaining rater-bylesson variance, or the residual variance.”
BackgroundMethodInfluential
Flagged first, then by citations
Research works printed but not linked: 31
- 10.1177/0013164410387344 · DOI
- Bell C. A., 2014, Designing Teacher Evaluation Systems: New Guidance from the Measures of Effective Teaching Project
- 10.3102/10769986013001001 · DOI
- 10.1007/978-1-4757-3456-0 · DOI · Paper page
- 10.1177/0013164413486987 · DOI
- 10.1111/j.1745-3984.2000.tb01081.x · DOI
- Danielson C., 2007, Enhancing professional practice: A framework for teaching
- 10.1002/j.2333-8504.2003.tb01893.x · DOI
- Gitomer D. H., Bell C. A., Qi Y., McCaffrey D. F., Hamre B. K., Pianta R. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6). Retrieved from http://www.tcrecord.org/library/abstract.asp?contentid=17460 · DOI
- Grossman P., Loeb S., Cohen J., Hammerness K., Wyckoff J., Boyd D., Lankford H. (2010). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores (NBER Working Paper No. 16015). Retrieved from http://www.nber.org/papers/w16015 · DOI
- 10.1111/j.1745-3984.2009.01068.x · DOI · Paper page
- 10.1007/978-0-387-84858-7 · DOI · Paper page
- 10.3102/0013189X12437203 · DOI · Paper page
- 10.1111/j.1745-3984.2001.tb01119.x · DOI · Paper page
- 10.1111/j.1744-6570.1997.tb01487.x · DOI
- Kingsbury F. A., 1922, Journal of Personnel Research, 1, 377
- 10.1080/00220671.1961.10882749 · DOI
- 10.1002/tea.20447 · DOI
- 10.1023/B:AHSE.0000012214.40340.03 · DOI
- McQueen J., 1997, Paper presented at the annual meeting of the American Educational Research Association
- 10.1080/10627197.2011.638884 · DOI
- 10.1111/j.1745-3984.2009.00088.x · DOI · Paper page
- Pianta R. C., 2007, Classroom Assessment Scoring System (CLASS), secondary manual
- Raudenbush S. W., 2002, Hierarchical linear models: Applications and data analysis methods
- 10.1086/428812 · DOI
- 10.1037/0033-2909.88.2.413 · DOI
- 10.1214/aos/1176344136 · DOI · Paper page
- Searle S. R., 1971, Linear models
- 10.1037/h0071663 · DOI · Paper page
- Wang Z., 2014, Monitoring of scoring using e–rater® and human raters on a writing test
- Wilson M., 2000, Objective measurement: Theory into practice, 5, 113
Cited by
Sorted by citations
Classroom observation systems in context: A case for the validation of observation systems
Liu, S., Bell, C. A., Jones, N. D., & McCaffrey, D. F. (2019) · Educational Assessment, Evaluation and Accountability · doi:10.1007/s11092-018-09291-3 · Cited by 46
Self-citationRater Performance Standards for Classroom Observation Instruments
White, M. C. (2018) · Educational Researcher · doi:10.3102/0013189x18785623 · Cited by 32
Using Classroom Observations in the Evaluation of Special Education Teachers
Jones, N. D., Bell, C. A., Brownell, M., Qi, Y., Peyton, D., Pua, D., Fowler, M., & Holtzman, S. (2022) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211068523 · Cited by 21
Making Inferences About Teacher Observation Scores Over Time
Briggs, D. C., & Alzen, J. L. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164419826237 · Cited by 14
Metadata
- Authors
- Casabianca, J. M., Lockwood, J. R., & McCaffrey, D. F.
- Affiliations
- The University of Texas at Austin (US)
- Educational Testing Service (US)
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2014-06-22
- Volume
- 75
- Issue
- 2
- Pages
- 311-337
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- PMC
- PMC5965595
- OpenAlex
- W2055440896
- Cited by
- 105
- References
- 1
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 7 · 2023: 6 · 2022: 13 · 2021: 14 · 2020: 14 · 2019: 17 · 2018: 13 · 2017: 4 · 2016: 4 · 2015: 1 · 2014: 1
- FWCI
- 3.27
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Educational and Psychological Measurement (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- classroom observation, rater severity, Classroom Assessment Scoring System, class S, junior high school teachers, rater variability, variance decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
