Classroom Composition and Measured Teacher Performance: What Do Teacher Observation Scores Really Measure?
Open accessThe DOI matches its Crossref record; 34 of the 38 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.
Abstract
As states and districts implement more rigorous teacher evaluation systems, measures of teacher performance are increasingly being used to support instruction and inform retention decisions. Classroom observations take a central role in these systems, accounting for the majority of teacher ratings upon which accountability decisions are based. Using data from the Measures of Effective Teaching study, we explore the extent to which classroom composition influences measured teacher performance based on classroom observation scores. The context in which teachers work—most notably, the incoming academic performance of their students—plays a critical role in determining teachers’ measured performance. Furthermore, the intentional sorting of teachers to students has a significant influence on measured performance. Implications for high-stakes teacher accountability policies are discussed.
Overview
CoChat Summary
TL;DR
Using Measures of Effective Teaching (MET) data, the paper finds that teachers’ observation-based performance scores rise when they are assigned higher-achieving students, even after accounting for teacher fixed effects. It also shows that nonrandom teacher-class matching can bias these observation scores, raising concerns about high-stakes evaluation systems that rely heavily on them.
Key findings
- In the 2009–2010 school year, ELA teachers were “more than twice as likely” to be rated in the top performance quintile when assigned the highest-achieving students versus the lowest-achieving students; math teachers were “more than 6 times as likely.”
- Roughly half of teachers assigned the highest-performing students were rated in the top two quintiles: 48% in ELA and 54% in math; among teachers assigned the lowest-performing students, the share was 37% in ELA and 18% in math.
- In the full sample, a 1 standard deviation increase in incoming achievement was associated with a 0.11 FFT-point increase in ELA measured performance and a 0.06 FFT-point increase in math in the main estimate set; in the IV sample, the paper reports 0.14 FFT points for ELA and 0.08 FFT points for math.
- For ELA teachers in partial-compliance classes, the OLS estimate was 0.21 FFT points and the IV estimate was 0.17 FFT points, both above the FE estimate of 0.14 FFT points, which the authors interpret as upward bias from nonrandom matching.
- For specific practices, higher-achieving students were associated with better ratings on ELA instructional components 3a and 3c, while measures tied more to questioning and assessment (3b and 3d) were not sensitive to incoming achievement; math practices were largely unrelated to incoming achievement except for component 2b.
Methods
- The study uses data from the MET project over two school years (2009–2010 and 2010–2011) across six districts, with an analytic teacher sample of 834 Grade 4–9 teachers (530 ELA, 480 math, with overlap).
- Teacher performance is measured using classroom observation scores from the Danielson Framework for Teaching (FFT), aggregating eight observed components from video-recorded lessons.
- The main analysis combines teacher fixed-effects models, OLS, and IV comparisons, and uses the Year 2 random assignment sample to assess causal effects and detect bias from nonrandom teacher-class matching.
Limitations
- The paper states that classroom composition data were available only for teachers’ actual classes, not for the class rosters to which teachers were randomly assigned, and that student data for non-MET teachers after randomization were not observed.
- The authors note that the FE approach can still be biased if there are time-varying shocks to teacher-class assignment, and the IV approach can be biased if there is partial/full noncompliance or selection into compliance classes.
- Evident from the text: the experimental ELA result is described as “not precisely estimated,” and the authors say the math randomization sample did not satisfy the covariate-balance test, so the experimental estimate for math is not treated as a valid causal effect.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 10 Not checked · 104 Own results and interpretations
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, we offer evidence that the intentional, nonrandom matching of teachers to classrooms due to unobserved time-invariant teacher effects biases the relationship between incoming student achievement and observed classroom performance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students' incoming achievement is more strongly associated with observed English Language Art (ELA) instruction than math instruction.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that teacher performance, based on classroom observation, is significantly influenced by the context in which teachers work.”
- Also stated in: § Rachel Garrett · boosted
“The context in which teachers work—most notably, the incoming academic performance of their students—plays a critical role in determining teachers' measured performance.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Rachel Garrett · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the influence of incoming achievement is concentrated among subject specialists, who work with multiple classes of students within a single school day, compared with classroom generalist teachers who work with the same classroom of students all day, every day”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we are unable to uniquely attribute these differences to either departmentalized (compared with generalist) class status or grade-level differences.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
70 more claims
Create a free account to see all 116 claims and every source verdict.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 23
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Incoming Achievement and Classroom Composition
“| Teachers | 180 | 350 | 177 | 303 |”
§ Teacher Sample
“The teacher-level analytic sample consists of 834 teachers in Grades 4 to 9, of which 354 taught ELA, 304 taught math, and 176 taught both subjects.”
Citations
Cites
Teachers, Schools, and Academic Achievement
Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00584.x · Cited by 4,501
The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data
Rockoff, J. E. (2004) · American Economic Review · doi:10.1257/0002828041302244 · Cited by 2,432 · cited 2×
“Revisions to teacher evaluation systems and the measures upon which teacher evaluation is based have been spurred by evidence in support of the critical role teachers play in improving student outcomes (Goldhaber, 2002; Rivkin, Hanushek, & Kain, 2005; Rockoff, 2004), while also acknowledging both the substantial withi…”
BackgroundTeachers and Student Achievement in the Chicago Public High Schools
Aaronson, D., Barrow, L., & Sander, W. (2007) · Journal of Labor Economics · doi:10.1086/508733 · Cited by 1,291
The Common Core State Standards in Mathematics
McCallum, W. (2015) · Selected Regular Lectures from the 12th International Congress on Mathematical Education · doi:10.1007/978-3-319-17187-6_31 · Cited by 770
American Economic Journal: Applied Economics
Duflo, E. (2009) · American Economic Review · doi:10.1257/aer.99.2.676 · Cited by 770
Teacher-Student Matching and the Assessment of Teacher Effectiveness
Clotfelter, C. T., Ladd, H. F., & Vigdor, J. L. (2006) · Journal of Human Resources · doi:10.3368/jhr.xli.4.778 · Cited by 739 · cited 3×
“…randomly assigned to classes of students.1 In principle, the random assignment of teachers to classes disrupts the assortative matching process that can confound measures of teacher effectiveness (Clotfelter et al., 2006; Garrett & Steinberg, 2015; Rothstein, 2009, 2010; Whitehurst et al., 2014).”
BackgroundTeacher Quality in Educational Production: Tracking, Decay, and Student Achievement*
Rothstein, J. (2010) · Quarterly Journal of Economics · doi:10.1162/qjec.2010.125.1.175 · Cited by 735 · cited 3×
“…randomly assigned to classes of students.1 In principle, the random assignment of teachers to classes disrupts the assortative matching process that can confound measures of teacher effectiveness (Clotfelter et al., 2006; Garrett & Steinberg, 2015; Rothstein, 2009, 2010; Whitehurst et al., 2014).”
Background(2007) · The Elementary School Journal · doi:10.1086/518627 · Cited by 642
(2009) · Issue Lab (Candid) · Cited by 552
Kraft, M. A., & Papay, J. P. (2014) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713519496 · Cited by 517 · cited 2×
“For example, Kraft and Papay (2014) find that school context—including supportive and collaborative environments, effective principal instructional leadership, and positive school culture—can facilitate greater returns to teacher experience or the extent to which teachers become more effective over…”
Background
Research works printed but not linked: 7
- Glazerman S., 2010, Evaluating teachers: The important role of value-added
- Goldhaber D. D., 2002, Education Next, 2, 50
- National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010, Common Core State Standards (Mathematics)
- Sartain L., Journal of Human Resources
- Steinberg M., Education Finance and Policy
- Watson J. G., 2009, The other 69 percent
- White M., 2012, A user guide to the “core study” data files available to MET early career grantees
Cited by
Teacher and Teaching Effects on Students’ Attitudes and Behaviors
Blazar, D., & Kraft, M. A. (2017) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373716670260 · Cited by 591
“A related concern is that the specific students in the classroom may influence teachers’ instructional quality (Hill et al., 2015; Steinberg & Garrett, 2016; Whitehurst, Chingos, & Lindquist, 2014).”
BackgroundBuilding a More Complete Understanding of Teacher Evaluation Using Classroom Observations
Cohen, J., & Goldhaber, D. (2016) · Educational Researcher · doi:10.3102/0013189x16659442 · Cited by 201
“Some studies find that measures of classroom climate, teacher-student interactions, and behavior management are particularly sensitive to students’ prior academic achievement (Lazarev & Newman, 2015; Steinberg & Garrett, 2016).”
BackgroundInfluentialTeacher Coaching in a Simulated Environment
Cohen, J., Wong, V., Krishnamachari, A., & Berlin, R. (2020) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373720906217 · Cited by 185
“This allows for clearer focus on specific teaching skills, decoupled from variations in students, curricula, or mentor teachers, all of which could influence assessments of candidates’ teaching (Steinberg & Garrett, 2016; Whitehurst et al., 2014).”
BackgroundWorking and Learning Together: Rethinking Human Resource Policies for Schools
OECD (2019) · OECD reviews of school resources · doi:10.1787/b7aaf050-en · Cited by 203
Observational Evaluation of Teachers: Measuring More Than We Bargained for?
Campbell, S. L., & Ronfeldt, M. (2018) · American Educational Research Journal · doi:10.3102/0002831218776216 · Cited by 111
“Steinberg and Garrett (2016) conclude, ‘‘Ultimately, we are unable to definitively determine whether the incoming achievement3 effect reflects bias in observation scores due to the composition of a teacher’s class or whether teachers are systematically higher performing when assigned to higher…”
BackgroundMethodResultInfluential
Metadata
- Authors
- Steinberg, M. P., & Garrett, R.
- Affiliations
- University of Pennsylvania (US)
- American Institutes for Research (US)
- Venue
- Educational Evaluation and Policy Analysis
- Published
- 2016-06
- Volume
- 38
- Issue
- 2
- Pages
- 293-317
- ISSN
- 0162-3737 (ISSN-L)
- 1935-1062
- OpenAlex
- W2276065988
- Semantic Scholar
- 2a3e4849f58644c5430aaa9f564ab056cbf8d5f9
- Cited by
- 150
- References
- 38
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 8 · 2023: 16 · 2022: 20 · 2021: 17 · 2020: 26 · 2019: 21 · 2018: 13 · 2017: 12 · 2016: 3
- FWCI
- 65.43
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Evaluation and Policy Analysis (publishedVersion)
- Funders
- National Academy of Education
- Topics
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Teacher Education and Assessments
- Social Sciences › Education › Educational Assessment and Improvement
- Keywords
- teacher evaluation, class composition, teacher performance ratings, classroom observation, academic performance, teacher sorting, teacher retention, teacher accountability, Measures of Effective Teaching, high-stakes testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Classroom Composition and Measured Teacher Performance: What Do Teacher Observation Scores Really Measure? https://app.cochat.ai/papers/10.3102/0162373715616249
- The paper
- Steinberg, M. P., & Garrett, R. (2016). Classroom Composition and Measured Teacher Performance: What Do Teacher Observation Scores Really Measure? Educational Evaluation and Policy Analysis. https://doi.org/10.3102/0162373715616249
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.