Classroom Composition and Measured Teacher Performance: What Do Teacher Observation Scores Really Measure?
Open accessThe DOI matches its Crossref record; 34 of the 38 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.
Abstract
As states and districts implement more rigorous teacher evaluation systems, measures of teacher performance are increasingly being used to support instruction and inform retention decisions. Classroom observations take a central role in these systems, accounting for the majority of teacher ratings upon which accountability decisions are based. Using data from the Measures of Effective Teaching study, we explore the extent to which classroom composition influences measured teacher performance based on classroom observation scores. The context in which teachers work—most notably, the incoming academic performance of their students—plays a critical role in determining teachers’ measured performance. Furthermore, the intentional sorting of teachers to students has a significant influence on measured performance. Implications for high-stakes teacher accountability policies are discussed.
Overview
CoChat Summary
TL;DR
Using Measures of Effective Teaching (MET) data, the paper finds that teachers’ observation-based performance scores rise when they are assigned higher-achieving students, even after accounting for teacher fixed effects. It also shows that nonrandom teacher-class matching can bias these observation scores, raising concerns about high-stakes evaluation systems that rely heavily on them.
Key findings
- In the 2009–2010 school year, ELA teachers were “more than twice as likely” to be rated in the top performance quintile when assigned the highest-achieving students versus the lowest-achieving students; math teachers were “more than 6 times as likely.”
- Roughly half of teachers assigned the highest-performing students were rated in the top two quintiles: 48% in ELA and 54% in math; among teachers assigned the lowest-performing students, the share was 37% in ELA and 18% in math.
- In the full sample, a 1 standard deviation increase in incoming achievement was associated with a 0.11 FFT-point increase in ELA measured performance and a 0.06 FFT-point increase in math in the main estimate set; in the IV sample, the paper reports 0.14 FFT points for ELA and 0.08 FFT points for math.
- For ELA teachers in partial-compliance classes, the OLS estimate was 0.21 FFT points and the IV estimate was 0.17 FFT points, both above the FE estimate of 0.14 FFT points, which the authors interpret as upward bias from nonrandom matching.
- For specific practices, higher-achieving students were associated with better ratings on ELA instructional components 3a and 3c, while measures tied more to questioning and assessment (3b and 3d) were not sensitive to incoming achievement; math practices were largely unrelated to incoming achievement except for component 2b.
Methods
- The study uses data from the MET project over two school years (2009–2010 and 2010–2011) across six districts, with an analytic teacher sample of 834 Grade 4–9 teachers (530 ELA, 480 math, with overlap).
- Teacher performance is measured using classroom observation scores from the Danielson Framework for Teaching (FFT), aggregating eight observed components from video-recorded lessons.
- The main analysis combines teacher fixed-effects models, OLS, and IV comparisons, and uses the Year 2 random assignment sample to assess causal effects and detect bias from nonrandom teacher-class matching.
Limitations
- The paper states that classroom composition data were available only for teachers’ actual classes, not for the class rosters to which teachers were randomly assigned, and that student data for non-MET teachers after randomization were not observed.
- The authors note that the FE approach can still be biased if there are time-varying shocks to teacher-class assignment, and the IV approach can be biased if there is partial/full noncompliance or selection into compliance classes.
- Evident from the text: the experimental ELA result is described as “not precisely estimated,” and the authors say the math randomization sample did not satisfy the covariate-balance test, so the experimental estimate for math is not treated as a valid causal effect.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 10 Not checked · 104 Own results and interpretations
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, we offer evidence that the intentional, nonrandom matching of teachers to classrooms due to unobserved time-invariant teacher effects biases the relationship between incoming student achievement and observed classroom performance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students' incoming achievement is more strongly associated with observed English Language Art (ELA) instruction than math instruction.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that teacher performance, based on classroom observation, is significantly influenced by the context in which teachers work.”
- Also stated in: § Rachel Garrett · boosted
“The context in which teachers work—most notably, the incoming academic performance of their students—plays a critical role in determining teachers' measured performance.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Rachel Garrett · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the influence of incoming achievement is concentrated among subject specialists, who work with multiple classes of students within a single school day, compared with classroom generalist teachers who work with the same classroom of students all day, every day”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we are unable to uniquely attribute these differences to either departmentalized (compared with generalist) class status or grade-level differences.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
70 more claims
Create a free account to see all 116 claims and every source verdict.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 23
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Incoming Achievement and Classroom Composition
“| Teachers | 180 | 350 | 177 | 303 |”
§ Teacher Sample
“The teacher-level analytic sample consists of 834 teachers in Grades 4 to 9, of which 354 taught ELA, 304 taught math, and 176 taught both subjects.”
Citations
Cites
Teachers, Schools, and Academic Achievement
Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005) · Econometrica · doi:10.1111/j.1468-0262.2005.00584.x · Cited by 4,501
The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data
Rockoff, J. E. (2004) · American Economic Review · doi:10.1257/0002828041302244 · Cited by 2,432 · cited 2×
“Revisions to teacher evaluation systems and the measures upon which teacher evaluation is based have been spurred by evidence in support of the critical role teachers play in improving student outcomes (Goldhaber, 2002; Rivkin, Hanushek, & Kain, 2005; Rockoff, 2004), while also acknowledging both the substantial withi…”
BackgroundTeachers and Student Achievement in the Chicago Public High Schools
Aaronson, D., Barrow, L., & Sander, W. (2007) · Journal of Labor Economics · doi:10.1086/508733 · Cited by 1,291
The Common Core State Standards in Mathematics
McCallum, W. (2015) · Selected Regular Lectures from the 12th International Congress on Mathematical Education · doi:10.1007/978-3-319-17187-6_31 · Cited by 770
American Economic Journal: Applied Economics
Duflo, E. (2009) · American Economic Review · doi:10.1257/aer.99.2.676 · Cited by 770
Teacher-Student Matching and the Assessment of Teacher Effectiveness
Clotfelter, C. T., Ladd, H. F., & Vigdor, J. L. (2006) · Journal of Human Resources · doi:10.3368/jhr.xli.4.778 · Cited by 739 · cited 3×
“…randomly assigned to classes of students.1 In principle, the random assignment of teachers to classes disrupts the assortative matching process that can confound measures of teacher effectiveness (Clotfelter et al., 2006; Garrett & Steinberg, 2015; Rothstein, 2009, 2010; Whitehurst et al., 2014).”
BackgroundTeacher Quality in Educational Production: Tracking, Decay, and Student Achievement*
Rothstein, J. (2010) · Quarterly Journal of Economics · doi:10.1162/qjec.2010.125.1.175 · Cited by 735 · cited 3×
“…randomly assigned to classes of students.1 In principle, the random assignment of teachers to classes disrupts the assortative matching process that can confound measures of teacher effectiveness (Clotfelter et al., 2006; Garrett & Steinberg, 2015; Rothstein, 2009, 2010; Whitehurst et al., 2014).”
Background(2007) · The Elementary School Journal · doi:10.1086/518627 · Cited by 642
(2009) · Issue Lab (Candid) · Cited by 552
Kraft, M. A., & Papay, J. P. (2014) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713519496 · Cited by 517 · cited 2×
“For example, Kraft and Papay (2014) find that school context—including supportive and collaborative environments, effective principal instructional leadership, and positive school culture—can facilitate greater returns to teacher experience or the extent to which teachers become more effective over…”
Background
Research works printed but not linked: 7
- Glazerman S., 2010, Evaluating teachers: The important role of value-added
- Goldhaber D. D., 2002, Education Next, 2, 50
- National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010, Common Core State Standards (Mathematics)
- Sartain L., Journal of Human Resources
- Steinberg M., Education Finance and Policy
- Watson J. G., 2009, The other 69 percent
- White M., 2012, A user guide to the “core study” data files available to MET early career grantees
Cited by
Teacher and Teaching Effects on Students’ Attitudes and Behaviors
Blazar, D., & Kraft, M. A. (2017) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373716670260 · Cited by 591
“A related concern is that the specific students in the classroom may influence teachers’ instructional quality (Hill et al., 2015; Steinberg & Garrett, 2016; Whitehurst, Chingos, & Lindquist, 2014).”
BackgroundBuilding a More Complete Understanding of Teacher Evaluation Using Classroom Observations
Cohen, J., & Goldhaber, D. (2016) · Educational Researcher · doi:10.3102/0013189x16659442 · Cited by 201
“Some studies find that measures of classroom climate, teacher-student interactions, and behavior management are particularly sensitive to students’ prior academic achievement (Lazarev & Newman, 2015; Steinberg & Garrett, 2016).”
BackgroundInfluentialTeacher Coaching in a Simulated Environment
Cohen, J., Wong, V., Krishnamachari, A., & Berlin, R. (2020) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373720906217 · Cited by 185
“This allows for clearer focus on specific teaching skills, decoupled from variations in students, curricula, or mentor teachers, all of which could influence assessments of candidates’ teaching (Steinberg & Garrett, 2016; Whitehurst et al., 2014).”
BackgroundWorking and Learning Together: Rethinking Human Resource Policies for Schools
OECD (2019) · OECD reviews of school resources · doi:10.1787/b7aaf050-en · Cited by 202
Observational Evaluation of Teachers: Measuring More Than We Bargained for?
Campbell, S. L., & Ronfeldt, M. (2018) · American Educational Research Journal · doi:10.3102/0002831218776216 · Cited by 111
“Steinberg and Garrett (2016) conclude, ‘‘Ultimately, we are unable to definitively determine whether the incoming achievement3 effect reflects bias in observation scores due to the composition of a teacher’s class or whether teachers are systematically higher performing when assigned to higher…”
BackgroundMethodResultInfluentialDonaldson, M. L., & Woulfin, S. (2018) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373718784205 · Cited by 105
“In observations, principals tend to assign teachers higher scores when students are higher performing (Steinberg & Garrett, 2016; Whitehurst et al., 2014).”
BackgroundEvaluating Prospective Teachers: Testing the Predictive Validity of the edTPA
Goldhaber, D. D., Cowan, J., & Theobald, R. J. (2017) · Journal of Teacher Education · doi:10.1177/0022487117702582 · Cited by 134
“Finally, given concerns about the fairness of teacher observations across classroom contexts (Steinberg & Garrett, 2016) and recent calls to place more student teachers in disadvantaged schools (Krieg, Theobald, & Goldhaber, 2016), policy makers would benefit from evidence about whether edTPA…”
BackgroundAssessment and the Future of Teacher Education
Richmond, G., Salazar, M. D. C., & Jones, N. D. (2019) · Journal of Teacher Education · doi:10.1177/0022487118824331 · Cited by 100
“We have learned, for example, that observation scores are often subject to bias and can be sensitive to a variety of contextual factors (Garrett & Steinberg, 2015; Gill, Shoji, Coen, & Place, 2016; Steinberg & Garrett, 2016; Whitehurst, Chingos, & Lindquist, 2014).”
BackgroundEvaluating School Principals: Supervisor Ratings of Principal Practice and Principal Job Performance
Grissom, J. A., Blissett, R. S. L., & Mitani, H. (2018) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373718783883 · Cited by 112
“This model estimates the association between TEAM ratings and experience within principal (see also Gill, Shoji, Coen, & Place, 2016; Steinberg & Garrett, 2016).”
MethodBacher-Hicks, A., Chin, M. J., Kane, T. J., & Staiger, D. O. (2019) · Economics of Education Review · doi:10.1016/j.econedurev.2019.101919 · Cited by 93
Bastian, K. C., McCord, D., Marks, J. T., & Carpenter, D. (2017) · AERA Open · doi:10.1177/2332858416684764 · Cited by 98
“We control for school characteristics in teacher evaluation analyses given recent work showing that teachers’ ratings are significantly correlated with measures of school and classroom context—even with the random assignment of teachers (Steinberg & Garrett, 2016; Whitehurst et al., 2014).”
MethodThorpe, K. J., Rankin, P. S., Beatton, T., Houen, S., Sandi, M., Siraj, I., & Staton, S. (2020) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2020.05.003 · Cited by 67
Teacher Skill Development: Evidence from Performance Ratings by Principals
Kraft, M. A., Papay, J. P., & Chi, O. L. (2019) · Journal of Policy Analysis and Management · doi:10.1002/pam.22193 · Cited by 62
Marsh, J. A., Bush-Mecenas, S. C., Strunk, K. O., Lincove, J. A., & Huguet, A. (2017) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373717698221 · Cited by 58
“…& Umland, 2011; Ho & Kane, 2013; Kane & Staiger, 2012; Kane, Taylor, Tyler, & Wooten, 2011; Kimball, 2002; Sartain, Stoelinga, & Brown, 2009; Steinberg & Garrett, 2016; Strunk, Weinstein, & Makkonnen, 2014; Taylor & Tyler, 2012) and the efficacy of evaluation programs (e.g., Dee & Wyckoff,…”
Kell, H. J. (2019) · ETS Research Report Series · doi:10.1002/ets2.12241 · Cited by 74
“…including student value-added measures (Ballou & Springer, 2015), tardiness and absenteeism (Jacob et al., 2016), student surveys (Bacher-Hicks, Chin, Kane, & Staiger, 2017), classroom observations (Steinberg & Garrett, 2016), and influence on students’ noncognitive skills (Cheng & Zamarro, 2016).”
Classroom observation systems in context: A case for the validation of observation systems
Liu, S., Bell, C. A., Jones, N. D., & McCaffrey, D. F. (2019) · Educational Assessment, Evaluation and Accountability · doi:10.1007/s11092-018-09291-3 · Cited by 46
“For example, Steinberg and Garrett (2016) explored the random assignment of students in the Measures of Effective Teaching project and found that the typical nonrandom assignment of students across teachers based on their prior academic achievement biases teacher evaluation ratings.”
BackgroundClose, K., Amrein‐Beardsley, A., & Collins, C. (2020) · Education Policy Analysis Archives · doi:10.14507/epaa.28.5252 · Cited by 72
“…contextual factors like the types of students with whom a teacher works, how a teacher’s gender interplays with his/her students’ gender, and other factors (Bailey, Bocala, Shakman, & Zweig, 2016; Geiger & Amrein-Beardsley, 2017; Steinberg & Garrett, 2016; Whitehurst, Chingos, & Lindquist, 2014).”
BackgroundDrake, S., Auletto, A., & Cowen, J. M. (2019) · American Educational Research Journal · doi:10.3102/0002831219835776 · Cited by 51
“This finding is consistent with work by Steinberg and Garrett (2016), where lower prior student achievement was found to predict lower classroom observation scores.”
ResultWhat Explains the Race Gap in Teacher Performance Ratings? Evidence From Chicago Public Schools
Steinberg, M. P., & Sartain, L. (2020) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373720970204 · Cited by 43
“…evidence showing that the incoming achievement of a teacher’s students affects teacher performance based on classroom observation scores (Steinberg & Garrett, 2016).12 Furthermore, teachers receive higher observation ratings, on average, when teaching in better school climates—schools…”
BackgroundInfluentialSelf-citationKelly, S., Bringe, R., Aucejo, E. M., & Fruehwirth, J. C. (2020) · Education Policy Analysis Archives · doi:10.14507/epaa.28.5012 · Cited by 55
“…at times in the construction of the protocols themselves, and there is indirect evidence of this possibility in Campbell and Ronfeldt (2018), Steinberg and Garrett (2016), and our own replication of Campbell and Ronfeldt’s analysis of the relationship between classroom composition and FFT…”
BackgroundLinks Among Teacher Preparation, Retention, and Teaching Effectiveness
Ronfeldt, M. (2021) · doi:10.31094/2021/3/1 · Cited by 60
“…and students of color tend to receive lower observation ratings, but these lower ratings do not appear to be explained by actual differences in teaching effectiveness (Campbell, 2020; Campbell & Ronfeldt, 2018; Grissom & Bartanen, forthcoming; Jiang & Sporte, 2016; Steinberg & Garrett, 2016).”
BackgroundPotential Race and Gender Biases in High‐Stakes Teacher Observations
Grissom, J. A., & Bartanen, B. (2021) · Journal of Policy Analysis and Management · doi:10.1002/pam.22352 · Cited by 41
Rodríguez, L. A., Swain, W. A., & Springer, M. G. (2020) · American Educational Research Journal · doi:10.3102/0002831220910989 · Cited by 51
Teacher Effectiveness and Classroom Composition: Understanding Match Effects in the Classroom
Aucejo, E. M., Coate, P., Fruehwirth, J. C., Kelly, S., & Mozenter, Z. (2022) · The Economic Journal · doi:10.1093/ej/ueac046 · Cited by 49
Ronfeldt, M., Matsko, K. K., Nolan, H. G., & Reininger, M. (2020) · Journal of Teacher Education · doi:10.1177/0022487120919753 · Cited by 42
“These results are consistent with studies outside of Chicago that have found observational ratings to be associated with characteristics of teachers and their students (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016).”
ResultTeacher Evaluation and Teacher Turnover in Equilibrium: Evidence From DC Public Schools
James, J., & Wyckoff, J. (2020) · AERA Open · doi:10.1177/2332858420932235 · Cited by 39
“There is a debate whether to control for student attributes when examining measures of teacher quality (e.g., Whitehurst et al., 2014; Steinberg & Garrett, 2016; Campbell & Ronfeldt, 2018).”
BackgroundThe Compositional Effect of Rigorous Teacher Evaluation on Workforce Quality
Cullen, J. B., Koedel, C., & Parsons, E. (2016) · Education Finance and Policy · doi:10.1162/edfp_a_00292 · Cited by 47
Youngs, P. A., Elreda, L. M., Anagnostopoulos, D., Cohen, J., Drake, C., & Konstantopoulos, S. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103576 · Cited by 46
Morris-Mathews, H., Stark, K. R., Jones, N. D., Brownell, M. T., & Bell, C. A. (2021) · Journal of Learning Disabilities · doi:10.1177/0022219420941804 · Cited by 30
“Prior research suggests that teachers who work with lower achieving students systematically receive lower ratings on FFT (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016).”
Backgroundvon Hippel, P. T., & Bellows, L. (2017) · Economics of Education Review · doi:10.1016/j.econedurev.2018.01.005 · Cited by 39
Luoto, J. M., Klette, K., & Blikstad‐Balas, M. (2022) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-022-09394-y · Cited by 32
“These factors include the cultural context and its norms associated with teaching (Bishop, 1988), teacher beliefs and values (Leder et al., 2003), student composition (Steinberg & Garrett, 2016), and expectations and support from school leaders (Kraft & Papay, 2014).”
BackgroundOECD Reviews of School Resources
(2015) · OECD reviews of school resources · doi:10.1787/24133841 · Cited by 28
Bastian, K. C., Patterson, K. M., & Pan, Y. (2017) · Journal of Teacher Education · doi:10.1177/0022487117718182 · Cited by 38
“Second, it is important to adjust for elements of school context when analyzing the evaluation ratings of TPP graduates (Steinberg & Garrett, 2016; Whitehurst et al., 2014).”
BackgroundHunter, S. B., & Springer, M. G. (2022) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211062913 · Cited by 42
“Although classroom observations have been used to assess teachers for far longer than VAM (Brophy & Good, 1986), researchers have only recently begun to evaluate the properties of observation scores (Campbell & Ronfeldt, 2018; Hunter, 2020; Steinberg & Garrett, 2016).”
BackgroundMethodClassroom composition and quality in early childhood education: A systematic review
Aguiar, A. L., & Aguiar, C. (2020) · Children and Youth Services Review · doi:10.1016/j.childyouth.2020.105086 · Cited by 39
Room for Improvement? Mentor Teachers and the Evolution of Teacher Preservice Clinical Evaluations
Goldhaber, D. D., Ronfeldt, M., Cowan, J., Gratz, T. M., Bardelli, E., & Truwit, M. (2022) · American Educational Research Journal · doi:10.3102/00028312211066867 · Cited by 36
“…clinical evaluations are predictive of teacher performance in the classroom, although there is some evidence that they can be influenced by classroom assignments (Bastian et al., 2018; Campbell & Ronfeldt, 2018; Chen et al., 2019, Goldhaber, Cowan, & Theobald, 2017; Steinberg & Garrett, 2016).”
BackgroundExamining Clinical Teaching Observation Scores as a Measure of Preservice Teacher Quality
Bartanen, B., & Kwok, A. (2021) · American Educational Research Journal · doi:10.3102/0002831221990359 · Cited by 30
“…2015; Grissom & Loeb, 2017; Kane et al., 2011), others have demonstrated that these scores in part reflect factors that teachers cannot control, such as their race, gender, classroom composition, and school context (Campbell & Ronfeldt, 2018; Cohen & Goldhaber, 2016; Steinberg & Garrett, 2016).”
BackgroundResultPaige, M. A., & Amrein‐Beardsley, A. (2020) · Educational Researcher · doi:10.3102/0013189x20923046 · Cited by 25
“…pertain to states’ and school districts’ uses of VAMs only.7 As such, we recognize but do not review the literature surrounding the systemic observations of teachers (see, e.g., Anderson, 2013; Goldring et al., 2015; Jiang, Sporte, & Luppescu, 2015; Steinberg & Garrett, 2016; Winerip, 2011).”
BackgroundExperimental Evidence on the Robustness of Coaching Supports in Teacher Education
Cohen, J., Wong, V. C., Krishnamachari, A., & Erickson, S. (2023) · Educational Researcher · doi:10.3102/0013189x231198827 · Cited by 44
“…ways, allowing teacher educators and researchers to focus on developing specific skills while limiting other sources of variation, such as instructional content (Cohen et al., 2018), the influence of mentors (Goldhaber et al., 2020), or the composition of students (Steinberg & Garrett, 2016).”
BackgroundTeaching to support students with diverse academic needs
Blazar, D., & Archer, C. (2020) · Educational Researcher · doi:10.3102/0013189x20931226 · Cited by 23
“…other than those in which we measured student outcomes and addressed concerns that observation scores may be biased because of the composition of students in the classroom (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016; Whitehurst, 300 EDucATIONAl RESEARcHER Chingos, & Lindquist, 2015).”
BackgroundMikeska, J. N., Holtzman, S. L., McCaffrey, D. F., Liu, S., & Shattuck, T. (2018) · Science Education · doi:10.1002/sce.21482 · Cited by 28
Does a teacher’s classroom observation rating vary across multiple classrooms?
Lei, X., Li, H., & Leroux, A. J. (2017) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-017-9269-x · Cited by 29
“There are certainly many other contextual factors that impact teachers’ classroom teaching performance (Bell et al. 2012; Steinberg and Garrett 2016).”
BackgroundGilmour, A. F., & Henry, G. T. (2018) · The Elementary School Journal · doi:10.1086/696140 · Cited by 23
“Some evidence suggests that principals’ ratings of teachers are influenced by the students in the teacher’s class rather than the teacher’s behavior alone (Steinberg & Garrett, 2016).”
BackgroundContext in research on teaching quality: bringing a complex idea into the spotlight
Praetorius, A., Jentsch, A., Luoto, J. M., Keller, S., & Fauth, B. (2025) · School Effectiveness and School Improvement · doi:10.1080/09243453.2025.2482577 · Cited by 27
“…in teaching-quality measurements exists within teachers between classes (Cherng et al., 2022, p. 185; see also Campbell & Ronfeldt, 2018, and Steinberg & Garrett, 2016, for similar findings drawing on the MET randomized teacher sample; Kane et al., 2013) and show that these differences are…”
Teaching Bias? Relations between Teaching Quality and Classroom Demographic Composition
Cherng, H. S., Halpin, P. F., & Rodríguez, L. A. (2021) · American Journal of Education · doi:10.1086/717676 · Cited by 24
Estimation and interpretation of teacher value added in research applications
Bacher-Hicks, A., & Koedel, C. (2022) · Handbook of the economics of education · doi:10.1016/bs.hesedu.2022.11.002 · Cited by 42
A Degree Above? The Value-Added Estimates and Evaluation Ratings of Teachers with a Graduate Degree
Bastian, K. C. (2018) · Education Finance and Policy · doi:10.1162/edfp_a_00261 · Cited by 18
Kokkinou, E., & Kyriakides, L. E. (2022) · School Effectiveness and School Improvement · doi:10.1080/09243453.2022.2030762 · Cited by 20
“…students, students with special needs, or relatively older students as compared to the grade-level average; e.g., Grossman et al., 2014; Mihaly & McCaffrey, 2014; Smylie et al., 2008; Steinberg & Garrett, 2016; Whitehurst et al., 2014); and grade (e.g., Goe et al., 2008; Mihaly & McCaffrey, 2014).”
BackgroundMethodInfluentialCompulsory class attendance versus autonomy
Goulas, S., Griselda, S., & Megalokonomou, R. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.06.018 · Cited by 20
Hunter, S. B. (2020) · AERA Open · doi:10.1177/2332858420929276 · Cited by 18
Strategies for Assessing Classroom Teaching: Examining Administrator Thinking as Validity Evidence
Bell, C. A., Jones, N. D., Qi, Y., & Lewis, J. M. (2018) · Educational Assessment · doi:10.1080/10627197.2018.1513788 · Cited by 18
“…and the lack of a relationship between reasoning and accuracy add to increasing concerns about the ways in which observation scores reflect something other than a latent teacher trait researchers often implicitly think of as “teacher quality” (Steinberg & Garrett, 2016; Whitehurst et al., 2014).”
BackgroundBackes, B., Cowan, J., Goldhaber, D. D., Koedel, C., Miller, L. C., & Xu, Z. (2017) · Economics of Education Review · doi:10.1016/j.econedurev.2017.10.004 · Cited by 19
Milanowski, A. (2017) · AERA Open · doi:10.1177/2332858416685550 · Cited by 25
“As Steinberg and Garrett (2016) observed, randomization was incomplete in the MET study, despite the incentives districts received to randomize from the Gates Foundation.”
BackgroundInfluentialCowan, J., Goldhaber, D. D., & Theobald, R. J. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2021.2018747 · Cited by 18
“…three potential influences on teacher performance ratings: observable features of the teacher’s classroom environment (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016), differences in the leniency of performance evaluation across districts and schools, and differences in teaching…”
BackgroundMalaysia, U. T. H. O., Hassan, B., Alias, M., Malaysia, U. T. H. O., Saleh, K. M., Malaysia, U. T. H. O., Awang, H. B., & Malaysia, U. T. H. O. (2017) · Path of Science · doi:10.22178/pos.27-4 · Cited by 32
“Even though there are many studies that have been conducted and opinions made on the perception of students relating to teachers’ performance in teaching, there still exist conflicting views of students’ perceptions about teachers performance in teaching their courses [53, 25, 13].”
BackgroundAddressing Competency Gaps for Vocational Instructor through Competency Modelling
Arifin, M. A., Rasdi, R. M., Anuar, M. A. M., & Omar, M. K. (2017) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v7-i4/2970 · Cited by 16
“7.0 Proposed competencies model The growing interest towards identifying effective teaching has encourages researchers to address both on how to measure teacher performance (Gupta, Chong, & Leong, 2015; Steinberg & Garrett, 2016).”
BackgroundInfluentialThe construct validity and reliability of the lesson plan assessment instrument in primary schools
Otaya, L. G., Kartowagiran, B., & Retnawati, H. (2020) · Jurnal Prima Edukasia · doi:10.21831/jpe.v8i2.33275 · Cited by 24
“Teachers who work with high performance tend to have higher academic achievement (Steinberg & Garrett, 2016; Stronge, 2018).”
BackgroundWhat Do Surveys of Program Completers Tell Us About Teacher Preparation Quality?
Bastian, K. C., Sun, M., & Lynn, H. H. (2019) · Journal of Teacher Education · doi:10.1177/0022487119886294 · Cited by 21
“Likewise, prior work suggests the need to control for teacher demographics and classroom and school context when examining evaluation ratings (Bastian et al., 2018; Steinberg & Garrett, 2016; Whitehurst, Chingos, & Lindquist, 2014) and teacher retention (Borman & Dowling, 2008).”
BackgroundThe Consistency of Composite Ratings of Teacher Effectiveness: Evidence From New Mexico
Doan, S., Schweig, J. D., & Mihály, K. (2019) · American Educational Research Journal · doi:10.3102/0002831219841369 · Cited by 13
“Additionally, some research shows that observation scores are potentially biased by observer (Grissom & Loeb, 2017; Ho & Kane, 2013) and student (Mihaly & McCaffrey, 2014; Steinberg & Garrett, 2016) characteristics.”
BackgroundLearning from what doesn’t work in teacher evaluation
Close, K., & Amrein‐Beardsley, A. (2018) · Phi Delta Kappan · doi:10.1177/0031721718797115 · Cited by 16
“Research suggests that VAMs, whether used on their own or in combination with classroom observations (Gill et al., 2016; Steinberg & Garrett, 2016; Whitehurst, Chingos, & Lindquist, 2014), tend to be not just unreliable but also biased.”
BackgroundBastian, K. C., Lys, D. B., & Whisenant, W. R. L. (2022) · Journal of Teacher Education · doi:10.1177/00224871221105814 · Cited by 19
“Like recent scholarship on inservice teacher evaluation systems, such associations may reflect bias in edTPA scoring (Ronfeldt & Campbell, 2016; Steinberg & Garrett, 2016).”
BackgroundTan, R., Schwab, S., & Perren, S. (2021) · International Journal of Early Years Education · doi:10.1080/09669760.2021.1983775 · Cited by 13
Amrein‐Beardsley, A., & Geiger, T. J. (2019) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-019-09311-w · Cited by 14
Administrators' Uses of Teacher Observation Protocol in Different Rating Contexts
Qi, Y., Bell, C. A., Jones, N. D., Lewis, J. M., Witherspoon, M. W., & Redash, A. (2018) · ETS Research Report Series · doi:10.1002/ets2.12205 · Cited by 16
“1 There has been a good deal of research in which observation scores are created by paid raters who are not administrators (e.g., Gill, Shoji, Coen, & Place, 2016; Kane & Staiger, 2012; Steinberg & Garrett, 2016).”
BackgroundPrincipals may inflate teacher evaluation scores to achieve important goals
Jones, E. A., Bergin, C. C., & Murphy, B. C. (2021) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-021-09366-8 · Cited by 19
“Research suggests that OTP ratings may have substantial bias (e.g., Garrett & Steinberg, 2015; Steinberg & Garrett, 2016).”
BackgroundHsieh, M.-C., & Chen, T. (2019) · Medical Education Online · doi:10.1080/10872981.2019.1620544 · Cited by 11
College Teaching from Multiple Angles: A Multi-trait Multi-method Analysis of College Courses
Campbell, C. M., Michel, J. O., Patel, S., & Gelashvili, M. (2018) · Research in Higher Education · doi:10.1007/s11162-018-9529-8 · Cited by 13
“According to Steinburg and Garrett (2016), observation-based teacher performance is poorly correlated over time.”
BackgroundSutton, E., Brown, J. L., Lowenstein, A. E., & Downer, J. T. (2021) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2021.101975 · Cited by 16
Measuring Preschool Teachers’ Social-emotional Practices: A Comparison of Two Measures
Rojas, N. M., Mattera, S. K., Morris, P., & Raver, C. (2021) · Early Education and Development · doi:10.1080/10409289.2020.1864838 · Cited by 17
“Prior evidence from studies of elementary and middle schools also suggests that observational measures of classroom climate and teacher practices tend to be weakly correlated over time (e.g., Garrett & Steinberg, 2015; Steinberg & Garrett, 2016).”
BackgroundMugizi, W., & Amwine, C. M. (2020) · Creative Education · doi:10.4236/ce.2020.112012 · Cited by 19
Complexity and scale in teaching effectiveness research: Reflections from the MET Study
Jensen, B., Wallace, T. L., Steinberg, M. P., Gabriel, R. E., Dietiker, L. C., Davis, D. S., Kelcey, B. M., Minor, E. C., Halpin, P. F., & Rui, N. (2019) · Education Policy Analysis Archives · doi:10.14507/epaa.27.3923 · Cited by 16
Self-citationTeachers and the Gender Gap in Reading Achievement
Aucejo, E. M., Fruehwirth, J. C., Kelly, S., & Mozenter, Z. (2022) · Journal of Human Capital · doi:10.1086/719731 · Cited by 12
Chi, O. L. (2021) · Education Finance and Policy · doi:10.1162/edfp_a_00367 · Cited by 11
“Steinberg and Garrett (2016) find that students’ incoming academic performance is positively related to observation ratings, and Campbell and Ronfeldt (2018) find that teachers with higher proportions of black, Hispanic, male, and low-performing students receive lower observation ratings even after…”
BackgroundHunter, S. B., & Ege, A. (2021) · Educational Administration Quarterly · doi:10.1177/0013161x211003134 · Cited by 9
“…(g) education levels, (h) measures of their prior-year performance (Jacob & Walsh, 2011; Kraft et al., 2020); and student (i) special education (SPED), (j) free or reduced-price lunch (FRPL), and (k) English as a second language (ESL) status (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016).”
BackgroundMeasuring effective teaching: Student perceptions of their modern languages lessons in England
Molway, L. (2020) · System · doi:10.1016/j.system.2020.102440 · Cited by 12
Kettler, R. J., Arnold-Berkovits, I., Reddy, L. A., Kurz, A., Dudek, C. M., Hua, A. N., & Lekwa, A. J. (2018) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2018.08.004 · Cited by 14
Olaivar, R., & Loayon, L. (2022) · International Journal of Research -GRANTHAALAYAH · doi:10.29121/granthaalayah.v10.i9.2022.4758 · Cited by 17
Kelly, S., Mozenter, Z., Aucjeo, E. M., & Fruehwirth, J. C. (2020) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146812012201102 · Cited by 8
“Analyzing the FFT scores, Steinberg and Garrett (2016) reported an influence of classroom achievement composition, while Campbell and Ronfeldt (2018) reported additional socio-demographic influences.”
BackgroundScience Teacher Motivation and Evaluation Policy in a High-Stakes Testing State
Mintz, J. A., & Kelly, A. M. (2018) · Educational Policy · doi:10.1177/0895904818810520 · Cited by 14
Examining the Factor Structure Underlying the TAP System for Teacher and Student Advancement
Sloat, E. F., Amrein‐Beardsley, A., & Sabo, K. E. (2017) · AERA Open · doi:10.1177/2332858417735526 · Cited by 11
“However, the intent has historically been formative, providing teachers with targeted feedback to improve pedagogical competency (Cohen & Goldberger, 2016; Danielson, 2010, 2011; Steinberg & Garrett, 2016).”
BackgroundMathematics teaching quality in classrooms of different compositions in Norway
Nilsen, T., Senden, B., Ye, W., Jentsch, A., Teig, N., & König, J. (2025) · ZDM · doi:10.1007/s11858-025-01722-y · Cited by 9
“The student composition within a classroom (e.g., by SES) can in-fluence teaching practices, just as teaching affects students (e.g. Aucejo et al., 2022; Fauth et al., 2021; Steinberg & Garrett, 2016).”
Links between instructional quality and classroom composition in classes taught by the same teacher
Heinschel, A., Henschel, S., & Rjosk, C. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.102005 · Cited by 14
Comparative education and comparative classroom observation systems
Luoto, J. M. (2023) · Comparative Education · doi:10.1080/03050068.2023.2173917 · Cited by 11
“Student characteristics include student prior knowledge, socio-economic status, as such aspects e ff ect both the instruction (Steinberg and Garrett 2016), as well as what students gain from school (Carnoy et al. 2016).”
BackgroundTeacher evaluation and training
Taylor, E. S. (2023) · Handbook of the economics of education · doi:10.1016/bs.hesedu.2023.03.002 · Cited by 13
White, M., Maher, B. L., & Rowan, B. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2081275 · Cited by 15
“Further, evaluations will generally persist across multiple years and the students, grade, and/or subject taught by a teacher can change across years, which can systematically affect observation scores (e.g. Grossman et al., 2014; Steinberg & Garrett, 2016).”
BackgroundRonfeldt, M., Bardelli, E., Brockman, S. L., & Mullman, H. (2019) · American Educational Research Journal · doi:10.3102/0002831219872952 · Cited by 10
“For example, researchers have found a relationship between teachers’ evaluation ratings and the characteristics of their students (Campbell & Ronfeldt, 2018; Jiang & Sporte, 2016; Steinberg & Garrett, 2016).”
BackgroundAssessing the Predictive Validity of the Massachusetts Candidate Assessment of Performance
Chen, B., Cowan, J., Goldhaber, D. D., & Theobald, R. J. (2023) · Educational Policy · doi:10.1177/08959048231174876 · Cited by 7
“…prior evidence about the sensitivity of performance ratings to classroom context (Campbell & Ronfeldt, 2018; Cowan et al., 2018; Gill et al., 2016; Steinberg & Garrett, 2016), we create regression-adjusted ratings that use performance aggregated from the individual professional standards and…”
BackgroundMethodWhite, M. (2022) · Universitetsforlaget eBooks · doi:10.18261/9788215045054-2021-03 · Cited by 7
Workforce Outcomes of Program Completers in High-Needs Endorsement Areas
Bardelli, E., & Ronfeldt, M. (2021) · American Journal of Education · doi:10.1086/716486 · Cited by 7
“One concern is that misspecification of value-added models might attribute low scores of SpEd students to their teachers (McCaffrey and Buzick 2014; Thurlow et al. 2011). Consistent with these concerns, Ronfeldt (2015) finds that teachers with SpEd certification have lower average achievement gains in mathematics; imp…”
BackgroundInfluentialPredictors of Parent-Teacher Relationship Quality in Head Start Classrooms
Torres-Rendón, L. M., & Zinsser, K. M. (2022) · HS Dialog The Research to Practice Journal for the Early Childhood Field · doi:10.55370/hsdialog.v25i1.1324 · Cited by 5
“This finding supports how classroom context and student ethnic background plays a role in perceptions of classroom practices (Steinberg & Garrett, 2016).”
ResultMisrina, M., Suroyo, S., & Tatminingsih, S. (2022) · Aksara Jurnal Ilmu Pendidikan Nonformal · doi:10.37905/aksara.8.1.165-178.2022 · Cited by 8
Sepúlveda, F., Arévalo, E. C., & Espinoza, M. J. (2019) · Revista Electrónica Educare · doi:10.15359/ree.23-2.8 · Cited by 7
A Taxonomy of Subjective Control: Teachers‘ Narrative Accounts of a Teacher Evaluation System
Pallas, A. M., & Touloukian, C. (2024) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/01614681241240283 · Cited by 7
“…of students substantially impacts teacher evaluation scores, and because teachers are not randomly assigned to schools, this also leads to inequi-table and inaccurate teacher ratings (Campbell & Ronfeldt, 2018; Grissom & Bartanen, 2021; Steinberg & Garrett, 2016; Steinberg & Sartain, 2021).”
BackgroundKaihoi, C. A., Braun, S. S., Bottiani, J. H., & Bradshaw, C. P. (2023) · Psychology in the Schools · doi:10.1002/pits.22995 · Cited by 10
Using test scores to evaluate and hold school teachers accountable in New Mexico
Geiger, T. J., Amrein‐Beardsley, A., & Holloway, J. (2020) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-020-09324-w · Cited by 6
Ronfeldt, M., Matsko, K. K., Nolan, H. G., et al. (2020) · Sage Journals Data · doi:10.25384/sage.c.5025251.v1 · Cited by 4
“These results are consistent with studies outside of Chicago that have found observational ratings to be associated with characteristics of teachers and their students (Campbell & Ronfeldt, 2018; Steinberg & Garrett, 2016).”
ResultHunter, S. B. (2019) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-019-09307-6 · Cited by 7
“Previous studies found observational ratings were influenced by the composition of students in classrooms (Campbell and Ronfeldt 2018; Steinberg and Garrett 2016).”
BackgroundTeachers’ Views on Work Values, Employee Performance and Ethical Climate
Argon, T., & TAŞKIN, S. (2023) · International Journal on Lifelong Education and Leadership · doi:10.25233/ijlel.1304309 · Cited by 5
Predictable Variation in the Implementation of a Curricular Intervention—and Why it Matters
HSIN, L. B., Miratrix, L. W., Kim, H. Y., LaRusso, M. D., & Snow, C. E. (2023) · The Elementary School Journal · doi:10.1086/725765 · Cited by 3
Joshi, E. (2023) · American Educational Research Journal · doi:10.3102/00028312221144755 · Cited by 4
“Classroom observation scores may pick up additional information about teacher effectiveness outside of VAMs and tend to have high face validity (Steinberg & Garrett, 2016).”
Background
Metadata
- Authors
- Steinberg, M. P., & Garrett, R.
- Affiliations
- University of Pennsylvania (US)
- American Institutes for Research (US)
- Venue
- Educational Evaluation and Policy Analysis
- Published
- 2016-06
- Volume
- 38
- Issue
- 2
- Pages
- 293-317
- ISSN
- 0162-3737 (ISSN-L)
- 1935-1062
- OpenAlex
- W2276065988
- Semantic Scholar
- 2a3e4849f58644c5430aaa9f564ab056cbf8d5f9
- Cited by
- 150
- References
- 38
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 8 · 2023: 16 · 2022: 20 · 2021: 17 · 2020: 26 · 2019: 21 · 2018: 13 · 2017: 12 · 2016: 3
- FWCI
- 65.43
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Evaluation and Policy Analysis (publishedVersion)
- Funders
- National Academy of Education
- Topics
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Teacher Education and Assessments
- Social Sciences › Education › Educational Assessment and Improvement
- Keywords
- teacher evaluation, class composition, teacher performance ratings, classroom observation, academic performance, teacher sorting, teacher retention, teacher accountability, Measures of Effective Teaching, high-stakes testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.