A Validity Argument Approach to Evaluating Teacher Value-Added Scores
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Value-added models have become popular in research and pay-for-performance plans. While scholars have focused attention on some aspects of their validity (e.g., scoring procedures), others have received less scrutiny. This article focuses on the extent to which value-added scores correspond to other indicators of teacher and teaching quality. The authors compared 24 middle school mathematics teachers’ value-added scores, derived from a large ( N = 222) district data set, to survey- and observation-based indicators of teacher quality, instruction, and student characteristics. This analysis found teachers’ value-added scores correlated not only with their mathematical knowledge and quality of instruction but also with the population of students they teach. Case studies illustrate problems that might arise in using value-added scores in pay-for-performance plans.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Content Knowledge for Teaching: What Makes It Special?
Loewenberg Ball, D., Thames, M. H., & Phelps, G. (2008) · Journal of Teacher Education · doi:10.1177/0022487108324554 · Cited by 5,388
Standards for Educational and Psychological Testing
(2007) · Encyclopedia of Measurement and Statistics · doi:10.4135/9781412952644.n432 · Cited by 3,126
Educational measurement, 3rd ed.
Linn, R. L. (1989) · Cited by 2,883
Effects of Teachers’ Mathematical Knowledge for Teaching on Student Achievement
Hill, H. C., Rowan, B., & Ball, D. L. (2005) · American Educational Research Journal · doi:10.3102/00028312042002371 · Cited by 2,812 · cited 2×
“Finally, teachers completed two surveys assessing background characteristics and mathematical knowledge for teaching (MKT) with items similar to those used in Hill, Rowan, and Ball (2005).”
MethodResultSelf-citationAmerican Educational Research Journal, 1964-1968
Vesta, F. J. D., & Grinder, R. E. (1968) · American Educational Research Journal · doi:10.2307/1162008 · Cited by 2,587
The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data
Rockoff, J. E. (2004) · American Economic Review · doi:10.1257/0002828041302244 · Cited by 2,432 · cited 3×
“Our estimates of teacher-level variance are similar to others in the field (e.g., Nye et al., 2004; Rockoff, 2004).”
BackgroundResultHumphreys, L. G., & Lindquist, E. F. (1952) · The American Journal of Psychology · doi:10.2307/1418848 · Cited by 1,502
How Large Are Teacher Effects?
Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737026003237 · Cited by 1,493
Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation
Sanders, W. L., Wright, S. P., & Horn, S. P. (1997) · Journal of Personnel Evaluation in Education · doi:10.1023/a:1007999204543 · Cited by 1,044
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996 · cited 2×
“These interviews included a ‘‘clinical interview’’ similar to those used in Hill et al. (2008) that asked teachers to solve and discuss items designed to gauge mathematical knowledge for teaching.”
BackgroundMethodSelf-citationResources, Instruction, and Research
Cohen, D. K., Raudenbush, S. W., & Ball, D. L. (2003) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737025002119 · Cited by 983
Current Concerns in Validity Theory
Kane, M. T. (2001) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2001.tb01130.x · Cited by 748
The Market for Teacher Quality
Hanushek, E., Kain, J., O'Brien, D., & Rivkin, S. (2005) · National Bureau of Economic Research · doi:10.3386/w11154 · Cited by 635
(2009) · Issue Lab (Candid) · Cited by 552
Classroom Effects on Children’s Achievement Trajectories in Elementary School
Pianta, R. C., Belsky, J., Vandergrift, N., Houts, R., & Morrison, F. J. (2008) · American Educational Research Journal · doi:10.3102/0002831207308230 · Cited by 542
Evaluating Value-Added Models for Teacher Accountability
McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003) · PsycEXTRA Dataset · doi:10.1037/e658712010-001 · Cited by 541
Models for Value-Added Modeling of Teacher Effects
McCaffrey, D. F., Lockwood, J. R., Koretz, D., Louis, T. A., & Hamilton, L. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001067 · Cited by 513
Controlling for Student Background in Value-Added Assessment of Teachers
Ballou, D., Sanders, W., & Wright, P. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001037 · Cited by 434
Rowan, B., Correnti, R., & Miller, R. J. (2002) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146810210400804 · Cited by 412
The Intertemporal Variability of Teacher Effect Estimates
McCaffrey, D. F., Sass, T. R., Lockwood, J. R., & Mihaly, K. (2009) · Education Finance and Policy · doi:10.1162/edfp.2009.4.4.572 · Cited by 331
What Does Certification Tell Us About Teacher Effectiveness? Evidence from New York City
Kane, T., Rockoff, J., & Staiger, D. (2006) · National Bureau of Economic Research · doi:10.3386/w12155 · Cited by 288
Milanowski, A. (2004) · Peabody Journal of Education · doi:10.1207/s15327930pje7904_3 · Cited by 247
Limitations in the Use of Achievement Tests as Measures of Educators' Productivity
Koretz, D. M. (2002) · The Journal of Human Resources · doi:10.2307/3069616 · Cited by 235
Papay, J. P. (2011) · American Educational Research Journal · doi:10.3102/0002831210362589 · Cited by 219
Koedel, C., & Betts, J. R. (2011) · Education Finance and Policy · doi:10.1162/edfp_a_00027 · Cited by 216
Methodological Concerns About the Education Value-Added Assessment System
Amrein-Beardsley, A. (2008) · Educational Researcher · doi:10.3102/0013189x08316420 · Cited by 199
Kupermintz, H. (2003) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737025003287 · Cited by 187
Re-Examining the Role of Teacher Quality In the Educational Production FunctionPreprint
Koedel, C., & Betts, J. R. (2007) · RePEc: Research Papers in Economics · Cited by 178
Certification Testing as an Illustration of Argument-Based Validation
Kane, M. (2004) · Measurement: Interdisciplinary Research & Perspective · doi:10.1207/s15366359mea0203_1 · Cited by 176
Paying for high- and low-quality teaching
Schacter, J., & Thum, Y. M. (2004) · Economics of Education Review · doi:10.1016/j.econedurev.2003.08.002 · Cited by 172
The Single Salary Schedule and Other Issues of Teacher Pay
Hanushek, E. A. (2007) · Peabody Journal of Education · doi:10.1080/01619560701602975 · Cited by 163
Hill, H. C. (2007) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707301711 · Cited by 161
Self-citation2007 Wallace Foundation Distinguished Lecture—What Makes Education Research “Educational”?
Ball, D. L., & Forzani, F. M. (2007) · Educational Researcher · doi:10.3102/0013189x07312896 · Cited by 159
An Empirical Comparison of Statistical Models for Value-Added Assessment of School Performance
Tekwe, C. D., Carter, R. L., Ma, C.-X., Algina, J., Lucas, M. E., Roth, J., Ariet, M., Fisher, T., & Resnick, M. B. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001011 · Cited by 155
Kimball, S. M., White, B., Milanowski, A. T., & Borman, G. (2004) · Peabody Journal of Education · doi:10.1207/s15327930pje7904_4 · Cited by 154
Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure?
Millman, J. (1997) · Medical Entomology and Zoology · Cited by 130
Principals as Agents: Subjective Performance Measurement in Education
Jacob, B., & Lefgren, L. (2005) · doi:10.3386/w11463 · Cited by 118 · cited 1×
“Other studies, including those that compare administrators’ ratings of teachers to value-added outcomes, return correlations of between .20 and .50 (Jacob & Lefgren, 2005; Kimball, White, & Milanowski, 2004; Medley & Coker, 1987; Milanowski, 2004).”
ResultLockwood, J. R., McCaffrey, D. F., Hamilton, L. S., Stecher, B., Le, V., & Martinez, J. F. (2007) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2007.00026.x · Cited by 105
Bayesian Methods for Scalable Multivariate Value-Added Assessment
Lockwood, J. R., McCaffrey, D. F., Mariano, L. T., & Setodji, C. (2007) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998606298039 · Cited by 87
Uncertainty in Rank Estimation: Implications for Value-Added Modeling Accountability Systems
Lockwood, J. R., Louis, T. A., & McCaffrey, D. F. (2002) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986027003255 · Cited by 85
Methodological issues in achieving school accountability1
Linn, R. L. (2008) · Journal of Curriculum Studies · doi:10.1080/00220270802105729 · Cited by 53
Multiple Resources, Multiple Outcomes: Testing the “Improved”School Finance With NELS88
Grubb, W. N. (2008) · American Educational Research Journal · doi:10.3102/0002831207308636 · Cited by 52
Impediments to the Estimation of Teacher Value Added
Ishii, J., & Rivkin, S. G. (2009) · Education Finance and Policy · doi:10.1162/edfp.2009.4.4.520 · Cited by 45
Identifying Effective Teachers Using Performance on the Job. Discussion Paper 2006-01.
Gordon, R. J., Kane, T. J., & Staiger, D. O. (2006) · Brookings Institution eBooks · Cited by 43
Comparisons Among Various Educational Assessment Value-Added Models
Sanders, W. L. (2006) · Cited by 34
Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools
Rockoff, J. E., Staiger, D. O., Kane, T. J., & Taylor, E. (2010) · National Bureau of Economic Research · doi:10.3386/w16240 · Cited by 20
Teachers' professional knowledge: Are we measuring acquired skills or common knowledge?
Miller, M. D., Poggio, J. P., & Glasnapp, D. R. (1987) · Journal of Personnel Evaluation in Education · doi:10.1007/bf00143279 · Cited by 16
Rockoff, J. E., Staiger, D. O., Kane, T. J., & Taylor, E. (2010) · National Bureau of Economic Research · Cited by 14
Cited by
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012) · Educational Researcher · doi:10.3102/0013189x12437203 · Cited by 331
Self-citationGrossman, P., Loeb, S., Cohen, J., & Wyckoff, J. (2013) · American Journal of Education · doi:10.1086/669901 · Cited by 271
Hill, H., & Grossman, P. (2013) · Harvard Educational Review · doi:10.17763/haer.83.2.d11511403715u376 · Cited by 255
Self-citationAn Argument Approach to Observation Protocol Validity
Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012) · Educational Assessment · doi:10.1080/10627197.2012.715014 · Cited by 236
Kraft, M. A., & Gilmour, A. F. (2016) · Educational Administration Quarterly · doi:10.1177/0013161x16653445 · Cited by 208
Metadata
- Authors
- Hill, H. C., Kapitula, L., & Umland, K.
- Venue
- American Educational Research Journal
- Published
- 2011-06
- Volume
- 48
- Issue
- 3
- Pages
- 794-831
- ISSN
- 0002-8312 (ISSN-L)
- 1935-1011
- OpenAlex
- W2128169318
- Cited by
- 269
- References
- 48
- FWCI
- 65.52
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.