Methodological Concerns About the Education Value-Added Assessment System
The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
Value-added models help to evaluate the knowledge that school districts, schools, and teachers add to student learning as students progress through school. In this article, the well-known Education Value-Added Assessment System (EVAAS) is examined. The author presents a practical investigation of the methodological issues associated with the model. Specifically, she argues that, although EVAAS is probably the most sophisticated value-added model, it has flaws that must be addressed before widespread adoption. She explores in depth the shortage of external reviews and validity studies of the model, its insufficient user-friendliness, and methodological issues about missing data, regression to the mean, and student background variables. She also examines a paradigm case in which the model was used to advance unfounded assertions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In addition to assuming that teacher effectiveness is wholly and fully captured by students' test scores-a clear threat to substantive validity [2]”
From Measurement and Fairness (2021) · cites this paper as [2]
In the citing paper
“Many value-added models, including the EVAAS MRM, lack convergent validity [2].”
From Measurement and Fairness (2021) · cites this paper as [2]
In the citing paper
“Many school districts have used such scores to make decisions about resource distribution and even teachers' continued employment, often without any way to contest these decisions [2,3].”
From Measurement and Fairness (2021) · cites this paper as [2]
Citations
Cites
EQUALITY OF EDUCATIONAL OPPORTUNITY
Coleman, J. S. (1968) · Equity & Excellence in Education · doi:10.1080/0020486680060504 · Cited by 5,308
Equality of Educational Opportunity.
Sewell, W. H., Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1967) · American Sociological Review · doi:10.2307/2091094 · Cited by 2,753
Standards for Educational and Psychological Testing.
García, E. J. (2017) · Revista Española de Pedagogía · doi:10.22550/2174-0909.3051 · Cited by 2,217
Handbook of Complementary Methods in Education Research
(2012) · Routledge eBooks · doi:10.4324/9780203874769 · Cited by 2,029
The relation between socioeconomic status and academic achievement.
White, K. R. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.91.3.461 · Cited by 1,650
Equality of Educational Opportunity.
Pfautz, H. W., Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1967) · American Sociological Review · doi:10.2307/2091096 · Cited by 1,278
Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation
Sanders, W. L., Wright, S. P., & Horn, S. P. (1997) · Journal of Personnel Evaluation in Education · doi:10.1023/a:1007999204543 · Cited by 1,044
Sanders, W. L., & Horn, S. P. (1998) · Journal of Personnel Evaluation in Education · doi:10.1023/a:1008067210518 · Cited by 590
Evaluating Value-Added Models for Teacher Accountability
McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003) · PsycEXTRA Dataset · doi:10.1037/e658712010-001 · Cited by 541
Models for Value-Added Modeling of Teacher Effects
McCaffrey, D. F., Lockwood, J. R., Koretz, D., Louis, T. A., & Hamilton, L. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001067 · Cited by 513
Controlling for Student Background in Value-Added Assessment of Teachers
Ballou, D., Sanders, W., & Wright, P. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001037 · Cited by 434
Sanders, W. L., & Horn, S. P. (1994) · Journal of Personnel Evaluation in Education · doi:10.1007/bf00973726 · Cited by 410
Goldhaber, D., & Anthony, E. (2007) · Review of Economics and Statistics · doi:10.1162/rest.89.1.134 · Cited by 403
Limitations in the Use of Achievement Tests as Measures of Educators' Productivity
Koretz, D. M. (2002) · The Journal of Human Resources · doi:10.2307/3069616 · Cited by 235
A Potential Outcomes View of Value-Added Assessment in Education
Rubin, D. B., Stuart, E. A., & Zanutto, E. L. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001103 · Cited by 233
Kim, J. S., & Sunderman, G. L. (2005) · Educational Researcher · doi:10.3102/0013189x034008003 · Cited by 229
What Are Value-Added Models Estimating and What Does This Imply for Statistical Practice?
Raudenbush, S. W. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001121 · Cited by 215
Is National Board Certification an Effective Signal of Teacher Quality
Cavalluzzo, L. C. (2004) · Cited by 211
Value-added indicators of school performance: A primer
Meyer, R. H. (1997) · Economics of Education Review · doi:10.1016/s0272-7757(96)00081-7 · Cited by 205
Gormley, W. T., & Weimer, D. L. (1999) · Harvard University Press eBooks · doi:10.2307/j.ctv22jnsnr · Cited by 202
National Board Certified Teachers andTheir Students' Achievement
Vandevoort, L. G., Amrein-Beardsley, A., & Berliner, D. C. (2004) · Education Policy Analysis Archives · doi:10.14507/epaa.v12n46.2004 · Cited by 201
Self-citationKupermintz, H. (2003) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737025003287 · Cited by 187
Sanders, W. L. (1998) · The School Administrator · Cited by 169
An Empirical Comparison of Statistical Models for Value-Added Assessment of School Performance
Tekwe, C. D., Carter, R. L., Ma, C.-X., Algina, J., Lucas, M. E., Roth, J., Ariet, M., Fisher, T., & Resnick, M. B. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001011 · Cited by 155
Complex by Design: Investigating Pathways Into Teaching in New York City Schools
Boyd, D. J., Grossman, P., Lankford, H., Loeb, S., Michelli, N. M., & Wyckoff, J. (2006) · Journal of Teacher Education · doi:10.1177/0022487105285943 · Cited by 141
Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure?
Millman, J. (1997) · Medical Entomology and Zoology · Cited by 130
Reflections on Statistical and Substantive Significance, with a Slice of Replication
Robinson, D. H., & Levin, J. R. (1997) · Educational Researcher · doi:10.2307/1176544 · Cited by 125
Sanders, W. L. (2000) · Journal of Personnel Evaluation in Education · doi:10.1023/a:1013008006096 · Cited by 118
Demographics and Diversity of Asian American College Students
Hune, S. (2002) · New Directions for Student Services · doi:10.1002/ss.35 · Cited by 113
The 39th Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes toward the Public Schools
Rose, L. C., & Gallup, A. M. (2007) · Phi Delta Kappan · doi:10.1177/003172170708900107 · Cited by 90
Sanders, W. L., Ashton, J. J., & Wright, S. P. (2005) · Cited by 83
Robinson, D. H., & Levin, J. R. (1997) · Educational Researcher · doi:10.3102/0013189x026005021 · Cited by 80
Research Synthesis: Effect Sizes
Thompson, B. (2006) · doi:10.4324/9780203874769.ch35 · Cited by 58
Topping, K. J., & Sanders, W. L. (2000) · School Effectiveness and School Improvement · doi:10.1076/0924-3453(200009)11:3;1-g;ft305 · Cited by 56
The Real World is More Complicated than We Would Like
Reckase, M. D. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001117 · Cited by 40
Two- and Three-Year Achievement Results From the Memphis Restructuring Initiative
Ross, S. M., Sanders, W. L., Wright, S. P., Stringfield, S., Wang, L. W., & Alberg, M. (2001) · School Effectiveness and School Improvement · doi:10.1076/sesi.12.3.323.3451 · Cited by 39
Marchant, G. J., Paulson, S. E., & Shunk, A. W. (2006) · Education Policy Analysis Archives · doi:10.14507/epaa.v14n30.2006 · Cited by 30
Using R for Estimating Longitudinal Student Achievement Models
Lockwood, J. R., III, Doran, H. C., & McCaffrey, D. F. (2003) · Cited by 19
Inside Systemic Elementary School Reform: Teacher Effects and Teacher Mobility
Ross, S. M., Stringfield, S., Sanders, W. L., & Wright, S. P. (2003) · School Effectiveness and School Improvement · doi:10.1076/sesi.14.1.73.13863 · Cited by 19
McCaffrey, D. F., Lockwood, J. R., III, Koretz, D. M., Louis, T. A., & Hamilton, L. S. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001139 · Cited by 18
Making the business our business
Lindgaard, G. (2004) · interactions · doi:10.1145/986253.986262 · Cited by 16
Value-Added Assessment: A View from a Practitioner
Andrejko, L. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001007 · Cited by 16
Value-added analysis and education policy
Rivkin, S. G. (2007) · PsycEXTRA Dataset · doi:10.1037/e722622011-001 · Cited by 16
Goldhaber, D. D., & Anthony, E. (2005) · PsycEXTRA Dataset · doi:10.1037/e723092011-001 · Cited by 15
Value-Added Student Assessment
Callender, J. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001005 · Cited by 13
Sanders, W. L. (2000) · Journal of Personnel Evaluation in Education · Cited by 11
Ross, S. M., Wang, L. W., Alberg, M., Sanders, W. L., Wright, S. P., & Stringfield, S. (2001) · Cited by 8
Carter, R. L. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029001135 · Cited by 1
Research works printed but not linked: 13
- Battelle for Kids, 2006, Achievement gains study of SOAR pilot and matched districts: Does SOAR make a difference? A summary of the final report to Battelle for Kids
- Braun HI, 2004, Value-added modeling: What does due diligence require?
- Lockwood JR, 2003, R News, 3, 17
- Morgan JP, 2002, Multiple choices: Testing students in Tennessee
- Northwest Evaluation Association, 2006, Achievement gaps: An examination of differences in student achievement and growth
- Ross SM, 2001, Paper presented at the annual meeting of the American Educational Research Association
- Sanders WL, 1998, The School Administrator, 55, 24
- SandersWLHow can value-added assessment lead to greater accountability?2004[Biographical sketch with article]. Report presented at First Annual Policy Conference of the New York State Educational Conference Board (Investment and Accountability for Student Success), Albany, NY. Retrieved February 10, 2007, fromhttp://www.nysecb.org/2004conference/04sanders.html
- Sanders WL, 1996, Cumulative and residual effects of teachers on future student academic achievement
- Sanders WL, 2002, Effects of building change on indicators of student achievement growth: Tennessee Value-Added Assessment System
- Stroup W, 1995, Assessment of the statistical methodology used in the Tennessee Value-Added Assessment System
- Walberg HJ, 1997, Grading teachers, grading schools: Is student achievement a valid evaluation measure?, 169
- 10.1037/0033-2909.91.3.461 · DOI · Paper page
Cited by
Neuman, S. B., & Cunningham, L. (2009) · American Educational Research Journal · doi:10.3102/0002831208328088 · Cited by 522
“Students of National Board–certified teachers, for example, make about 1 month’s greater growth than students of regularly certified teachers (Amrein-Beardsley, 2008).”
BackgroundApproaches to Evaluating Teacher Effectiveness: A Research Synthesis.
Goe, L., Bell, C. A., & Little, O. M. (2008) · Cited by 673
What Makes Great Teaching? Review of the Underpinning Research.
Coe, R., Aloisi, C., Higgins, S., & Major, L. E. (2014) · Durham Research Online (Durham University) · Cited by 660
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362
“r teaching history and status in the community. If these attributes are relevant to our understanding of teacher quality, then this model lacks content validity as well. Furthermore, Amrein-Beardsley [3] demonstrates the score’s lack of construct validity|particularly the lack of convergent validity and content validi…”
BackgroundMethodA Validity Argument Approach to Evaluating Teacher Value-Added Scores
Hill, H. C., Kapitula, L., & Umland, K. (2011) · American Educational Research Journal · doi:10.3102/0002831210387916 · Cited by 269
Metadata
- Authors
- Amrein-Beardsley, A.
- Affiliations
- AUDREY AMREIN-BEARDSLEY is an assistant professor at Arizona State University, College of Teacher Education and Leadership, P.O. Box 37100, Phoenix, AZ 85069; . Her areas of interest are educational policy, research methods, and tests and assessments
- Venue
- Educational Researcher
- Published
- 2008-03
- Volume
- 37
- Issue
- 2
- Pages
- 65-75
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2113233414
- Cited by
- 199
- References
- 48
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 7 · 2023: 5 · 2022: 3 · 2021: 8 · 2020: 19 · 2019: 8 · 2018: 14 · 2017: 14 · 2016: 17 · 2015: 9 · 2014: 18 · 2013: 19 · 2012: 18
- FWCI
- 26.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Education › Educational Assessment and Improvement
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- value-added models, student achievement improvement, school accountability, teacher evaluation, external validation, missing data, regression to the mean, student background, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.