An Argument Approach to Observation Protocol Validity
Abstract
This article develops a validity argument approach for use on observation protocols currently used to assess teacher quality for high-stakes personnel and professional development decisions. After defining the teaching quality domain, we articulate an interpretive argument for observation protocols. To illustrate the types of evidence that might compose a validity argument, we draw on data from a validity study of the Classroom Assessment Scoring System for secondary classrooms. Based on data from 82 Algebra classrooms, we illustrate how data from observation scores, value-added models, generalizability studies, and measures of teacher knowledge, student achievement, and teacher and student beliefs could be used to build a validity argument for observation protocols. Strengths and limitations of the validity argument approach as well as the issues the approach raises for observation protocol validity research are considered.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Establishing the validity of inferences based on observation protocol scores requires a multifaceted array of evidence.
“Establishing the validity of inferences based on observation protocol scores requires a multifaceted array of evidence, and studies along these lines have become more common in recent years.”
From Making Inferences About Teacher Observation Scores Over Time (2019) · cites this paper as Bell et al., 2012
In prior studies using generalizability theory to decompose variance in observation protocol scores, there is an important hidden facet to the measurement procedure: the occasion or temporal location of the observation within the academic school year.
“However, in all of these studies, it can be argued that there is an important ''hidden'' facet to the measurement procedure involved in the scoring of a classroom observation. This facet is the occasion or, more specifically, the temporal location of the observation within the academic school year.”
From Making Inferences About Teacher Observation Scores Over Time (2019) · cites this paper as Bell et al., 2012
Cites
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007) · Child Development · doi:10.1111/j.1467-8624.2007.00995.x · Cited by 3,784 · see citing works
The Relation Between Self-Beliefs and Academic Achievement: A Meta-Analytic Review
Valentine, J. C., DuBois, D. L., & Cooper, H. (2004) · Educational Psychologist · doi:10.1207/s15326985ep3902_3 · Cited by 1,315 · see citing works
Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008) · Cognition and Instruction · doi:10.1080/07370000802177235 · Cited by 996 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Classroom observation systems in context: A case for the validation of observation systems
Liu, S., Bell, C. A., Jones, N. D., & McCaffrey, D. F. (2019) · Educational Assessment, Evaluation and Accountability · doi:10.1007/s11092-018-09291-3 · Cited by 46 · see citing works
Self-citationRater Performance Standards for Classroom Observation Instruments
White, M. C. (2018) · Educational Researcher · doi:10.3102/0013189x18785623 · Cited by 32 · see citing works
Using Classroom Observations in the Evaluation of Special Education Teachers
Jones, N. D., Bell, C. A., Brownell, M., Qi, Y., Peyton, D., Pua, D., Fowler, M., & Holtzman, S. (2022) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737211068523 · Cited by 21 · see citing works
Self-citationMaking Inferences About Teacher Observation Scores Over Time
Briggs, D. C., & Alzen, J. L. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164419826237 · Cited by 14 · see citing works
- Authors
- Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y.
- Affiliations
- Educational Testing Service (US)
- Rutgers, The State University of New Jersey (US)
- RAND Corporation (US)
- University of Virginia (US)
- Venue
- Educational Assessment
- Publisher
- Informa UK Limited
- Published
- 2012-04
- Volume
- 17
- Issue
- 2-3
- Pages
- 62-87
- ISSN
- 1062-7197 (ISSN-L)
- 1532-6977
- OpenAlex
- W2085259499
- Cited by
- 236
- References
- 3
- Citations by year
- 2026: 10 · 2025: 13 · 2024: 17 · 2023: 18 · 2022: 23 · 2021: 15 · 2020: 17 · 2019: 28 · 2018: 22 · 2017: 23 · 2016: 15 · 2015: 10 · 2014: 12 · 2013: 11 · 2012: 2
- FWCI
- 54.21
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Assessment (publishedVersion)
- Topics
- Social Sciences › Education › Teacher Education and Assessments
- Social Sciences › Decision Sciences › Information Systems and Management › Educational Assessment and Improvement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- observation protocol, teacher quality assessment, validity argument, Classroom Assessment Scoring System, value-added models, teacher knowledge, academic performance, teacher beliefs, student beliefs, secondary education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
