The DOI matches its DataCite record, with a small difference in title or year; 43 of the 44 references it cites were found in a registry or an index; 3 of the 25 sources its claims cite were checked for support.
Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability
Open accessAbstract
Although inter-rater reliability is an important aspect of using observational instruments, it has received little theoretical attention. In this article, we offer some guidance for practitioners and consumers of classroom observations so that they can make decisions about inter-rater reliability, both for study design and in the reporting of data and results. We reviewed articles in two major journals in the fields of reading and mathematics to understand how researchers have measured and reported inter-rater reliability in a recent decade. We found that researchers have tended to report measures of inter-rater agreement above the .80 threshold with little attention to the magnitude of score differences between raters. Then, we conducted simulations to understand both how different indices for classroom observation reliability are related to each other and the impact of reliability decisions on study results. Results from the simulation studies suggest that mean correlations with an outcome are slightly lower at lower levels of percentage of exact agreement but that the magnitude of score differences has a more dramatic effect on correlations. Therefore, adhering to strict thresholds for inter-rater agreement is less helpful than reporting exact point estimates and also examining measures of rater consistency.
Overview
CoChat Summary
TL;DR
The paper reviews how classroom observation inter-rater reliability (IRR) was measured and reported in 17 studies from JRME and RRQ (2007–2016), then uses Monte Carlo simulations to show how different IRR indices and scoring decisions affect results. It argues that researchers should report exact IRR point estimates and both consensus and consistency indices, because the magnitude of rater disagreement—not just whether agreement exceeds .80—can substantially reduce correlations with outcomes.
Key findings
- In the review, only 17 studies met inclusion criteria (8 from JRME, 9 from RRQ); the authors note this was 3.4% of studies in the two journals. Only 7 of 17 studies reported both training reliability and data-collection reliability. In JRME, 3 of 8 studies reported training reliability and 6 of 8 reported data-collection reliability; in RRQ, 7 of 9 reported training reliability and 6 of 9 reported data-collection reliability.
- Simulation results showed that the magnitude of disagreement matters. At 60% exact agreement, off-by-1 disagreement yielded kappa = .459, rho = .830, alpha = .900; off-by-2 disagreement yielded kappa = .470, rho = .312, alpha = .502. At 90% exact agreement, off-by-1 yielded kappa = .862, rho = .954, alpha = .975; off-by-2 yielded kappa = .863, rho = .809, alpha = .896. Consistency indices (rho and alpha) were always above .8 and greater than consensus indices when scores were off by 1, but lower than percent agreement when scores were off by 2.
- For the simulated outcome, with an expected correlation of .4, mean correlations between classroom observation scores and simulated student achievement ranged from .317 to .398 across exact-agreement levels, with individual correlations from .041 to .594. The mean correlation at 60% exact agreement off by 1 (r = .381) was nearly the same as at 90% exact agreement off by 2 (r = .383); one example had an expected correlation of .400 but an actual correlation of .148 at 60% exact agreement off by 2.
- The authors recommend reporting a validation argument for the instrument, training IRR, and data-collection IRR; using both a consensus index (e.g., Cohen’s kappa) and a consistency index (e.g., correlation); and reporting exact point estimates rather than only whether reliability exceeded a threshold such as .80.
Methods
- Systematic review: articles published 2007–2016 in JRME (272 articles) and RRQ (230 articles) were screened for inclusion if they used a classroom observational instrument measuring student and/or teacher behavior, used quantitative data analysis, and examined classroom observation data as an independent variable. Seventeen studies met criteria (8 JRME, 9 RRQ); coding was double-coded and disagreements resolved by consensus.
- Monte Carlo simulation: a hypothetical instrument with 10 rubrics scored 0–4 (mean 2.5) was simulated for 100 classrooms/observations, repeated 100 times. Data were generated at 60%, 70%, 80%, and 90% exact agreement, with disagreements either off by 1 or off by 2; Cohen’s kappa, Spearman’s rho, and Cronbach’s alpha were calculated.
- For the outcome analysis, simulated classroom observation scores were averaged across raters and rubrics and correlated with simulated student achievement averages designed to have a mean correlation of .4; differences in correlations were interpreted as the impact of IRR decisions.
Limitations
- Stated: the review examined only two journals (JRME and RRQ) over 10 years, so other education journals may differ in prevalence and reporting of classroom observation IRR; the intent was to capture current practice, not conduct a comprehensive literature review.
- Stated: the simulation used a hypothetical instrument resembling but not identical to well-known instruments, and it did not capture other observation approaches such as time sampling or frequency counts, nor the decision of which instrument to use.
- Evident from the text: the simulation findings depend on assumed normally distributed ordinal scores, 10 rubrics, averaging across raters and rubrics, and a simulated achievement outcome; they are not validated against real classroom observation data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 0 Not valid · 17 Not checked · 43 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet, despite the different foci on consensus or consistency via the different indices, there seems to be a standard practice of taking a reliability (or agreement) level of .7 or .8 as sufficient for research (Lance, Butts, & Michels, 2006;McHugh, 2012).”
§ Exploring Differences in Measurement and Reporting of Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability Classroom Observation Inter-Rater Reliability > Statistics and interpretation guidelines.
What the sources say
- Not matchedcites Lance, Butts, & Michels, 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites McHugh, 2012 sourceExperimental — accuracy not yet measured
In “Theoretical issues in measurement of rater reliability” (page not available), McHugh says many texts recommend 80% as the minimum acceptable interrater agreement. The paper does not establish 0.7 as a standard sufficient threshold across reliability or agreement measures; its proposed kappa interpretation instead calls values below 0.60 inadequate. (LLM, conf=0.85)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that researchers have tended to report measures of inter-rater agreement above the .80threshold with little attention to the magnitude of score differences between raters.”
§ Exploring Differences in Measurement and Reporting of Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability Classroom Observation Inter-Rater Reliability > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we were struck by the relatively small percentage of studies (3.4%) that utilized data from quantitative classroom observation tools as a variable in quantitative analyses.”
§ Exploring Differences in Measurement and Reporting of Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability Classroom Observation Inter-Rater Reliability > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results from the simulation studies suggest that mean correlations with an outcome are slightly lower at lower levels of percentage of exact agreement but that the magnitude of score differences has a more dramatic effect on correlations.”
- Also stated in: § Research Question 3: Understanding the Impact of Classroom Observation Scoring and Reliability Decisions · plain
“Further, this trend was exacerbated in the case where scores differed by 2 points.”
§ Exploring Differences in Measurement and Reporting of Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability Classroom Observation Inter-Rater Reliability > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Research Question 3: Understanding the Impact of Classroom Observation Scoring and Reliability Decisions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results from the simulation studies suggest that mean correlations with an outcome are slightly lower at lower levels of percentage of exact agreement but that the magnitude of score differences has a more dramatic effect on correlations.”
- Also stated in: § Research Question 3: Understanding the Impact of Classroom Observation Scoring and Reliability Decisions · plain
“When comparing correlations at different percentages of exact agreement, mean correlations decreased as percentages of exact agreement decreased.”
§ Exploring Differences in Measurement and Reporting of Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability Classroom Observation Inter-Rater Reliability > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Research Question 3: Understanding the Impact of Classroom Observation Scoring and Reliability Decisions · plain
30 more claims
Create a free account to see all 62 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 19
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- [3]3. Blood, E. & Spratt, K. F. (2007). Disagreement on agreement: Two alternate agreement coefficients. SAS Global forum 2007. Retrieved from: http://www2.sas.com/proceedings/forum2007/186- 2007.pdf
- [8]8. Wilhelm, Gillespie Rouse & Jones, Classroom Observation Inter-Rater Reliability Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284-290.
- [11]11. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, New Jersey: Lawrence Erlbaum Associates.
- [16]16. Fleiss, J. (1981). Statistical methods for rates and proportions. New York: Wiley.
- [20]20. Gwet, K. L. (2002). Kappa statistics is not satisfactory for assessing the extent of agreement between raters. Statistical Methods for Inter-Rater Reliability, 1(6), 1-5.
- [21]21. Gwet, K. L. (2012). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters. Third Edition. Gaithersburg, MD: Advanced Analytics, LLC.
- [22]22. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data. Tutor Quantitative Methods Psychology, 8(1), 23-34.
- [23]23. Hartmann, D. P., Barrios, B. A., & Wood, D. D. (2004). Principles of behavioral observation. In M. Hersen (Ed.), Comprehensive Handbook of Psychological Assessment (Vol. 3, pp. 108-137).
- [24]24. Hoboken, N. J.: John Wiley & Sons.
- [30]30. Jonnson, A. & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity, and educational consequences. Educational Research Review 2(2), 130-144.
- [32]32. Kazdin, A. E. (1982). Single case research designs: Methods for clinical and applied settings. New York: Oxford Press.
- [33]33. Wilhelm, Gillespie Rouse & Jones, Classroom Observation Inter-Rater Reliability Kelcey, B., & Carlisle, J. F. (2013). Learning about teachers' literacy instruction from classroom observations. Reading Research Quarterly, 48(3), 301-317.
- [35]35. Kelsey, B. & Carlisle, J. F., (2013). Learning about teacher' literacy instruction from classroom observations. Reading Research Quarterly, 48(3), 301-317.
- [37]37. Krippendorff, K. (2013). Content analysis. An introduction to its Methodology (3rd ed.). Thousand Oaks, CA: Sage.
- [39]39. Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria what did they really say? Organizational Research Methods, 9(2), 202-220.
- [54]54. Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approached to estimating integrated reliability. Practical Assessment, Research & Evaluation, 9(4), 1-11.
- [56]56. Strong, M. (2011). The highly qualified teacher: what is teacher quality and how do we measure it? NY: New York: Teachers College Press.
- [59]59. Wilhelm, Gillespie Rouse & Jones, Classroom Observation Inter-Rater Reliability Implementation. Learning Disabilities Quarterly, 35(2), 115-126.
- [62]62. Vaughn, S., & Briggs, K. L., (Eds.) (2003). Reading in the classroom: Systems for the observation of teaching and learning. Baltimore: Brookes Publishing.
Cited by
Grabner-Hagen, M. M., & Kingsley, T. L. (2023) · Computers and Education Open · doi:10.1016/j.caeo.2023.100131 · Cited by 89
Kundu, A., Bej, T., & Rice, M. F. (2020) · Education and Information Technologies · doi:10.1007/s10639-020-10306-0 · Cited by 83
Suresh, A., Jacobs, J., Lai, V., Tan, C., Ward, W., Martin, J., & Sumner, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.07949 · Cited by 54
Beckler, D. T., Thumser, Z. C., Schofield, J. S., & Marasco, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0606-7 · Cited by 28
White, M., & Ronfeldt, M. (2024) · Educational Assessment · doi:10.1080/10627197.2024.2354311 · Cited by 29
Metadata
- Authors
- Wilhelm, A. G., Rouse, A. G., & Jones, F.
- Venue
- University of Massachusetts Amherst
- Publisher
- University of Massachusetts Amherst
- Published
- 2018
- Volume
- 23
- Issue
- 4
- Pages
- 4
- OpenAlex
- W2799770324
- Cited by
- 20
- References
- 0
- Citations by year
- 2025: 1 · 2024: 5 · 2023: 4 · 2022: 4 · 2021: 3 · 2020: 1 · 2019: 1 · 2018: 1
- FWCI
- 1.83
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by-nc-nd
- Locations
- Scholarworks (University of Massachusetts Amherst) (submittedVersion, cc-by-nc-nd)
- University of Massachusetts (UMass) Amherst
- Practical assessment, research & evaluation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Research and Methods
- Keywords
- inter-rater reliability, classroom observation, observation instruments, reporting practices, rater agreement, rater consistency, reliability threshold, simulation study, reading research, mathematics research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability. https://app.cochat.ai/papers/10.7275/at67-md25
- The paper
- Wilhelm, A. G., Rouse, A. G., & Jones, F. (2018). Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability. University of Massachusetts Amherst. https://doi.org/10.7275/at67-md25
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.



