CONSIDERATIONS IN THE CHOICE OF INTEROBSERVER RELIABILITY ESTIMATES
Open accessAbstract
Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data‐acquisition system: trial reilability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and “correlational” measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter-rater reliability was 90.1% percentage agreement, demonstrating high agreement between raters.
“Our percentage agreement was 90.1%, demonstrating a high level of agreement between raters [31,35].”
From Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users (2023) · cites this paper as [35]
Cites
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093 · see citing works
A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder
Fabiano, G. A., Pelham, W. E., Coles, E. K., Gnagy, E. M., Chronis-Tuscano, A., & O'Connor, B. C. (2009) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.11.001 · Cited by 588 · see citing works
The integrity of independent variables in behavior analysis
Peterson, L., Homer, A. L., & Wonderlich, S. A. (1982) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1982.15-477 · Cited by 461 · see citing works
Social escape behaviors in children with fragile X syndrome
Hall, S., DeBernardis, M., & Reiss, A. (2006) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-006-0132-z · Cited by 107 · see citing works
Understanding Screen-Reader Users’ Experiences with Online Data Visualizations
Sharif, A., Chintalapati, S. S., Wobbrock, J. O., & Reinecke, K. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471202 · Cited by 85 · see citing works
Supporting Accessible Data Visualization Through Audio Data Narratives
Siu, A., S-H Kim, G., O'Modhrain, S., & Follmer, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517678 · Cited by 64 · see citing works
Floyd, F. J., O'Farrell, T. J., & Goldberg, M. (1987) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.55.3.2200 · Cited by 34 · see citing works
Sharif, A., Zhang, A. M., Reinecke, K., & Wobbrock, J. O. (2023) · 20th International Web for All Conference · doi:10.1145/3587281.3587284 · Cited by 21 · see citing works
“1%, demonstrating a high level of agreement between raters [31, 35].”
BackgroundMethod
- Authors
- Hartmann, D. P.
- Affiliations
- University of Utah (US)
- Corresponding author
- Hartmann, Donald P.
- Venue
- Journal of Applied Behavior Analysis
- Publisher
- Wiley
- Published
- 1977-03
- Volume
- 10
- Issue
- 1
- Pages
- 103-116
- ISSN
- 0021-8855 (ISSN-L)
- 1938-3703
- PMC
- PMC1311156
- OpenAlex
- W1983196970
- Cited by
- 762
- References
- 2
- Citations by year
- 2026: 7 · 2025: 13 · 2024: 9 · 2023: 16 · 2022: 17 · 2021: 46 · 2020: 20 · 2019: 14 · 2018: 17 · 2017: 9 · 2016: 11 · 2015: 8 · 2014: 23 · 2013: 13 · 2012: 15
- FWCI
- 57.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Applied Behavior Analysis (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Psychology › Developmental and Educational Psychology › Behavioral and Psychological Studies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- inter-rater reliability, treatment studies, reliability estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
