Balanced Incomplete Block Designs for Inter-Rater Reliability Studies
Open accessAbstract
Occasionally, an inter-rater reliability study must be designed so that each subject is rated by fewer than all the participating raters. If there is interest in comparing the raters' mean levels of rating, and if it is desired that each mean be estimated with the same precision, then a balanced incomplete block design for the reliability study is indicated. Methods for executing the design and for analyzing the resulting data are presented, using data from an actual study for illustration.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
According to Fleiss (1981), kappa coefficients above 0.75 are regarded as excellent agreement indicators.
“According to Fleiss (1981), kappa coefficients above 0.75 are regarded as excellent agreement indicators.”
From Young Children's Epistemic Operations and Emergent Argumentation in Inquiry‐Based Discussions: The Role of Teacher Questioning and Collaborative Interactions (2025) · cites this paper as Fleiss (1981)
The Analysis of Subject Effects section discusses an intraclass correlation coefficient and describes an estimate as indicating good reliability. A search of our copy for kappa, the 0.75 threshold, and an ‘excellent agreement’ interpretation found no passage addressing the claimed rule. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Cochran, W. G., & Cox, G. M. Experimental designs (2nd ed.). New York: Wiley, 1957.
- Feldt, L. S. The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty. Psychometrika , 1965, 30 , 357-370.
- Scheffé, H. The analysis of variance. New York: Wiley, 1959.
- Wald, A. On the analysis of variance in case of multi- ple classifications with unequal class frequencies. Annals of Mathematical Statistics , 1941, 12 , 346-350.
- Yates, F. Incomplete randomised blocks. Annals of Eugenics, 1936, 7 , 121-140.
Cited by
Sorted by citations
Guidelines for establishing reliability when coding narrative data
Syed, M., & Nelson, S. C. (2015) · Emerging Adulthood · doi:10.1177/2167696815587648 · Cited by 801 · see citing works
Feldner, M. T., Zvolensky, M. J., Eifert, G. H., & Spira, A. P. (2003) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(02)00020-7 · Cited by 325 · see citing works
Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.76.4.544 · Cited by 284 · see citing works
Hill, C. E., O'Grady, K. E., & Elkin, I. (1992) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.60.1.73 · Cited by 190 · see citing works
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0548-2 · Cited by 122 · see citing works
Assessing fidelity in individual and family therapy for adolescent substance abuse
Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., Reiner, R. H., & Liddle, H. A. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2007.09.002 · Cited by 78 · see citing works
Hogue, A., Liddle, H. A., Dauber, S., & Samuolis, J. (2004) · Psychotherapy: Theory, Research, Practice, Training · doi:10.1037/0033-3204.41.2.83 · Cited by 54 · see citing works
Hogue, A., Bobek, M., Dauber, S., Henderson, C. E., McLeod, B. D., & Southam-Gerow, M. A. (2019) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2018.1555762 · Cited by 53 · see citing works
Reliability of Therapist Self-Report on Treatment Targets and Focus in Family-Based Intervention
Hogue, A., Dauber, S., Henderson, C. E., & Liddle, H. A. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-013-0520-6 · Cited by 52 · see citing works
Atzil–Slonim, D., Wiseman, H., & Tishby, O. (2016) · Psychotherapy Research · doi:10.1080/10503307.2015.1010627 · Cited by 9 · see citing works
Caño, L., & Sanz, J. (2025) · Journal of Research in Science Teaching · doi:10.1002/tea.70010 · Cited by 5 · see citing works
- Authors
- Fleiss, J. L.
- Affiliations
- New York State Psychiatric Institute (US)
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Applied Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 1981-01
- Volume
- 5
- Issue
- 1
- Pages
- 105-112
- ISSN
- 0146-6216 (ISSN-L)
- 1552-3497
- OpenAlex
- W1999460227
- Cited by
- 313
- References
- 1
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 16 · 2023: 13 · 2022: 16 · 2021: 20 · 2020: 19 · 2019: 15 · 2018: 18 · 2017: 23 · 2016: 19 · 2015: 12 · 2014: 8 · 2013: 11 · 2012: 7
- FWCI
- 0.92
- Type
- journal article
- Language
- en
- Open access
- green
- purl.umn.edu
- Locations
- Applied Psychological Measurement (publishedVersion)
- University of Minnesota Digital Conservancy (University of Minnesota) (submittedVersion)
- Funders
- National Institute of Mental Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- balanced incomplete block designs, inter-rater reliability, experimental design, reliability data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
