Interobserver agreement, reliability, and generalizability of data collected in observational studies.

Abstract

Research in developmental and educational psychology has come to rely less on conventional psychometric tests and more on records of behavior made by human observers in natural and quasi-natural settings. Three coefficients that purport to reflect the quality of data collected in these observational studies are dis-cussed: the interobserver agreement percentage, the reliability coefficient, and the generalizability coefficient. It is concluded that although high interobserver agreement is desirable in observational studies, high agreement alone is not sufficient to insure the quality of the data that are collected. Evidence of the reliability or generalizability of the data should also be reported. Further ad-vantages of generalizability designs are discussed. Almost everyone engaged in research recog-nizes the need for reliable measuring instru-ments. Reliability is a central topic particu-larly for courses and textbooks concerned with the behavioral sciences. In spite of vary-ing theoretical derivations, its definition is remarkably uniform: A reliable instrument is one with small errors of measurement, one that shows stability, consistency, and depend-ability of scores for individuals on the trait, characteristic, or behavior being assessed. The preparation of this article was supported in

Cites (1)

1. Convergent and discriminant validation by the multitrait-multimethod matrix.
   Campbell, Donald T., Fiske, Donald W. · 1959 · Psychological Bulletin

Show all 1 references

Cited by (20)

1. Estimating within-group interrater reliability with and without response bias
   James, Lawrence R., Demaree, Robert G., Wolf, Gerrit · 1984 · Journal of Applied Psychology
2. rwg: an assessment of within-group interrater agreement
   James, Lawrence R., Demaree, Robert G., Wolf, Gerrit · 1993 · Journal of Applied Psychology
3. Reliability of Nominal Data Based on Qualitative Judgments
   Perreault, William D., Leigh, Laurence E. · 1989 · Journal of Marketing Research
4. A longitudinal study of joint attention and language development in autistic children
   Mundy, Peter, Sigman, Marian, Kasari, Connie · 1990 · Journal of Autism and Developmental Disorders
5. Loneliness and friendship in high-functioning children with autism
   Bauminger, Nirit, Kasari, Connie · 2000 · Child Development

See all 445