The DOI matches its Crossref record; the references it cites are not checked yet.
The Generalizability of Systematic Direct Observations Across Time and Setting: A Preliminary Investigation of the Psychometrics of Behavioral Observation
Abstract
This study examined the generalizability of systematic direct observation across setting and time. Participants included 14 students from an intact inclusionary fifth grade classroom. On-task/off-task behavior was directly observed using momentary time-sampling recording, twice a day, for 10 school days. Using Generalizability (G) theory, results suggested that of the total amount of variance present, individual variation among participants accounted for 62% of the observed variance and unexplained error 24%. Although adequate levels of interobserver agreement were attained, resulting reliability coefficients of the observational procedures were in the low.60s. Results are discussed in terms of both behavioral and classic measurement theory and implications for practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interrater agreement assesses the degree of agreement or consensus between raters and provides no information about the alignment of those ratings with the phenomena of interest.
“Interrater agreement is a dimension of reliability and assesses the degree of agreement or consensus between raters (Hintze & Matthews, 2004;Kazdin, 1982;Stemler, 2004); it provides no information about the alignment of those ratings with the phenomena of interest (e.g., the occurrence of the behavior) (Hintze & Matthews, 2004).”
From Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability (2018) · cites this paper as Hintze & Matthews, 2004
Correlation-based measures of consistency better account for the magnitude of disagreement between raters, whereas measures of consensus better account for the agreement between raters.
“For example, we found that correlation-based measures of consistency better account for the magnitude of disagreement between raters, whereas the measures of consensus better account for the agreement between raters (Hintze & Matthews, 2004).”
From Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability (2018) · cites this paper as Hintze & Matthews, 2004
Citations
The citation lists haven’t been fetched yet
References 19 · Cited by 127
Research works printed but not linked: 25
- Alessi, G. (1988). Direct observation methods for emotional/behavior problems. In E. S. Shapiro & T. R. Kratochwill (Eds.), Behavioral assessment in schools: Conceptual foundations and practical applications (pp. 14–75). New York: Guilford Press.
- Ary D., 1984, Behavioral Assessment, 6, 221
- Butler A., 1990, Validation of a classroom observation code for behavior disordered and learning disabled students.
- 10.1016/S0005-7894(77)80077-4 · DOI
- 10.1016/S0005-7894(78)80020-3 · DOI
- Cone J. D., 1981, Behavioral assessment: A practical approach, 38
- Cone J. D., 1988, Behavioral assessment: A practical approach, 42
- Cronbach L. J., 1972, The dependability of behavioral measurements.
- Fleiss J. L., 1981, Statistical methods for rates and proportions.
- 10.1007/978-1-4613-0905-5_2 · DOI
- 10.1037/h0088778 · DOI · Paper page
- Hintze J. M., 2002, Best practices in school psychology IV, 993
- Individuals with Disabilities Education Act Amendments of 1997 (PL 105–17). 20 USC Chapter 33, (Sections 1400 et seq.). (Statute)
- Johnston J. M., 1993, Strategies and tactics of behavioral research
- Kazdin A. E., 1982, Single-case research designs: Methods for clinical and applied settings.
- Merrell K. W., 2002, Behavioral, social, and emotional assessment of children and adolescents
- 10.1016/S0005-7894(77)80078-6 · DOI
- Salvia J., 2004, Assessment
- Shadish W. R., 2002, Experimental and quasi-experimental designs for generalized causal inference.
- Shapiro E. S., 1996, Academic skills problems workbook.
- Skinner C. H., 2000, Behavioral assessment in schools: Theory, research, and clinical foundations, 19
- Suen H. K., 1990, Principles of test theories.
- Suen H. K., 1989, Analyzing quantitative behavioral observation data.
- Wilson M. S., 1996, School Psychology Review, 25, 9, 10.1080/02796015.1996.12085799 · DOI · Paper page
- 10.1097/00004703-199806000-00003 · DOI · Paper page
Metadata
- Authors
- Hintze, J. M., & Matthews, W. J.
- Affiliations
- University of Massachusetts Amherst (US)
- Corresponding author
- Hintze, John M.
- ORCID
- Matthews, William J.
- Venue
- School Psychology Review
- Publisher
- Informa UK Limited
- Published
- 2004-06-01
- Volume
- 33
- Issue
- 2
- Pages
- 258-270
- ISSN
- 0279-6015 (ISSN-L)
- 2372-966X
- OpenAlex
- W1480929733
- Cited by
- 127
- References
- 19
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 3 · 2023: 1 · 2022: 6 · 2021: 6 · 2020: 5 · 2019: 4 · 2018: 6 · 2017: 11 · 2016: 8 · 2015: 5 · 2014: 14 · 2013: 10 · 2012: 2
- FWCI
- 7.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- School Psychology Review (publishedVersion)
- ScholarWorks@UMassAmherst (University of Massachusetts Amherst) (submittedVersion)
- Scholarworks (University of Massachusetts Amherst) (submittedVersion)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Behavioral and Psychological Studies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- generalizability theory, on-task behavior, off-task behavior, interobserver agreement, reliability coefficients, measurement error, individual differences, fifth grade students, inclusive classrooms, behavioral observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
