CoChat

When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching

Weston, Timothy J., Hayward, Charles N., Laursen, Sandra L. · 2021 · American Journal of Evaluation

The DOI matches its Crossref record; 60 of the 63 references it cites were found in a registry or an index.

Abstract

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.

Overview

Integrity signals

Registry

Crossref · 10.1177/1098214020931941

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 60 references

Research works printed but not linked: 8

  1. Brewer C., 2011, Vision and change in undergraduate biology education: A call to action
  2. Budd D. A., 2013, Journal of Geoscience Education, 61, 461
  3. Hayward C. N., 2018, Proceedings of the 21st annual conference on the research in undergraduate mathematics education, 727
  4. Kane T. J., 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
  5. Nadelson L. S., 2013, Learning Communities Journal, 5, 97
  6. Nolan B. A., 2002, Alzheimer’s Care Today, 3, 46
  7. Pilburn M., 2000, Reformed Teaching Observation Protocol (RTOP)
  8. Weisberg D., Sexton S., Mulhern J., Keeling D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness (2nd ed.). The New Teacher Project. https://tntp.org/assets/documents/TheWidgetEffect_2nd_ed.pdf

Cited by

See all 20

Metadata

Authors
Weston, T. J., Hayward, C. N., & Laursen, S. L.
Affiliations
University of Colorado Boulder (US)
Corresponding author
Weston, Timothy J.
Venue
American Journal of Evaluation
Published
2021-07-20
Volume
42
Issue
3
Pages
377-398
ISSN
  • 1098-2140 (ISSN-L)
  • 1557-0878
OpenAlex
W3186113926
Cited by
20
References
63
Citations by year
2026: 3 · 2025: 6 · 2024: 1 · 2023: 2 · 2022: 6 · 2021: 2
FWCI
4.83
Type
journal article
Language
en
Open access
closed
Locations
American Journal of Evaluation (publishedVersion)
Funders
National Science Foundation (1245436)
Topics
  • Social Sciences › Education › Evaluation of Teaching Practices
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Education › Educational Research and Methods
Keywords
classroom observation, generalizability theory, reliability statistics, teaching practice, sample size, inter-rater reliability, educational assessment, research on teaching, observational data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching. https://app.cochat.ai/papers/10.1177/1098214020931941
The paper
Weston, T. J., Hayward, C. N., & Laursen, S. L. (2021). When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching. American Journal of Evaluation. https://doi.org/10.1177/1098214020931941

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.