When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
The DOI matches its Crossref record; 60 of the 63 references it cites were found in a registry or an index.
Abstract
Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
Active learning increases student performance in science, engineering, and mathematics
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1319030111 · Cited by 9,787
SAGE Handbook of Mixed Methods in Social & Behavioral Research
Tashakkori, A., & Teddlie, C. (2010) · doi:10.4135/9781506335193 · Cited by 6,215
Hake, R. R. (1998) · American Journal of Physics · doi:10.1119/1.18809 · Cited by 6,072
The Journal of College Science Teaching
(1971) · Journal of Chemical Education · doi:10.1021/ed048p284 · Cited by 3,332
The ICAP framework: Linking cognitive engagement to active learning outcomes
Chi, M. T. H., & Wylie, R. (2014) · Educational Psychologist · doi:10.1080/00461520.2014.965823 · Cited by 2,911
Where's the evidence that active learning works?
Michael, J. (2006) · Advances in Physiology Education · doi:10.1152/advan.00053.2006 · Cited by 1,648
Pianta, R. C., & Hamre, B. K. (2009) · Educational Researcher · doi:10.3102/0013189x09332374 · Cited by 1,331 · cited 1×
“However, the products of structured teacher observations are decidedly quantitative (Pianta & Hamre, 2009).”
BackgroundGuidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audige, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M., & Streiner, D. L. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.01.016 · Cited by 997
Journal of the College of Physicians and Surgeons Pakistan
(2017) · Journal of College of Physicians And Surgeons Pakistan · doi:10.29271/jcpsp · Cited by 920
Research works printed but not linked: 8
- Brewer C., 2011, Vision and change in undergraduate biology education: A call to action
- Budd D. A., 2013, Journal of Geoscience Education, 61, 461
- Hayward C. N., 2018, Proceedings of the 21st annual conference on the research in undergraduate mathematics education, 727
- Kane T. J., 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
- Nadelson L. S., 2013, Learning Communities Journal, 5, 97
- Nolan B. A., 2002, Alzheimer’s Care Today, 3, 46
- Pilburn M., 2000, Reformed Teaching Observation Protocol (RTOP)
- Weisberg D., Sexton S., Mulhern J., Keeling D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness (2nd ed.). The New Teacher Project. https://tntp.org/assets/documents/TheWidgetEffect_2nd_ed.pdf
Cited by
Data Analysis of Educational Evaluation Using K-Means Clustering Method
Liu, R. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3762431 · Cited by 46
“…and value has become an important topic of common concern at home and abroad to tackle the dif-ficult problems in educational data processing [3]. introduction of information management technology to tackle data problems produced by the influx of students in the field of education has played a…”
BackgroundDenaro, K., Kranzfelder, P., Owens, M. T., Sato, B. K., Zuckerman, A. L., Hardesty, R. A., Signorini, A., Aebersold, A., Verma, M., & Lo, S. M. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00365-9 · Cited by 30
“However, several studies since then have shown that to characterize the teaching styles of individual instructors, it is necessary to observe them as many as 9-11 times because instructors display a lot of variability session-to-session in how they teach (Sbeglia et al. 2021, Weston et al. 2021).”
BackgroundHaag, K., Pickett, S. B., Trujillo, G., & Andrews, T. C. (2022) · CBE—Life Sciences Education · doi:10.1187/cbe.22-08-0169 · Cited by 34
“, 2018), but two different research groups have since demonstrated that four observations are often inadequate to reliably document instructional change across time (Sbeglia et al., 2021; Weston et al., 2021).”
BackgroundWhite, M., Göllner, R., & Kleickmann, T. (2025) · School Effectiveness and School Improvement · doi:10.1080/09243453.2025.2482583 · Cited by 14
“…would contribute to weak and inconsistent relationships: The reliability of measures is often lower than desired (e.g., Gaertner & Brunner, 2018; Weston et al., 2021); rater and respondent errors introduce construct-irre-levant variation in scores (e.g., Schweig & Martínez, 2021; White &…”
Weston, T. J., Laursen, S. L., & Hayward, C. N. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00413-y · Cited by 12
“The assumptions for the PND measure are that (1) enough classes are observed for each teacher to create a reliable measure and proportion (Brennan, 2001; Weston et al., 2021), and (2) profiles derived from LPA reflect both traditional and reform teaching practices.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Weston, T. J., Hayward, C. N., & Laursen, S. L.
- Affiliations
- University of Colorado Boulder (US)
- Corresponding author
- Weston, Timothy J.
- Venue
- American Journal of Evaluation
- Publisher
- SAGE Publications
- Published
- 2021-07-20
- Volume
- 42
- Issue
- 3
- Pages
- 377-398
- ISSN
- 1098-2140 (ISSN-L)
- 1557-0878
- OpenAlex
- W3186113926
- Semantic Scholar
- e941de70feee75af1a3355e34456241ca1afc971
- Cited by
- 20
- References
- 63
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 1 · 2023: 2 · 2022: 6 · 2021: 2
- FWCI
- 4.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Evaluation (publishedVersion)
- Funders
- National Science Foundation (1245436)
- Topics
- Social Sciences › Education › Evaluation of Teaching Practices
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Research and Methods
- Keywords
- classroom observation, generalizability theory, reliability statistics, teaching practice, sample size, inter-rater reliability, educational assessment, research on teaching, observational data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching. https://app.cochat.ai/papers/10.1177/1098214020931941
- The paper
- Weston, T. J., Hayward, C. N., & Laursen, S. L. (2021). When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching. American Journal of Evaluation. https://doi.org/10.1177/1098214020931941
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.