When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
The DOI matches its Crossref record; 60 of the 63 references it cites were found in a registry or an index.
Abstract
Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1979) · Medical Entomology and Zoology · Cited by 11,252
Active learning increases student performance in science, engineering, and mathematics
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1319030111 · Cited by 9,787 · cited 1×
“Recent reports (Brewer & Smith, 2011; Freeman et al., 2014) present multiple studies supporting the efficacy of reform-based teaching methods and advocate their use in undergraduate STEM classes.”
ResultSAGE Handbook of Mixed Methods in Social & Behavioral Research
Tashakkori, A., & Teddlie, C. (2010) · doi:10.4135/9781506335193 · Cited by 6,215
Hake, R. R. (1998) · American Journal of Physics · doi:10.1119/1.18809 · Cited by 6,072 · cited 2×
“Most of these methods emphasize less lecture and more student participation using broadly defined approaches such as active learning (Chi & Wiley, 2014), inquiry-based learning (Laursen et al., 2014), active-inquiry teaching (Hake, 1998), and interactive engagement (Turpen & Finkelstein, 2009).”
MethodThe Journal of College Science Teaching
(1971) · Journal of Chemical Education · doi:10.1021/ed048p284 · Cited by 3,332
The ICAP framework: Linking cognitive engagement to active learning outcomes
Chi, M. T. H., & Wylie, R. (2014) · Educational Psychologist · doi:10.1080/00461520.2014.965823 · Cited by 2,911
Where's the evidence that active learning works?
Michael, J. (2006) · Advances in Physiology Education · doi:10.1152/advan.00053.2006 · Cited by 1,648
Pianta, R. C., & Hamre, B. K. (2009) · Educational Researcher · doi:10.3102/0013189x09332374 · Cited by 1,331 · cited 1×
“However, the products of structured teacher observations are decidedly quantitative (Pianta & Hamre, 2009).”
BackgroundGuidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audige, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M., & Streiner, D. L. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.01.016 · Cited by 997
Journal of the College of Physicians and Surgeons Pakistan
(2017) · Journal of College of Physicians And Surgeons Pakistan · doi:10.29271/jcpsp · Cited by 920
Anatomy of STEM teaching in North American universities
Stains, M., Harshman, J., Barker, M. K., Chasteen, S. V., Cole, R., DeChenne-Peters, S. E., Eagan, M. K., Esson, J. M., Knight, J. K., Laski, F. A., et al. (2018) · Science · doi:10.1126/science.aap8892 · Cited by 772
Smith, M. K., Jones, F. H. M., Gilbert, S. L., & Wieman, C. E. (2013) · CBE—Life Sciences Education · doi:10.1187/cbe.13-08-0154 · Cited by 571
Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002) · School Science and Mathematics · doi:10.1111/j.1949-8594.2002.tb17883.x · Cited by 564 · cited 4×
“Instrument developers publish the reliability of their measures to provide information about interrater agreement (Hora et al., 2013; Sawada et al., 2002).”
BackgroundMethodInfluentialCompetency-based medical education in postgraduate medical education
Iobst, W. F., Sherbino, J., Cate, O. T., Richardson, D. L., Dath, D., Swing, S. R., Harris, P., Mungroo, R., Holmboe, E. S., Frank, J. R., et al. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.500709 · Cited by 478
Blanchard, M. R., Southerland, S. A., Osborne, J. W., Sampson, V. D., Annetta, L. A., & Granger, E. M. (2010) · Science Education · doi:10.1002/sce.20390 · Cited by 424
What We Say Is Not What We Do: Effective Evaluation of Faculty Professional Development Programs
Ebert-May, D., Derting, T. L., Hodder, J., Momsen, J. L., Long, T. M., & Jardeleza, S. E. (2011) · BioScience · doi:10.1525/bio.2011.61.7.9 · Cited by 420 · cited 5×
“Ebert-May et al. (2011) compared college teachers’ self-report of their teaching methods to their self-reported activities from surveys using two observations per semester, as did Lewin et al. (2016) in a similar study.”
BackgroundMethodResultInfluentialGardner, F. (2000) · Clinical Child and Family Psychology Review · doi:10.1023/a:1009503409699 · Cited by 402
Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012) · Phi Delta Kappan · doi:10.1177/003172171209300603 · Cited by 402 · cited 3×
“In K–12 education, direct observation is used in teacher evaluation as principals and peer teachers visit classrooms to assess teachers and college faculty (Darling-Hammond et al., 2012).”
BackgroundInfluentialHill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012) · Educational Researcher · doi:10.3102/0013189x12437203 · Cited by 331
An Introduction to Classroom Observation
Wragg, T. (2002) · Routledge eBooks · doi:10.4324/9780203018958 · Cited by 311
4 Reliability Coefficients and Generalizability Theory
Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006) · Handbook of Statistics · doi:10.1016/s0169-7161(06)26004-8 · Cited by 271
Getting Teacher Evaluation Right: What Really Matters for Effectiveness and Improvement
Darling‐Hammond, L. (2013) · Medical Entomology and Zoology · Cited by 261
Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2011) · CBE—Life Sciences Education · doi:10.1187/cbe.10-08-0101 · Cited by 258
Laursen, S. L., Hassi, M.-L., Kogan, M., & Weston, T. J. (2014) · Journal for Research in Mathematics Education · doi:10.5951/jresematheduc.45.4.0406 · Cited by 205
Self-citationDevelopment and Evaluation of a Genetics Literacy Assessment Instrument for Undergraduates
Bowling, B. V., Acra, E. E., Wang, L., Myers, M. F., Dean, G. E., Markle, G. C., Moskalik, C. L., & Huether, C. A. (2008) · Genetics · doi:10.1534/genetics.107.079533 · Cited by 204
Building a More Complete Understanding of Teacher Evaluation Using Classroom Observations
Cohen, J., & Goldhaber, D. (2016) · Educational Researcher · doi:10.3102/0013189x16659442 · Cited by 201
Applying Generalizability Theory using EduG
Cardinet, J., Johnson, S., & Pini, G. (2011) · Routledge eBooks · doi:10.4324/9780203866948 · Cited by 180
Turpen, C., & Finkelstein, N. D. (2009) · Physical Review Special Topics - Physics Education Research · doi:10.1103/physrevstper.5.020101 · Cited by 166
A Campus-Wide Study of STEM Courses: New Perspectives on Teaching Practices and Perceptions
Smith, M. K., Vinson, E. L., Smith, J. A., Lewin, J. D., & Stetzer, M. R. (2014) · CBE—Life Sciences Education · doi:10.1187/cbe.14-06-0108 · Cited by 139
Lund, T. J., Pilarz, M., Velasco, J. B., Chakraverty, D., Rosploch, K., Undersander, M., & Stains, M. (2015) · CBE—Life Sciences Education · doi:10.1187/cbe.14-10-0168 · Cited by 132
A New Tool for Measuring Student Behavioral Engagement in Large University Classes
Lane, E. S., & Harris, S. E. (2015) · Journal of College Science Teaching · doi:10.2505/4/jcst15_044_06_83 · Cited by 130
Gitomer, D., Bell, C., Qi, Y., Mccaffrey, D., Hamre, B. K., & Pianta, R. C. (2014) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146811411600607 · Cited by 114
Adamson, S. L., Banks, D. L., Burtch, M., Cox, F., Judson, E., Turley, J., Benford, R., & Lawson, A. E. (2003) · Journal of Research in Science Teaching · doi:10.1002/tea.10117 · Cited by 96
Hintze, J. M., Owen, S. V., Shapiro, E. S., & Daly, E. J. (2000) · School Psychology Quarterly · doi:10.1037/h0088778 · Cited by 88 · cited 1×
“Hintze et al. (2000) used ten observations of reading fluency and saw 8% of total variability due to observations.”
BackgroundHalpin, P. F., & Kieffer, M. J. (2015) · Educational Researcher · doi:10.3102/0013189x15590804 · Cited by 79
Short and Long-Term Impacts of the Cottrell Scholars Collaborative New Faculty Workshop
Stains, M., Pilarz, M., & Chakraverty, D. (2015) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5b00324 · Cited by 75
Naeem, N. (2013) · PubMed · Cited by 73
Cash, A. H., Hamre, B. K., Pianta, R. C., & Myers, S. S. (2012) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2011.12.006 · Cited by 72 · cited 5×
“While the confident use of a protocol does depend upon the characteristics of that instrument, the actual interrater reliability of raters in practice depends upon the amount and type of training given to raters, and on ongoing calibration of raters over time (Cash et al., 2012).”
BackgroundInfluentialVariation of instructor-student interactions in an introductory interactive physics course
West, E., Paul, C., Webb, D. J., & Potter, W. H. (2013) · Physical Review Special Topics - Physics Education Research · doi:10.1103/physrevstper.9.010109 · Cited by 63
Mashburn, A. J., Downer, J. T., Rivers, S. E., Brackett, M. A., & Martínez, A. (2013) · Prevention Science · doi:10.1007/s11121-012-0357-3 · Cited by 61
Hora, M. T., & Ferrare, J. J. (2014) · The journal of college science teaching · doi:10.2505/4/jcst14_043_03_36 · Cited by 60
Characterizing Teaching in Introductory Geology Courses: Measuring Classroom Practices.
Budd, D. A., Kraft, K. J. V. D. H., McCONNELL, D. A., & Vislova, T. (2013) · Journal of Geoscience Education · doi:10.5408/12-381.1 · Cited by 58
van der Lans, R. M., van de Grift, W. J. C. M., van Veen, K., & Fokkens-Bruinsma, M. (2016) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2016.08.001 · Cited by 55
A Campus-Wide Investigation of Clicker Implementation: The Status of Peer Discussion in STEM Classes
Lewin, J. D., Vinson, E. L., Stetzer, M. R., & Smith, M. K. (2016) · CBE—Life Sciences Education · doi:10.1187/cbe.15-10-0224 · Cited by 54
Murray, H. G. (2007) · Springer eBooks · doi:10.1007/1-4020-5742-3_6 · Cited by 53
Amrein‐Beardsley, A., & Popp, S. E. O. (2011) · Educational Assessment Evaluation and Accountability · doi:10.1007/s11092-011-9135-1 · Cited by 50
Teaching historical contextualization: the construction of a reliable observation instrument
Huijgen, T., van de Grift, W. J. C. M., van Boxtel, C., & Holthuis, P. (2016) · European Journal of Psychology of Education · doi:10.1007/s10212-016-0295-8 · Cited by 40
The application of generalizability analysis to observational studies
Marcoulides, G. A. (1989) · Quality & Quantity · doi:10.1007/bf00151898 · Cited by 29
The Uses of the Classroom Observation Schedule to Improve Classroom Instruction
Waxman, H. C., & Padrón, Y. N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511616419.004 · Cited by 25
Newton, X. A. (2010) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2010.10.002 · Cited by 22
Lei, P., Smith, M., & Suen, H. K. (2007) · Psychology in the Schools · doi:10.1002/pits.20235 · Cited by 21
Campbell, T., Der, J. P., Wolf, P. G., et al. (2012) · The journal of college science teaching · Cited by 18
Using video surveillance footage to support validity of self-reported classroom data
Lee, D., Arthur, I. T., & Morrone, A. S. (2015) · International Journal of Research & Method in Education · doi:10.1080/1743727x.2015.1075496 · Cited by 18
Understanding Instructional Quality in English Language Arts
Grossman, P., Cohen, J., & Brown, L. (2015) · doi:10.1002/9781119210856.ch10 · Cited by 17
Making Inferences About Teacher Observation Scores Over Time
Briggs, D. C., & Alzen, J. L. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164419826237 · Cited by 14 · cited 1×
“As noted, this is a common practice in other Gstudies that examine the number of occasions classes have been observed (Briggs & Alzen, 2019; Hill et al., 2012).”
ResultInstructor Use of Group Active Learning in an Introductory Biology Sequence
Auerbach, A. J., & Schussler, E. E. (2016) · The journal of college science teaching · doi:10.2505/4/jcst16_045_05_67 · Cited by 13
Mertler, C. A. (2016) · Routledge eBooks · doi:10.4324/9781315266756 · Cited by 11
13: Co-Teaching as a Faculty Development Model
Beach, A. L., Henderson, C., & Famiano, M. (2008) · To improve the academy · doi:10.1002/j.2334-4822.2008.tb00509.x · Cited by 10
Carins, J. (2016) · Springer eBooks · doi:10.1007/978-981-10-1829-9_7 · Cited by 2
Boodoo, G. M. (2014) · Encyclopedia of Special Education · doi:10.1002/9781118660584.ese0989 · Cited by 1
Research works printed but not linked: 8
- Brewer C., 2011, Vision and change in undergraduate biology education: A call to action
- Budd D. A., 2013, Journal of Geoscience Education, 61, 461
- Hayward C. N., 2018, Proceedings of the 21st annual conference on the research in undergraduate mathematics education, 727
- Kane T. J., 2012, Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains
- Nadelson L. S., 2013, Learning Communities Journal, 5, 97
- Nolan B. A., 2002, Alzheimer’s Care Today, 3, 46
- Pilburn M., 2000, Reformed Teaching Observation Protocol (RTOP)
- Weisberg D., Sexton S., Mulhern J., Keeling D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness (2nd ed.). The New Teacher Project. https://tntp.org/assets/documents/TheWidgetEffect_2nd_ed.pdf
Cited by
Data Analysis of Educational Evaluation Using K-Means Clustering Method
Liu, R. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3762431 · Cited by 46
“…and value has become an important topic of common concern at home and abroad to tackle the dif-ficult problems in educational data processing [3]. introduction of information management technology to tackle data problems produced by the influx of students in the field of education has played a…”
BackgroundDenaro, K., Kranzfelder, P., Owens, M. T., Sato, B. K., Zuckerman, A. L., Hardesty, R. A., Signorini, A., Aebersold, A., Verma, M., & Lo, S. M. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00365-9 · Cited by 30
“However, several studies since then have shown that to characterize the teaching styles of individual instructors, it is necessary to observe them as many as 9-11 times because instructors display a lot of variability session-to-session in how they teach (Sbeglia et al. 2021, Weston et al. 2021).”
BackgroundHaag, K., Pickett, S. B., Trujillo, G., & Andrews, T. C. (2022) · CBE—Life Sciences Education · doi:10.1187/cbe.22-08-0169 · Cited by 40
“, 2018), but two different research groups have since demonstrated that four observations are often inadequate to reliably document instructional change across time (Sbeglia et al., 2021; Weston et al., 2021).”
BackgroundWhite, M., Göllner, R., & Kleickmann, T. (2025) · School Effectiveness and School Improvement · doi:10.1080/09243453.2025.2482583 · Cited by 14
“…would contribute to weak and inconsistent relationships: The reliability of measures is often lower than desired (e.g., Gaertner & Brunner, 2018; Weston et al., 2021); rater and respondent errors introduce construct-irre-levant variation in scores (e.g., Schweig & Martínez, 2021; White &…”
Weston, T. J., Laursen, S. L., & Hayward, C. N. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00413-y · Cited by 12
“The assumptions for the PND measure are that (1) enough classes are observed for each teacher to create a reliable measure and proportion (Brennan, 2001; Weston et al., 2021), and (2) profiles derived from LPA reflect both traditional and reform teaching practices.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Weston, T. J., Hayward, C. N., & Laursen, S. L.
- Affiliations
- University of Colorado Boulder (US)
- Corresponding author
- Weston, Timothy J.
- Venue
- American Journal of Evaluation
- Publisher
- SAGE Publications
- Published
- 2021-07-20
- Volume
- 42
- Issue
- 3
- Pages
- 377-398
- ISSN
- 1098-2140 (ISSN-L)
- 1557-0878
- OpenAlex
- W3186113926
- Semantic Scholar
- e941de70feee75af1a3355e34456241ca1afc971
- Cited by
- 20
- References
- 63
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 1 · 2023: 2 · 2022: 6 · 2021: 2
- FWCI
- 4.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Evaluation (publishedVersion)
- Funders
- National Science Foundation (1245436)
- Topics
- Social Sciences › Education › Evaluation of Teaching Practices
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Research and Methods
- Keywords
- classroom observation, generalizability theory, reliability statistics, teaching practice, sample size, inter-rater reliability, educational assessment, research on teaching, observational data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.