Measuring Reform Practices in Science and Mathematics Classrooms: The Reformed Teaching Observation Protocol
Abstract
The National Science Foundation has funded 22 Collaboratives for Excellence in Teacher Preparation. Despite the remarkable allocation of resources to this effort, it has proven exceptionally difficult to demonstrate the effectiveness of collaborative reform. In large part, this has resulted because of the difficulty of defining and measuring reform. The Reformed Teaching Observation Protocol (RTOP) was designed by the Evaluation Facilitation Group of the Arizona Collaborative for Excellence in the Preparation of Teachers (ACEPT). It is a 25‐item classroom observation protocol that is (a) standards based, (b) inquiry oriented, and (c) student centered. This instrument has provided the definition for reform and the basis for evaluation of the ACEPT collaborative. The data upon which this report is based were collected over a period of more than 2 years from 153 public school, college, and university mathematics and science classrooms. A trained team of observers consisting of two faculty members and seven graduate students was able to achieve exceptionally high levels of interrater reliability. Internal consistency, as estimated by Cronbach's alpha, was also remarkably high. Correlation coefficients ranging from 0.88 to 0.97 between RTOP scores for classrooms, and mean normalized gain scores for students in those classrooms on achievement measures demonstrate that reform, as defined by ACEPT and measured by the RTOP, has been effective.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evidence shows that FLCs increase faculty interest in teaching and learning and provide support to change longstanding instructional practice.
“Evidence shows that FLCs increase faculty interest in teaching and learning and provide support to change longstanding instructional practice [29,30].”
From Faculty online learning communities: A model for sustained teaching transformation (2019) · cites this paper as [30]
The Reformed Teaching Observation Protocol (RTOP) instrument was used during classroom observations after the intervention to determine whether the transfer of new knowledge and skills took place.
“The Reformed Teaching Observation Protocol (RTOP) instrument (Sawada, Piburn, and Judson 2002) was used during classroom observations after the intervention, to determine whether the transfer of new knowledge and skills took place.”
From Insights provided into the decolonisation of the science curriculum, and teaching and learning of indigenous knowledge, using Cultural-Historical Activity Theory (2021) · cites this paper as Sawada, Piburn, and Judson 2002
Cites
Those Who Understand: Knowledge Growth in Teaching
SHULMAN, L. S. (1986) · Educational Researcher · doi:10.3102/0013189x015002004 · Cited by 18,546 · see citing works
Benchmarks for Science Literacy
McComas, W. F. (2013) · SensePublishers eBooks · doi:10.1007/978-94-6209-497-0_11 · Cited by 1,718 · see citing works
National Science Education Standards
(1996) · PsycEXTRA Dataset · doi:10.1037/e302882005-001 · Cited by 1,575 · see citing works
Inquiry and the National Science Education Standards
Board on Science Education, & Education, D. O. B. A. S. S. A. (2000) · National Academies Press eBooks · doi:10.17226/9596 · Cited by 980 · see citing works
Assessments and Accountability
Linn, R. L. (2000) · Educational Researcher · doi:10.2307/1177052 · Cited by 678 · see citing works
Assessment Standards for School Mathematics
Elliott, P. C. (1995) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 674 · see citing works
Project 2061: Science for All Americans
Primack, J. R. (1990) · Academe · doi:10.2307/40249601 · Cited by 547 · see citing works
Science Teaching and Development Of Thinking
Lawson, A. E. (1994) · Cited by 506 · see citing works
Measuring Instructional Practice: Can Policymakers Trust Survey Data?
Mayer, D. P. (1999) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737021001029 · Cited by 367 · see citing works
Relationships between Research and the NCTM Standards
Hiebert, J. (1999) · Journal for Research in Mathematics Education · doi:10.2307/749627 · Cited by 315 · see citing works
Beyond Fourth-Grade Science: Why Do U.S. and Japanese Students Diverge?
Linn, M. C., Lewis, C., Tsuchida, I., & Songer, N. B. (2000) · Educational Researcher · doi:10.3102/0013189x029003004 · Cited by 292 · see citing works
Establishing a research agenda: critical issues of science curriculum reform
Shymansky, J. A., & Kyle, W. C. (1992) · Journal of Research in Science Teaching · doi:10.1002/tea.3660290803 · Cited by 125 · see citing works
Science assessment in the service of reform
(1992) · Choice Reviews Online · doi:10.5860/choice.29-5812 · Cited by 88 · see citing works
Reformed Teaching Observation Protocol (RTOP) TRAINING GUIDE
Sawada, D. (2000) · Cited by 40 · see citing works
Self-citationSCIENCE AND SCHOOLS: Mixed Grades for NSF's Bold Reform of Statewide Education
Mervis, J. (1998) · Science · doi:10.1126/science.282.5395.1800 · Cited by 4 · see citing works
Tittle, C. K., & Pape, S. J. (1996) · Cited by 3 · see citing works
The Mathematics Reform Movement: Assessing the Degree of Reform in Secondary Mathematics Classrooms.
Calhoun, D. O., Bohlin, C. F., Bohlin, R. M., et al. (1997) · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
What We Say Is Not What We Do: Effective Evaluation of Faculty Professional Development Programs
Ebert-May, D., Derting, T. L., Hodder, J., Momsen, J. L., Long, T. M., & Jardeleza, S. E. (2011) · BioScience · doi:10.1525/bio.2011.61.7.9 · Cited by 420 · see citing works
Faculty online learning communities: A model for sustained teaching transformation
Dancy, M., Lau, A. C., Rundquist, A., & Henderson, C. (2019) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.15.020147 · Cited by 53 · see citing works
Boom-Cárcamo, E., Buelvas-Gutiérrez, L., Acosta-Oñate, L., & Boom-Cárcamo, D. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101614 · Cited by 29 · see citing works
Teaching to support students with diverse academic needs
Blazar, D., & Archer, C. (2020) · Educational Researcher · doi:10.3102/0013189x20931226 · Cited by 23 · see citing works
Weston, T. J., Hayward, C. N., & Laursen, S. L. (2021) · American Journal of Evaluation · doi:10.1177/1098214020931941 · Cited by 20 · see citing works
de Beer, J. J., & Kriek, J. (2021) · South African Journal of Higher Education · doi:10.20853/35-6-4217 · Cited by 19 · see citing works
- Authors
- Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I.
- Affiliations
- University of Alberta (CA)
- Arizona State University (US)
- Corresponding author
- Piburn, Michael D.
- ORCID
- Judson, Eugene
- Venue
- School Science and Mathematics
- Publisher
- Wiley
- Published
- 2002-10
- Volume
- 102
- Issue
- 6
- Pages
- 245-253
- ISSN
- 0036-6803 (ISSN-L)
- 1949-8594
- OpenAlex
- W1990731512
- Cited by
- 564
- References
- 18
- Citations by year
- 2026: 18 · 2025: 19 · 2024: 25 · 2023: 22 · 2022: 36 · 2021: 40 · 2020: 53 · 2019: 47 · 2018: 34 · 2017: 39 · 2016: 51 · 2015: 24 · 2014: 29 · 2013: 31 · 2012: 22
- FWCI
- 14.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- School Science and Mathematics (publishedVersion)
- Topics
- Social Sciences › Education › Science Education and Pedagogy
- Social Sciences › Education › Mathematics Education and Teaching Techniques
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- Reformed Teaching Observation Protocol, classroom observation instrument, inquiry-based learning, student-centered instruction, science education reform, mathematics education reform, inter-rater reliability, academic performance, teacher preparation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
