Measuring Instructional Practice: Can Policymakers Trust Survey Data?

Abstract

Policymakers' new interest in reforming teaching has created a demand for accurate data on instructional practice. Most such data come from teacher surveys, although the accuracy of these surveys in assessing practice is virtually unknown. This exploratory study examines the reliability and validity of using a survey to gauge the percentage of time that algebra teachers use practices that are consistent with professional standards for teaching mathematics. A 13-item composite measuring the frequency with which teachers use these practices had a test-retest reliability of .69 based on two waves of survey data collected four months apart, In addition, there was a .85 correlation between a composite based on survey data and a parallel composite based on classroom observations. Both sets of results suggest that the composite is quite reliable, and the second set of results suggests that it has some validity: This composite fails, however, to capture the quality with which teachers engage in reform practices. In addition, the results indicate that teachers do not reliably report how much time they spend using one practice or another. This suggests that the trend of presenting individual indicators, rather than composite indicators, in state and national reports may be misguided. More research should be done to improve the reliability and validity of both.