Curriculum-Based Measurement of Oral Reading: Standard Errors Associated With Progress Monitoring Outcomes From DIBELS, AIMSweb, and an Experimental Passage Set

“The results of other research generally converge with that finding (Ardoin & Christ, 2009; Christ, 2006; Hintze & Christ, 2004; Hintze, Daly, & Shapiro, 1998; Hintze & Shapiro, 1997; Hintze et al., 2000; Jenkins et al., 2005; Jenkins et al., 2009).”

“…result, the vast majority of all ORF growth studies report constant rates of change over time, estimated most often in weekly expected gains (e.g., Ardoin & Christ, 2008, 2009; Christ, 2006; Christ, Zopluoglu, Long, & Monaghen, 2012; Deno et al., 2001; Fuchs et al., 1993; Graney et al., 2009;…”

“As measurement error increases, the accuracy of our decisions regarding students’ RtI decreases (Ardoin and Christ 2009).”

“For instance, one problem the field is currently facing is the development of passages that are equivalent in level of difficulty for use in CBM oral reading (CBM-R) progress monitoring (Ardoin & Christ, 2009; Christ & Ardoin, 2009).”

“…test also exhibits form/passage effects when used with middle-grade students is consistent with prior work on ORF passages in the elementary grades (Ardoin & Christ, 2009; Christ & Ardoin, 2009; Dunn & Eckert, 2002; Francis et al., 2008; Hintze, Shapiro, & Lutz, 1994; Poncy, Skinner, & Axtell,…”

“These form effects (Ardoin & Christ, 2009; Christ & Silberglitt, 2007; Francis et al., 2008) refer to passage-level variability that persists in spite of the application of a priori readability formulae.”

“Previous research established that a dataset with an SEE of approximately 5 WRCM is very good, 10 is good, 15 is poor, and 20 is very poor (Ardoin & Christ, 2009; Christ, 2006).”

“Growth trends estimated from GOM/CBM tend to be quite inconsistent and unreliable, with large confidence intervals that make data interpretation difficult (Ardoin and Christ 2009; Christ and Coolong-Chaffin 2007).”

“…Foster et al., 2013; Vorstius et al., 2013), including some challenges and limits to such an approach, e.g., the need for standardized and well-controlled reading passages (Ardoin & Christ, 2009) and the prohibitive cost and issues of durability of the equipment (see also Joseph et al., 2013).”

“Although some studies have assumed linear growth with enough occasions to model nonlinearity (e.g., Ardoin & Christ, 2009; Christ, 2006), most research in this area has explored and used nonlinear models.”

“This is problematic as currently there are no well validated measures for monitoring progress across short periods of time (e.g., 0–15 weeks) (Ardoin & Christ, 2009).”

“Using the likely range of SEEs, as reported by Christ and Silberglitt (2007) 1 and Ardoin and Christ (2009), 20,000 time-series data sets were simulated per condition to model student reading growth of either two wcpm/wk or one wcpm/wk across 15 consecutive weeks of intervention.”

“However, as cautioned by Ardoin and Christ (2009), it is essential for researchers to be mindful that the results obtained using progress monitoring tools, such as CBM-R and CBM Maze, serve as an approximation of student reading levels and growth rates.”

“…continue, modify, or discontinue intervention based on CBM-R progress monitoring data has so much error embedded in the decision that decisions are likely to be incorrect (Ardoin & Christ, 2009; Francis et al., 2008; Good & Shinn, 1990; Poncy, Skinner, & Axtell, 2005; Shinn, Good, & Stein, 1989).”

“…average range of SEE values, a proxy for data set variability or measurement error, was between 7 and 10 WRCM, values considered typical in applied practice when using commercial instruments and collecting data in standardized conditions (Ardoin & Christ, 2009; Christ, 2006; Christ et al., 2013).”

“First, differences in progress monitoring outcomes as a function of passage type have been addressed elsewhere (see Ardoin & Christ, 2009).”

“…evidence examining the equivalence of CBM-R probes within probe sets, evidence of the reliability and validity of probes is not sufficient to establish them as appropriate for progress monitoring (Ardoin & Christ, 2008; Ardoin & Christ, 2009; Ardoin, Christ, Morena, Cormier, & Klingbeil, 2013).”

“Data used for mediumto high-stakes decisions should include measurement error, so the likelihood of a wrong or “regrettable decision” will be known (Ardoin & Christ, 2009; Goodman, 1999; Hintze & Christ, 2004).”

“Previous studies examining the influence of passage variability have also demonstrated the influence of error related to single CBM points on the value of the slopes generated by the data (Ardoin & Christ, 2009; Christ, 2006; Francis et al., 2008).”

“…as a viable solution to avoid these measurement biases and to guarantee the scores’ equivalence (Albano & Rodriguez, 2012; Ardoin et al., 2010; Ardoin & Christ, 2009; Betts et al., 2009; Cummings et al., 2013; Denton et al., 2011; Francis et al., 2008; Petscher & Kim, 2011; Stoolmiller…”

“…systems has surpassed mere progress monitoring to include additional uses such as screening for academic difficulties, setting individualized goals, benchmarking, and informing eligibility decisions (Ardoin & Christ, 2009; Good, Simmons & Kame’enui, 2001; Reschly et al., 2009; Wiley & Deno, 2005).”

“Furthermore, a line of research by Christ and colleagues has demonstrated a profound impact of SEM on estimates of slope and, ultimately, decisions about adequate student progress (Ardoin & Christ, 2009; Christ, 2006; Christ, Zopluoglu, Long, & Monaghen, 2012).”

“Researchers observed that the variation of CBM-R scores around the OLSR line was 11, 12, and 15 WRCM (Dynamic Indicators of Basic Early Literacy Skills (DIBELS); University of Oregon Center on Teaching and Learning, 2017) when they used each of three different probe sets (Ardoin & Christ, 2009) .”

“…administrations (Hintze & Shapiro, 1997) and over 12 weeks (with approximately 2 administrations per week) to determine standard errors of measurement (Ardoin & Christ, 2009); likewise, passage difficulty level on oral reading progress monitoring has been documented (Hintze, Daly, & Shapiro, 1998).”

“While these studies examined differences in student performance within a developed set of passages, additional research (i.e., Ardoin & Christ, 2009) has found differences when comparing students’ performance across CBM tools as well.”

“However, recent measurement issues have been identified pertaining to curriculum-based measures of reading (CBM-R), such as form effects (Ardoin & Christ, 2009; Christ & Ardoin, 2009; Francis et al., 2008; Petscher & Kim, 2011) and error of measurement (Christ & Silberglitt, 2007).”

“Ardoin and Christ (2009) reported a mean standard error of the estimate (SEE) for third-grade Table 3 Mean WCPM on word lists and passages across conditions Name Vowel team/ passage Base 1 WLT Base/Maint 2 WLT Base/Maint 3 WLT Maint 4 Dante AW T 45.5 56.5 70.0 76.5 G 45.0 16.0 51.5 63.5 68.5 OI T…”

“No reliability or validity analyses have been completed on the use of AIMSweb for comparison of individual student performance over time as was employed in this study (Ardoin & Christ, 2009).”

“Of note, the aimsweb passages resulted in significantly less measurement error when used to measure word reading as compared to the DIBELS passages—another progress monitoring tool that is commonly used in schools (Ardoin & Christ, 2009).”

“…administration (Christ, White, Ardoin, & Eckert, 2013; Cummings, Biancarosa, Schaper, & Reed, 2014; Derr & Shapiro, 1989), qualities of instrumentation (Ardoin & Christ, 2009; Francis et al., 2008), and the schedule and duration of data collection (Christ, Zopluoglu, Monaghen, & Van Norman, 2013).”

“Perhaps the most important consideration is obtaining a generated output with an accurate reading level (Ardoin & Christ, 2009; Shapiro, 2011).”

“We elected to readminister passages, rather than supplement with another vendor’s passages, because passages assigned the same grade-level difficulty across CBM vendors are not necessarily functionally equivalent (Ardoin & Christ, 2009; Ford et al., 2017).”

“Weekly administrations of RCBM are also commonly used for monitoring student progress and responsiveness to intervention (Ardoin & Christ, 2009; Fuchs, 2004; Silberglitt & Hintze, 2007).”

“Results revealed that the generalizability reliability for relative decisions ranged from r = .90 to .99 with the SEM values ranging from 4 to 12 CWPM, and the dependability reliability for absolute decisions ranged from r = .81 to .97 with a range of SEM values from 6 to 18 CWPM. Ardoin and Christ (2009) suggested th…”

“Writing a set of meaningful passages that conforms to a certain level of difficulty is challenging, and problems with passage nonequivalence have been demonstrated with CBM-R measures (Ardoin & Christ, 2009; Francis et al., 2008).”

“But CBM-R has several practical and technical inadequacies, including large measurement error, or standard error of measurement (SEM; Ardoin & Christ, 2009; Christ et al., 2012; Christ & Silberglitt, 2007; Poncy et al., 2005).”

“Each student dyad also received the following materials to conduct their peer-mediated RR sessions: (1) an 8” x 11” sheet of laminated paper with RR intervention (coaching) steps listed to reference and to mark each step when it was completed; (2) FAST (Ardoin & Christ, 2009) ORF progress monitoring passages at level…”

“Researchers and test publishers further refined their passage development to include field-testing and additional statistical equating to determine the passages with the least variability in practice within each grade level (e.g., Ardoin & Christ, 2009; Poncy et al., 2005).”

“The findings presented here also support the idea that test developers should attempt to discover outlier passages (Ardoin & Christ, 2009) through tests such as empirical Bayes mean estimates.”

“Many studies evaluating variability of CBM ORF have included participants who score at or above benchmark proficiency (Ardoin & Christ, 2009; Betts, Pickart, & Heistad, 2009; Briggs, 2011; Christ & Ardoin, 2009; Francis et al., 2008; Hintze & Christ, 2004; Poncy et al., 2005).”

“Even after rigorous field testing, differences in the complexity and variety of words in the passages result in probes that are systematically harder or easier than one another (Ardoin and Christ 2009).”

“The SE of the slope estimate quantifies the variability, or precision, of the slope estimate that has been often used in CBM-R research (e.g., Ardoin & Christ, 2009) to evaluate the accuracy of growth estimates.”

“Readability formulae imprecisely estimate the difficulty level of passages and are not adequate for equating passages (Ardoin & Christ, 2009; Francis, Santi, et al., 2008).”

“In a demonstration of the implications of inconsistent probe difficulty on progress monitoring outcomes, Ardoin and Christ (2009) administered aimsweb, DIBELS, and an experimental passage set twice weekly for 11 weeks to 68 students.”

“An additional limitation: ROIs derived from CBM-R are prone to error that may be impacted by the quality and schedule of progress monitoring data (e.g., Ardoin & Christ, 2009; Christ, 2006).”

“This sample size was informed by prior studies of progress monitoring (e.g., Ardoin & Christ, 2009; Thornblad & Christ, 2014).”

“According to the literature (e.g., Ardoin & Christ, 2009; Van Norman & Christ, 2016), SEE values usually vary between 5 and 20, where 5 means very low, and 20 very high variability.”

“SEE values often vary between 5 and 20, with 5 meaning a very low and 20 a very high variability e.g., [62,63].”

“Prior research indicates that the estimate of students’ ORF growth rate is highly dependent on the passage sets used (Ardoin & Christ, 2009) and the order in which the passages were administered (Francis et al., 2008).”

“This individual variation may be due to changes in difficulty of progress monitoring forms (e.g., Ardoin & Christ, 2009; Cummings, Park, & Bauer Schaper, 2013; Francis et al., 2008) that, despite strong relative correlations between forms, do not take into account absolute score differences across…”

“…measures for progress monitoring depressive symptoms are notably lacking (von der Embse, Scott, & Kilgus, 2015) and, without such measures, it will be impossible to engage in data-driven decision making within an MTSS framework (Ardoin & Christ, 2009; Hawken, Vincent, & Schumann, 2008).”

“First, our sample included students of varied initial performance, as did previous studies of R-CBM slopes (e.g., Ardoin & Christ, 2009; Thornblad & Christ, 2014).”

“In the area of reading, curriculumbased measures of reading (CBM-R) are among the most researched and most feasible options for this purpose.”

“In line with this, learning progress monitoring is predominantly thought of as CBM since emerging in the early 1970s (Deno, 1985; Klauer, 2014).”

“We purposefully developed our four student cases to reflect two levels of variability in the data based on prior research with CBM ORF passage sets (Ardoin & Christ, 2009; Christ et al., 2013; Christ & Silber-glitt, 2007; Cummings et al., 2013); a very low level of variability ( ± 6 points around…”

“The type of CBM used and system (e.g. DIBELS; University of Oregon, n.d.) can also affect how growth is modeled for students (Ardoin & Christ, 2009; Solomon et al., 2020).”