Data-informed curriculum reform: Which data, what purposes, and promoting and hindering factors

“Black and Wiliam (1998b) found, for example, that assessment data are usually not shared between teachers within schools, nor are they critically reviewed in relation to what they actually assess.”

“School staff can also abuse data or use data for educational triage practices (Gilborn & Youdell, 2000 cited in Booher-Jennings, 2005) or the focus on so-called bubble kids (Booher-Jennings, 2005).”

“Combining information from several sources of data, like standardized assessments and results from students' daily work is important in this respect (Schildkamp & Kuiper, 2010).”

“Yet, the process of interpreting data and implementing data-driven decisions requires considerable time, effort, and expertise (e.g., Schildkamp and Kuiper, 2010; Vanlommel et al., 2017), which may inevitably overburden teachers, given that they are primarily concerned with delivering instruction.”

“In addition, the Dutch schools also believe in the power of harvesting feedback data from learners to produce change (Visscher, 2008); while policy makers shall consider learner data as baseline to instructional decisions to increase learner performance (Schildkamp & Kuiper, 2010).”

“…success, as it plays a critical role in shaping the broader academic environment, benefiting not only students but also educators, policymakers, and society at large [4]; this analysis is crucial for informed decision-making at various levels of the educational sys-tem [5], [6], [7], [8], [9].”

“Principals’ comments are in accordance with research findings that there is simply not enough time for principals and teachers to sort through data collected by external agencies about their schools (Schildkamp and Kuiper, 2010; Shen and Cooley, 2008), or that they are unprepared for data analysis believing that it ad…”

“This form of data use has been promoted widely within the teaching profession over the past 20 years, often in the guise of “data-informed instruction”, “data-driven instruction”, “databased instruction”, “teaching analytics”, and “data-driven decision making” (e.g. Schildkamp & Kuiper, 2010).”

“For example, teachers would be more involved in the decision process of an individual student or group of students while principals would be more focused on the decision process at the school level (Schildkamp & Kuiper, 2010).”

“Teacher’s review/ feedback Student’s Reflections GenAI Feedback GenAI Rewrite Many studies advocate for approaches to formative assessment such as Data Based Decision Making (DBDM) and assessment for learning (A4L) (Klenowski, 2009; Schildkamp & Kuiper, 2010).”

“As a result, teachers became less motivated to find other causes for low achievement, and were less likely to interrogate their practice or change their instruction in response to low achievement (Bertrand & Marsh, 2015; Diamond, 2008; Schildkamp & Kuiper, 2010).”

“Learning Analytics and DBDM pursue the same goal: Both concepts aim to support teachers in making pedagogical decisions based on learning data and not only on experience and intuition, for example, to enable individualized learning processes (Schildkamp and Kuiper, 2010; Greller et al., 2014).”

“It is yet difficult to translate results from the studies and apply them to educational environments without a formalized identification of children with reading problems (Ingram et al., 2004; Schildkamp & Kuiper, 2010).”

“Educators take a more class- or student-level perspective (i.e., micro level; Schildkamp & Kuiper, 2010), which according to Staman et al. (2014) suggests that “the most important data and decisions within school systems relate directly to what happens and will happen in classrooms” (p. 79).”

“However, educators often do not use data (Schildkamp and Kuiper 2010), but instead make use of their intuition and limited observations (Ingram, Louis, and Schroeder 2004).”

“…to which the school environment is safe (e.g., for openly discussing an attempt to improve performance), improvement-oriented, cooperative, and supportive and facilitates improvement in terms of the resources required for working on improvement (e.g., time and money; Schildkamp & Kuiper, 2010).”

“However, teachers often adapt their teaching based on their intuition and experience (Schildkamp & Kuiper, 2010), considering what they notice in their classroom and limited observations (Ingram, et al., 2004) rather than purposely collected data.”

“It differs from one of the most well-known forms of formative assessment, namely data-based decision making (Schildkamp and Kuiper, 2010; Van der Kleij et al., 2015; Heitink et al., 2016).”

“This is helpful to support developing practitioners’ competence in the use of data (Schildkamp and Kuiper, 2010) in quality management for OBL (Ehlers, 2007).”

“While resistance to DDDM in practicing teachers has been identified in the literature (Brown et al., 2010; Dunn et al., 2013c; Remesal, 2011; Schildkamp & Kuiper, 2010), the results of this study suggest that resistance begins before teachers fully enter the profession.”

“…systems to time to coaches—may vary across contexts (e.g., Anderson, Leithwood, & Strauss, 2010; Datnow & Park, 2014; Hamilton et al., 2009; Huguet, Marsh, & Farrell, 2014; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Mandinach & Jackson, 2012; Schildkamp & Kuiper, 2010; Wayman & Jimerson, 2014).”

“A supporting school leader appears to be an important factor in enabling teachers to effectively use data (Schildkamp and Kuiper, 2010).”

“Many commentators suggest that an “ideal sequence” of teachers’ and schools’ engagement with research, evidence or data is that described in conceptual frameworks, which model the process of evidence and data use as a complex, cognitive, knowledge-based problem-solving or inquiry cycle with consecutive phases, that ar…”

“…use interventions, do not think that evidence-based reflection on their own work can improve their practice, and instead prefer to rely on their “years of experience” and are likely to attribute class performance to the students’ inherent ability (Schildkamp & Kuiper, 2010; Wayman et al., 2012 ).”

“…regard to decisionmaking (e.g., Messelt, 2004), performance feedback (e.g., Schildkamp & Teddlie, 2008), self-evaluation (e.g., Schildkamp, 2007; Schildkamp, Visscher, & Luyten, 2009), and curriculum reform (Schildkamp & Kuiper, 2010), though much of the research relates solely to primary schools.”

“Mentoring as a Support for Data-Informed Instruction Teachers’ ability to engage in data-informed practice is considered an essential component of broader instructional improvement efforts (Mandinach & Jackson, 2012; Schildkamp & Kuiper, 2010; Wayman, Jimerson, & Cho, 2012).”

“Schildkamp and Kuiper (2010) identify 12 uses, also categorizing them as genuine improvement actions (e.g., personnel decisions, instructional purposes) and unintended responses (e.g., misuse).”

“For example, teachers would be more involved in the decision process at the level of the individual student or class, while school leaders would be more focused on the decision process at the level of the school (Brookhart & Nitko, 2008; Schildkamp & Kuiper, 2010).”

“Research tells us that data use is complicated, even for adults (Mandinach and Gummer 2016; Means et al. 2011; Schildkamp and Kuiper 2010; Wayman and Jimerson 2014).”

“…However, current findings show that schools have substantial deficits and teachers’ actions often are hardly evidence-based at all (see the according articles in Zlatkin-Troitschanskaia, 2016; Zlatkin-Troitschanskaia et al., 2013; see also Schildkamp and Kuiper, 2010; Schildkamp and Teddlie, 2008).”

“In line with a worldwide increasing interest in DBDM (Earl and Katz 2002; Schildkamp and Kuiper 2010), the Dutch government set DBDM as an educational reform in primary schools in 2009, assuming that doing so would lead to increased academic achievement of pupils and improvements in education (Lai…”

“…included informal classroom walkthroughs (Braaten et al., 2017; Means et al., 2010; Taylor & La Cava, 2011), student surveys (Makkonen et al., 2016; Schildkamp & Kuiper, 2010), and hallway conversations with teachers about students’ strengths, weaknesses, and family circumstances (Young et al.,…”

“For example, our participants apparently still drew on their own prior assumptions about the importance of praise when writing feedback (Dagenais et al. 2012; Schildkamp and Kuiper 2010).”

“The analysis of students‘ learning outcomes is an important aspect of DBDM, in which high-quality data from standardised tests are often used to support decision-making processes (Schildkamp & Kuiper, 2010).”

“Schildkamp and Kuiper (2010) point out that teachers need data for instructional change, monitoring progress, identifying areas of need, evaluating their own performance, and supporting conversations with parents.”

“4 Developing a positive attitude towards the use of data among teachers, or “buy-in/belief in data” (Schildkamp and Kuiper, 2010), is necessary if such approaches are to be successfully implemented in schools (Datnow et al.”

“Although supportive school leaders apparently play an important role in enabling teachers to effectively use data (Schildkamp and Kuiper, 2010), according to Daly (2012), many studies have suggested that leaders may not have the skill sets to provide the leadership necessary to support data use.”

“Regardless of 17 years of federal law, many teachers continue to hold negative views of data-driven decision making (DDDM) (Brown, Lake, & Matters, 2011; Dunn, Airola, Lo, & Garrison, 2013a; Remesal, 2011; Schildkamp & Kuiper, 2010).”

“…example, time, collaboration, and accessible data systems facilitate the ability of teachers to use data to inform practice (see Cho and Wayman 2014; Hamilton et al. 2009; Ikemoto and Marsh 2007; Louis et al. 2010; Mandinach 2012; Park and Datnow 2009; Schildkamp and Kuiper 2010; Supovitz 2012).”

“DDDM in education has become expected practice across the globe (Hoogland et al., 2016; Levin & Datnow, 2012; Schildkamp & Kuiper, 2010).”

“Moreover, decisions based on personal knowledge often provide superficial, practical solutions for complex situation (Schildkamp & Kuiper, 2010).”

“Data-driven decision-making, or data-based decision-making (DBDM), has been defined by Schildkamp and Kuiper (2010, p. 482) as “systematically analyzing existing data sources within the school, applying outcomes of analyses to innovate teaching, curricula, and school performance, and, implementing…”

“Educational data cannot directly be used to decision making because data provides no judgment or interpretation, and no sustainable basis of action [13].”

“Unlike data, information has ‘meaning, relevance and purpose by contextualizing, calculating, connecting and summarizing data’ [8] (p.”

“But research has shown that the organizational context of a school can facilitate improved use of data (Schildkamp & Kuiper, 2010).”

“Versions of this framework were published in [3] and [2].”

“Finally, self-efficacy was included as a covariate because teachers’ belief in their ability to impact change in their students’ performance has been found to impact teacher use of data (Ingram et al., 2004; Schildkamp & Kuiper, 2010).”

“Setting expectations for this many goals using a single assessment instrument is quite a formidable challenge (Schildkamp & Kuiper, 2010; Ungerleider, 2006).”

“Predating the rise of data science and LA, these forms of information were used for formative assessment, to provide timely feedback on students’ educational progress and improve learning and teaching approaches [39].”

“The model draws largely on the model by Schildkamp and Kuiper (2010), with the integration of other factors described in reviews by Heitink et al. (2016), Hoogland et al. (2016), Schildkamp and Poortman (2015), and van Geel et al. (2016); on the theory of action for data use described by Schildkamp…”

“The results of this study are consistent with the literature on principals’ datadriven decision making practices (e.g., LaFee, 2002; Mathew, 2002; Means. et al., 2010; Salpeter, 2004; Schildkamp & Kuiper, 2010).”

“…contributors to the context and conditions in which teachers interpret and use assessment data are teachers’ personal and professional understandings, views and experiences with assessment data and how to use it for instructional decision-making (Schildkampf & Kuiper, 2010; Segers & Tillema, 2011).”

“Student data and how to use them to improve teacher effectivity, therefore, are underutilized (Stecker et al. 2005, Schildkamp and Kuiper 2010) and there is a need for initiatives and support to assist teachers in how to use student data for change (Mandinach 2012, Marsh and Farrell 2014).”