The Next Black Box of Formative Assessment: A Model of the Internal Mechanisms of Feedback Processing

Probably not a substantial claim: may not pass S3 (load-bearing).

Kingston and Nash (2011) found a modest, weighted mean effect size of 0.20 of formative assessment on learning in a meta-analysis.

“For example, Kingston and Nash (2011) found a modest, weighted mean effect size of 0.20 of formative assessment on learning in a meta-analysis.”

• cites Kingston and Nash (2011)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Hattie (2009) concluded that under the right conditions feedback in a formative context can contribute significantly to students' achievement, with an average effect size of 0.73.

“In a review of 12 meta-analyses on feedback in classrooms, Hattie (2009) concluded that, under the right conditions, feedback in a formative context can contribute significantly to students' achievement, with an average effect size of 0.73.”

• cites Hattie (2009)Not checked yet

  Not checked in the free run.

May combine two claims.

Empirical studies on students' perceptions of feedback include individuals' emotions, judgments, conceptions, and uses of feedback, suggesting that these processes are all relevant to responses to feedback and might be co-occurring.

“Empirical studies on students' perceptions of feedback include but are not limited to individuals' emotions, judgments, conceptions, and uses of feedback, suggesting that these processes are all relevant to responses to feedback and might be co-occurring.”

Classified as probably not a substantial claim.

Feedback for students is a key element of formative assessment.

“Feedback for students is a key element of formative assessment (Black and Wiliam, 2009).”

• cites Black and Wiliam, 2009 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Panadero and Lipnevich (2021) conceptualize feedback processing as a mediator between the feedback provided and its cognitive, motivational, and metacognitive effects.

“As a remedy, they conceptualize feedback processing as a mediator between the feedback provided and its cognitive, motivational, and metacognitive effects, and propose a model that will be included in the review section of this paper.”

• cites Panadero and Lipnevich (2021)Not checked yet

  Not checked in the free run.

May combine two claims.

Graham et al. (2015) meta-analysis of true and quasi-experimental studies on formative assessment of writing in grades one to eight yielded average weighted effect sizes of 0.87 for teacher feedback, 0.58 for peer feedback, 0.62 for self feedback, and 0.38 for computer feedback.

“The metaanalysis by Graham et al. (2015), which examined true and quasi-experimental design studies on formative assessment of writing in grades one to eight, categorized the source of formative feedback in terms of teachers, peers, self, and computers, and yielded average weighted effect sizes of 0.87, 0.58, 0.62, and 0.38, respectively.”

• cites Graham et al. (2015) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Gamlem and Smith (2013) found that these perceptions left students feeling frustrated and attacked instead of motivated.

“Emotionally, these perceptions left students feeling frustrated and attacked instead of motivated.”

• cites Gamlem and Smith (2013) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Conceptions, perceptions, and emotions are distinct constructs.

“While conceptions, perceptions, and emotions might often co-exist, they are distinct constructs that we will parse in this paper to better identify and operationalize them as distinct internal mechanisms of feedback processing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our review revealed that scholars used different terminology to represent perceptions, including beliefs, conceptions, voice, views, and perspectives.

“Our review (see also Fisk, 2017) revealed that scholars used different terminology to represent perceptions, including beliefs, conceptions, voice, views, and perspectives.”

• cites Fisk, 2017 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Empirically, researchers tended to focus on conceptions of, perceptions of, and emotions elicited by feedback as key internal processes.

“Empirically, researchers tended to focus on conceptions of, perceptions of, and emotions elicited by feedback as key internal processes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Black and Wiliam found evidence that formative assessment could raise standards and increase student learning by yielding actionable information on learning as it occurs.

“Upon reviewing over 250 articles related to formative assessment, Black and Wiliam found evidence that this promising new conception of assessment-formative assessment-could raise standards and increase student learning by yielding actionable information on learning as it occurs.”

• cites Black and WiliamNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Several influential reviews have found an overall positive association between formative assessment and student learning.

“Several influential reviews have found an overall positive association between formative assessment and student learning.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qualitative studies have shown that various factors, including students' emotions and judgments about the feedback, might influence behavioral responses, as well as how well students understand the feedback.

“Qualitative studies have shown that various factors, including students' emotions and judgments about the feedback, might influence behavioral responses, as well as how well students understand the feedback (e.g., Lipnevich and Smith, 2009b;Wingate, 2010;Winstone et al., 2021).”

• cites Lipnevich and Smith, 2009bNot checked yet

  Not checked in the free run.

• cites Wingate, 2010 sourceNot checked yet

  Not checked in the free run.

• cites Winstone et al., 2021 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Shute (2008) noted that the specific mechanisms relating feedback to learning are still mostly murky, with very few general conclusions.

“Shute (2008) noted in her review of the literature on formative feedback that “despite the plethora of research on the topic, the specific mechanisms relating feedback to learning are still mostly murky, with very few (if any) general conclusions” (p.156).”

• cites Shute (2008) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Wingate (2010) suggests that critical feedback tended to be a motivator for improving writing for undergraduate students with higher levels of self-efficacy, whereas for students with lower levels of self-efficacy feedback was discouraging and promoted negative emotions.

“An exception is a study by Wingate (2010), which suggests that critical feedback tended to be a motivator for improving writing for undergraduate students with higher levels of self-efficacy than students with lower levels of self-efficacy, for whom feedback was discouraging and promoted negative emotions.”

• cites Wingate (2010) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Internal cognitive and affective mechanisms are not as well defined as observable behavioral responses.

“Internal cognitive and affective mechanisms are not as well defined as observable behavioral responses.”

Classified as probably not a substantial claim.

Panadero and Lipnevich (2021) attribute the murkiness to scholarship focusing on the role of the teacher, the process of feedback delivery, and the type of feedback message, rather than on the uptake of feedback.

“Panadero and Lipnevich (2021) attribute this murkiness to the tendency of scholarship to focus on the role of the teacher, the process of feedback delivery, and the type of feedback message, rather than on the uptake of feedback.”

• cites Panadero and Lipnevich (2021)Not checked yet

  Not checked in the free run.

May combine two claims.

Conceptions of feedback are partially task general and are also influenced by the assessment culture where learning happens.

“While conceptions of feedback are partially task general, they are also influenced by the assessment culture where learning happens.”

• cites Havnes et al., 2012 sourceNot checked yet

  Not checked in the free run.

• cites Harris et al., 2014 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Wisniewski et al. (2020) suggest that to better understand the effects of feedback on achievement, it is essential to differentiate between cognitive, motor, motivational, and behavioral outcomes and the types and amount of feedback information conveyed.

“Similarly, Wisniewski and colleagues (2020) suggest that to better understand the effects of feedback on achievement, it is essential to differentiate between cognitive, motor, motivational, and behavioral outcomes and the types and amount of feedback information conveyed.”

• cites Wisniewski and colleagues (2020)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

High school students who have been exposed to student-centered assessment found value in using feedback.

“In contrast, high school students who have been exposed to student-centered assessment found value in using feedback (Harris et al., 2014).”

• cites Harris et al., 2014 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Other reviews and meta-analyses have reported that feedback tends to have a positive association with learning and achievement.

“Consistent with Hattie, other reviews and meta-analyses have reported that feedback tends to have a positive association with learning and achievement (Hattie and Timperley, 2007;Shute, 2008;Lipnevich and Smith, 2009a;Wiliam, 2010;2013;Ruiz-Primo and Li, 2013;Van der Kleij et al., 2015;Wisniewski et al., 2020).”

• cites Hattie and Timperley, 2007 sourceNot checked yet

  Not checked in the free run.

• cites Shute, 2008 sourceNot checked yet

  Not checked in the free run.

• cites Lipnevich and Smith, 2009aNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Other non-metaanalytic reviews echoed that improvements in student learning can be meaningful, if not consistently as large as reported in early studies, and the positive effects hold across different age groups, core school subjects, and countries.

“Other non-metaanalytic reviews (Wiliam et al., 2004;Shute, 2008;Bennett, 2011;Filsecker and Kerres, 2012;McMillan et al., 2013) echoed the findings: Improvements in student learning can be meaningful, if not consistently as large as reported in early studies, and the positive effects hold across different age groups, core school subjects, and countries.”

• cites Wiliam et al., 2004 sourceNot checked yet

  Not checked in the free run.

• cites Shute, 2008 sourceNot checked yet

  Not checked in the free run.

• cites Bennett, 2011 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Kluger and DeNisi (1996) reported a mean effect size of 0.41 for formative feedback on performance, but 38% of these effects were negative.

“For example, Kluger and DeNisi (1996) reported a mean effect size of 0.41 for formative feedback on performance, but 38% of these effects were negative.”

• cites Kluger and DeNisi (1996) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Studies on task interest have shown that it plays a significant role in students' emotions, goal-setting, academic achievement, and behaviors.

“Studies on task interest have shown that it plays a significant role in students' emotions, goal-setting (Horvath et al., 2006), academic achievement, and behaviors (Harackiewicz et al., 2000;Harackiewicz and Hulleman, 2010;Boekaerts, 2011).”

• cites Horvath et al., 2006 sourceNot checked yet

  Not checked in the free run.

• cites Harackiewicz et al., 2000 sourceNot checked yet

  Not checked in the free run.

• cites Boekaerts, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The reviewed models illustrate the hypothesized processes and interactions between and among identified factors.

“They also illustrate the hypothesized processes and interactions between and among identified factors.”

May combine two claims.

Ten of the 14 models focus on students' internal processing of feedback but sometimes do not describe it in sufficient detail.

“Ten of the 14 models (indicated by * in Table 1) focus on students' internal processing of feedback but sometimes do not describe it in sufficient detail (e.g., Panadero and Lipnevich, 2021).”

• cites Panadero and Lipnevich, 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Empirical studies on responses to feedback have focused on students' conceptions of, perceptions of, and emotions elicited by feedback, as well as behavioral responses to it.

“Empirical studies on responses to feedback have focused on students' conceptions of, perceptions of, and emotions elicited by feedback, as well as behavioral responses to it.”

May combine two claims.

Emotions could more easily change attitudes that are weak, whereas emotions would further strengthen or bias attitudes that are strong.

“That is, emotions could more easily change attitudes that are weak, whereas emotions would further strengthen or bias attitudes that are strong.”

May combine two claims.

Initial states are usually represented as antecedents to responses to feedback, but they also end every feedback loop, perhaps altered by the previous learning experiences.

“They are usually represented as antecedents to responses to feedback, but they also end every feedback loop, perhaps altered by the previous learning experiences (Tulis et al., 2016).”

• cites Tulis et al., 2016 sourceNot checked yet

  Not checked in the free run.

The quote only partly states this claim.

Taken together, the reviewed models posit that responses to feedback include initial states, internal responses to feedback, and overt external responses to feedback.

“Taken together, the models posit that responses to feedback include initial states, which are factors and processes internal to the learner, such as task motivation, self-motivational beliefs, conceptions or beliefs about assessment and feedback, and prior knowledge; internal responses to feedback, which include regulatory learning processes, interpretations and perceptions of, and attributions”

May combine two claims.

When students receive feedback as input, they decide on their next steps, which leads to behavioral outputs.

“Combined with the empirical findings reviewed above, these models show that when students receive feedback as input, they decide on their next steps, which leads to behavioral outputs (Lipnevich et al., 2016).”

• cites Lipnevich et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Feedback is the external factor that activates internal processing.

“Feedback is the external factor that activates internal processing (A in Figure 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Emotions can influence thinking differently depending on the timing and intensity of the thoughts.

“For example, emotions can influence thinking differently depending on the timing and intensity of the thoughts (Petty and Brinol, 2014).”

• cites Petty and Brinol, 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Initial motivational states drive both affective and cognitive/metacognitive responses to feedback.

“Hypothesized processes include initial motivational states (B), which drive both affective and cognitive/metacognitive responses to feedback (C-E); that is, the internal processes in the rounded dark grey box are rooted in students' existing knowledge, beliefs, and motivational states, filtering information that learners receive and influencing the way they receive it.”

May combine two claims.

Students might decide to engage with the feedback at a surface level, through reflection on the comments, or with an in-depth synthesis of the feedback to find common themes and methods to address them.

“In turn, students might decide to engage with the feedback at a surface level, through reflection on the comments, or with an in-depth synthesis of the feedback to find common themes and methods to address them (Winstone et al., 2021).”

• cites Winstone et al., 2021 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

When feedback from the teacher is clear, it can help students understand what they know (or do not know) and perhaps even how they know it (or do not know it).

“When feedback from the teacher is clear, it can help students understand what they know (or do not know) and perhaps even how they know it (or do not know it).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The reviewed models are comprehensive, comprising contextual, external, internal, and personal factors.

“The models are comprehensive, comprising contextual, external, internal, and personal factors.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The model posits an interplay between emotions, interpretations, and decisions, suggesting that they could change as students process the feedback.

“The model also posits an interplay between these factors, suggesting that emotions, interpretations, and decisions could change as students process the feedback.”

May combine two claims.

Katz et al. (2006) found that for students with high interest, interest was related to intrinsic motivation for both genders, with a slight decrease from the presence to the absence of positive feedback.

“For the students with high interest, interest was related to intrinsic motivation for both genders, with a slight decrease from the presence to the absence of positive feedback.”

• cites Katz et al. (2006) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Much is known about the effects of feedback on self-efficacy but far less about the relationships between self-efficacy and students' responses to feedback.

“In terms of self-efficacy, much is known about the effects of feedback on self-efficacy (e.g., Chan and Lam, 2010;Hier and Mahony, 2018) but far less about the relationships between selfefficacy and students' responses to feedback.”

• cites Chan and Lam, 2010Not checked yet

  Not checked in the free run.

• cites Hier and Mahony, 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Kleij and Lipnevich's review found a relationship between emotions and perceptions of feedback, both negative and positive, and an association between negative emotions and vague or unclear feedback.

“Kleij and Lipnevich's (2021) review of the literature found a relationship between emotions and perceptions of feedback, both negative and positive, and an association between negative emotions and vague or unclear feedback.”

• cites Kleij and Lipnevich's (2021) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Students make various causal inferences about why they received the feedback they did, each influencing the decisions about next steps.

“Students make various causal inferences about why they received the feedback they did, each influencing the decisions about next steps.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Students judged the feedback as more valuable and desirable when it required them to think about their work.

“Students judged the feedback as more valuable and desirable by students when it required them to think about their work (Gamlem and Smith, 2013).”

• cites Gamlem and Smith, 2013 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Emotion serves as a tool for monitoring the gaps in where students are in relation to their goals and as a guide for their next steps.

“Tulis and colleagues' model highlights two roles that emotion plays: First, it serves as a tool for monitoring the gaps in where students are in relation to their goals; second, it serves as a guide for their next steps.”

Probably not a substantial claim: may not pass S4 (based).

Once feedback filters through students' initial states, it enters the cognitive and affective processing phases: students' responses to feedback.

“Once feedback filters through students' initial states, it enters the cognitive and affective processing phases: students' responses to feedback (C-E).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Internal processes are rooted in students' existing knowledge, beliefs, and motivational states, filtering information that learners receive and influencing the way they receive it.

“Hypothesized processes include initial motivational states (B), which drive both affective and cognitive/metacognitive responses to feedback (C-E); that is, the internal processes in the rounded dark grey box are rooted in students' existing knowledge, beliefs, and motivational states, filtering information that learners receive and influencing the way they receive it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reviewed models are grounded in formative assessment and self-regulated learning theories.

“In general, the models vary in type and include cycles, typologies, frameworks, theoretical arguments, processes, and taxonomies; they are grounded in formative assessment and self-regulated learning theories.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Forsythe and Johnson (2017) reported that undergraduate students with fixed mindsets, compared to those with growth mindsets, tended to score lower on positive adaptive factors and higher on negative affect regulating factors and defensive behaviors.

“For example, Forsythe and Johnson (2017) reported that undergraduate students with fixed mindsets, compared to those with growth mindsets, tended to score lower on positive adaptive factors (i.e., self-observation, self-assertion, anticipation, sublimation, and humor) and higher on negative affect regulating factors (i.e., intellectualization, dissociation, and isolation) and defensive behaviors (”

• cites Forsythe and Johnson (2017)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The reviewed models acknowledge the importance of contextual, external factors, including the nature of the feedback, the task, and the culture of assessment and learning in which the feedback is given.

“The models also acknowledge the importance of contextual, external factors, including the nature of the feedback, the task, and the culture of assessment and learning in which the feedback is given.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Anger is a negative, activating outcome emotion that triggers the intrinsic motivation to act on the outcome.

“Anger is a negative, activating outcome emotion that triggers the intrinsic motivation to act on the outcome (Pekrun et al., 2007).”

• cites Pekrun et al., 2007Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Formative assessment is an inferential process in which external feedback is interpreted to inform decisions and behavioral responses.

“From a learner's perspective, formative assessment is an inferential process in which external feedback is interpreted to inform decisions and behavioral responses (Bennett, 2011).”

• cites Bennett, 2011 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The first author conducted a search of the post-1998 literature on feedback processing, perceptions, responses, reactions, and receptivity to feedback within a formative assessment context, resulting in over 200 potential screened sources and 100 sources to be reviewed.

“The first author conducted a search of the post-1998 literature on feedback processing, perceptions, responses, reactions, and receptivity to feedback within a formative assessment context.This resulted in over 200 potential screened sources and 100 sources to be reviewed.”

May combine two claims.

VandeWalle et al. (2001) found that an initially positive relationship between performance-approach goal orientation and performance decreased when undergraduate students received performance feedback; there was no change in relationships between mastery-goal orientation and performance-avoidant orientation and performance.

“For example, an initially positive relationship between performance-approach goal orientation and performance decreased when undergraduate students received performance feedback; there was no change in relationships between mastery-goal orientation and performance-avoidant orientation and performance (VandeWalle et al., 2001).”

• cites VandeWalle et al., 2001 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Feedback processing is often cyclical and interactive: the outputs of a previous process inform the inputs of the next, and the processes interact with each other.

“While for the purposes of this study Figure 1 represents the processes sequentially, from inputs to outputs, we acknowledge that feedback processing is often cyclical and interactive: the outputs of a previous process inform the inputs of the next, and the processes interact with each other (Panadero and Lipnevich, 2021).”

• cites Panadero and Lipnevich, 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This body of research also reveals the role that self-efficacy, goal orientation, mindset, and task motivation play in how students make decisions and respond behaviorally to feedback.

“This body of research also reveals the role that self-efficacy, goal orientation, mindset, and task motivation play in how students make decisions and respond behaviorally to feedback.”

May combine two claims.

Research on students' perceptions of feedback involves a wide range of examined variables and lacks a common theoretical framework.

“We concur with the conclusion made by Van der Kleij and Lipnevich (2021) in their critical scoping review of students' perceptions of feedback: research on this topic involves “a wide range of examined variables” (p.367) and lacks a common theoretical framework.”

• cites Van der Kleij and Lipnevich (2021) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Theoretical models echo the importance of these factors and posit additional factors, including interpretations of feedback.

“Theoretical models echo the importance of these factors (i.e., emotions, perceptions, decision-making, motivation) and posit additional factors, including interpretations of feedback (i.e., meaning making, judgments, and attributions).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The model proposed in this paper operationalizes responses to feedback in measurable ways, so we can transition from postulating to measuring.

“The model proposed in this paper (Figure 1) operationalizes responses to feedback in measurable ways, so we can transition from postulating to measuring.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Feedback is discipline-specific.

“Given that feedback is discipline-specific (Andrade et al., 2019), research is also needed on how responses to feedback differ by discipline and grade level.”

• cites Andrade et al., 2019 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The model includes variables and hypothesized processes with rigorous theoretical and empirical support.

“In this paper, we introduced the next black box of formative assessment and presented the model of the internal mechanisms of feedback processing, a theoretical framework of students' responses to feedback that includes variables and hypothesized processes with rigorous theoretical and empirical support.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

This model of the mechanisms of feedback processing can shed light into the next black box by guiding empirical studies of these processes and their relationships with each other, thereby moving in the direction of treating feedback as something received.

“This model of the mechanisms of feedback processing can shed light into the next black box by guiding empirical studies of these processes and their relationships with each other, thereby moving in the direction of treating feedback as something received.”