Affective learning: Improving engagement and enhancing learning with affect-aware feedback

May combine two claims.

The Bayesian network of the affective state presentation model, trained on 266 cases, achieved accuracy=82.38%, Kappa=0.53, recall true=0.65, recall false=0.87 in 10-fold cross-validation.

“As before, with this data set and employing a 10-fold cross-validation, we have encouraging results (accuracy=82.38%; Kappa=0.53; recall true=0.65; false=0.87).”

May combine two claims.

The Bayesian network of the affective state reasoner, trained on human-annotated data (265 data points), achieved accuracy=79.25%, Kappa=0.50, recall true=0.62, recall false=0.87 in 10-fold cross-validation.

“Figure 4 shows the Bayesian network of the affective state reasoner on which we employed a 10-fold cross-validation that showed promising results (accuracy=79.25%; Kappa=0.50; recall true=0.62; recall false=0.87) and encouraged us to proceed to the full implementation of the system.”

May combine two claims.

In both conditions, students were mainly on task; fewer had an on task conversation, were off task, or reflecting on the task; only seldomly were students gaming the system.

“In both conditions, students were mainly on task. Fewer students had an on task conversation , were off task , or reflecting on the task . Only seldomly were students gaming the system.”

May combine two claims.

In the affect condition, students ignored 74 low-interruptive feedback messages (M=1.80, SD=2.076); in the non-affect condition, 448 (M=12.44, SD=10.814).

“In the affect condition, students ignored 74 of the low-interruptive feedback messages (M=1.80, SD=2.076). In the non-affect condition, students ignored 448 of the low-interruptive feedback messages (M=12.44, SD=10.814).”

May combine two claims.

Students in the affect condition were more likely to view low-interruptive feedback (81%) than students in the non-affect condition (69%), a statistically significant difference with medium effect size (t(75)=2.40, p=.019, d=.55).

“In percentage terms, the students in the affect condition were more likely to view the low-interruptive feedback (81%) than students in the non-affective condition (69%). A t-test showed that this difference was a medium effect size and was statistically significant, t(75)=2.40, p=.019, d=.55.”

May combine two claims.

The combination of speech and interaction enabled detection of frustration (precision: 28.6%; recall: 80.0%) and boredom (precision: 80.0%; recall: 10.8%).

“However, the combination of speech and interaction enabled the detection of frustration (precision: 28.6%; recall: 80.0%) and boredom (precision: 80.0%; recall: 10.8%).”

May combine two claims.

Keywords were not detected often (recall in flow: 16.2%; recall confused: 63.7%).

“Unfortunately, however, as we further discuss in Section 6, keywords were not detected that often (recall in flow : 16.2%; recall confused : 63.7%).”

May combine two claims.

In the affect condition, students were provided with 389 low-interruptive messages (M=9.49, SD=3.551); in the non-affect condition, 1441 (M=40.03, SD=12.098).

“In the affect condition, students were provided with 389 low-interruptive messages (M=9.49, SD=3.551); while in the non-affect condition, students were provided with 1441 low-interruptive messages (M=40.03, SD=12.098)”

May combine two claims.

There were no significant differences between conditions at pre-test (t(75)=.12, p=.91, d=.03) or at post-test (t(75)=1.37, p=.17, d=.32).

“but there were no significant differences between conditions at pre-test (t(75)=.12, p=.91, d=.03) or at post-test (t(75)=1.37, p=.17, d=.32)”

May combine two claims. The quote only partly states this claim.

The effect of affect adaptation was significant only for off-task behaviour, which was only half as frequent in the affect condition as in the non-affect condition (affect M=5.75 vs non-affect M=12.31, t(45)=-2.862, p=.006, d=-0.84).

“Follow-up t-tests showed that the effect of affect adaptation was significant only for off-task behaviour .”

• Also stated in: § 5.5.2 Task behaviour using BROMP · p. 30 · plain

  “Off task behaviour was only half as frequent in the affect condition as in the non-affect condition.”

May combine two claims.

The precision of detection of in flow from interaction data was 82.5%, and recall was 89.2%.

“The precision of the detection of in flow from the interaction data was not as high (82.5%), however, the recall of in flow from the interaction data was high (89.2%).”

May combine two claims.

When keywords were detected, the precision of automatic detection was 100.0% for in flow and 96.7% for confused.

“When keywords were detected, the match with the human-detected student affect was very high ( in flow : 100.0% precision; confused : 96.7% precision).”

May combine two claims.

In both conditions, students were mainly in flow, followed by confusion and boredom; only rarely were students frustrated, delighted, or surprised.

“In both conditions, students were mainly in flow . This was followed by confusion and boredom . Only rarely were students frustrated , delighted , or surprised .”

May combine two claims. The quote only partly states this claim.

The effect of affect adaptation was statistically significant only for boredom, which was only half as frequent in the affect condition as in the non-affect condition (affect M=9.74 vs non-affect M=20.38, t(45)=-2.739, p=.009, d=-0.81).

“Follow-up t-tests showed that the effect of affect adaptation was statistically significant only for boredom (please see Table 6 for statistical details, statistically significant results highlighted in bold font), which was only half as frequent in the affect condition as in the non-affect condition.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Internal consistency of the fractions knowledge scale was α =.57 at both pre- and post-test.

“Internal consistency of this scale was α =.57 at both pre- and post-test.”

Classified as probably not a substantial claim.

A randomly allocated subset of students (affect condition: N=25; non-affect condition: N=22) were monitored using BROMP.

“While students engaged with the system, a randomly allocated subset of students (affect condition: N=25; non-affect condition: N=22) sat at computers in the centre of the computer room in a way that allowed researchers to walk all around them, were monitored using the Baker-Rodrigo Ocumpaugh Monitoring Protocol (BROMP, Ocumpaugh et al., 2012).”

• cites Ocumpaugh et al., 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

80 students took part in the summative evaluation, but the final dataset comprises 77 students.

“80 students took part in the summative evaluation, although for several reasons our final dataset comprises 77 students.”

Classified as probably not a substantial claim.

Adaptive feedback is essential in exploratory learning environments.

“Adaptive feedback (which is well-known to be essential in exploratory learning environments, Kirschner et al., 2006) is provided to students based on their interactions with the system and their affective states.”

• cites Kirschner et al., 2006 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

BROMP specifies strict guidelines for how affective states and task behaviour are detected.

“The BROMP specifies strict guidelines for how affective states and task behaviour are detected.”

• cites Ocumpaugh et al., 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Affect boosts can help enhance a student's motivation to solve a particular learning task.

“As described in (Woolf et al., 2009), affect boosts can help to enhance a student's motivation to solve a particular learning task.”

• cites Woolf et al., 2009 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Students in a state of heightened engagement (flow) are absorbed in the learning material and are thus primed for learning.

“For example, Csikszentmihalyi (1990) argues that students in a state of heightened engagement, that he calls in flow , are absorbed in the learning material and are thus primed for learning.”

• cites Csikszentmihalyi, 1990Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Effective support in learning situations needs to answer when support should be provided, what it should contain, and how it should be presented.

“Carenini et al. (2014) describe how effective support in learning situations needs to answer three main questions: (i) When should the support be provided? (ii) What should the support contain? And, (iii) how should the support be presented?”

• cites Carenini et al., 2014Not checked yet

  Not checked in the free run.

May combine two claims.

Being over- or under-challenged may trigger boredom or anxiety.

“The SNA sequences the tasks according to the student's level of challenge, which is inferred from the student's interaction using the amount of feedback provided as a key indicator, in order to avoid students being over- or under-challenged, which may trigger boredom or anxiety (as described by Acee et al., 2010).”

• cites Acee et al., 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The system includes two Bayesian networks, one for feedback type and one for feedback presentation, which were trained with data from earlier Wizard-of-Oz studies in which their predictive accuracy was tested.

“It includes two Bayesian networks, one for each adaptation (feedback type and feedback presentation ), which were trained with data from earlier Wizard-of-Oz studies in which their predictive accuracy was tested.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Confusion is likely under certain conditions to promote learning.

“D'Mello et al. (2014), for example, elaborate how confusion, which initially might be thought of as a negative affective state, is likely under certain conditions to promote learning.”

• cites D'Mello et al., 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two series of sessions, one for each condition, were undertaken over several days in each school at a variety of times of day which were balanced as far as possible between conditions.

“Two series of sessions, one for each condition, were undertaken over several days in each school (in the school computer rooms, each of which was equipped with around 30 individual computers) at a variety of times of day which were balanced as far as possible between conditions.”

Classified as probably not a substantial claim.

Students were randomly allocated one version of the knowledge test at pre-test and the other at post-test.

“Students were randomly allocated one version at the first time of measurement (pre-test) and the other version at the second time of measurement (posttest).”

Classified as probably not a substantial claim.

A further Wizard-of-Oz study investigated whether there was a difference in a student's affective state when different types of feedback were presented in a low-interruptive way versus a high-interruptive way.

“We conducted a further Wizard-of-Oz study that investigated if there was a difference in a student's affective state when the different types of feedback were either presented in the low-interruptive way, with the light bulb, or in the high-interruptive way, with the pop-up window (Grawemeyer et al., 2015a).”

• cites Grawemeyer et al., 2015aNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Students were roughly stratified according to previous teacher assessments of mathematical ability and then randomly allocated to two sub-groups with approximately equal size and similar numbers of high, middle and low achieving students.

“The participating students were roughly stratified, according to previous teacher assessments of the children's mathematical ability, and then randomly allocated to two sub-groups (approximately equal in size, with each group having approximately the same number of high, middle and low achieving students).”

May combine two claims.

Boredom is associated with poor learning and with gaming the system.

“In addition, Baker et al. (2010) found that certain types of affective states, such as boredom, were associated with poor learning and with gaming the system.”

• cites Baker et al., 2010 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Asking students to talk without them perceiving any benefit could have been found intrusive by some students.

“TALK ALOUD prompts were also not provided because asking the students to talk without them perceiving any benefit could have been found intrusive by some students (Mavrikis et al., 2014; Grawemeyer et al., 2015a,b).”

• cites Mavrikis et al., 2014Not checked yet

  Not checked in the free run.

• cites Grawemeyer et al., 2015a,bNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The researchers who undertook the coding were trained in the BROMP method.

“The researchers who undertook the coding, and who were trained in the BROMP method, recorded the student affective states and task behaviour data using the Human Affect Recording Tool (HART) Android mobile app.”

Classified as probably not a substantial claim.

Grawemeyer et al. (2015a) describe that when students are in a particular affective state, such as frustration, low-interruptive feedback might be ignored because of cognitive load.

“Grawemeyer et al. (2015a) describe that when students are in a particular affective state, such as frustration , low-interruptive feedback might be ignored because of cognitive load.”

• cites Grawemeyer et al. (2015a)Not checked yet

  Not checked in the free run.

May combine two claims.

The analysis confirms the potential of automatic detection of affective states using information from speech and that the combination of interaction data is promising.

“Nevertheless, overall, our analysis confirms the potential of the automaticdetection of affective states using information from speech (as identified in other work such as Forbes-Riley and Litman, 2011a) and that the combination of interaction data is promising and warrants further research.”

• cites Forbes-Riley and Litman, 2011aNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This is a well known issue in evaluating interactive adaptive systems.

“This is a well known issue in evaluating interactive adaptive systems (c.f. Paramythis et al., 2010).”

• cites Paramythis et al., 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The adaptations of the feedback type as well as the adaptation of the feedback's presentation based on the student's affective state had an effect on their engagement with the task.

“Here, the adaptations of the feedback type as well as the adaptation of the feedback's presentation based on the student's affective state had an effect on their engagement with the task.”

May combine two claims.

Responding appropriately to student affect is likely to encourage students in a more productive engagement, which in turn will lead to better learning outcomes.

“by responding appropriately to student affect, teachers (not just technologies) are likely to encourage students in a more productive engagement, which in turn will lead to better learning outcomes”

• AssumptionThe authors state this as an assumption: Boredom and off-task behaviour can have a negative impact on learning.

May combine two claims.

Both annotators were trained in the BROMP method, and one of them was not fully aware of the existence of different conditions nor the exact research questions.

“both annotators were trained in the BROMP method, and one of them was not fully aware of the existence of different conditions nor the exact research questions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The difference in the rate of viewing and ignoring low-interruptive feedback between the two conditions may result from the greater frequency of low-interruptive feedback in the non-affect condition.

“However, the difference in the rate of viewing and ignoring lowinterruptive feedback between the two conditions may result from the greater frequency of low-interruptive feedback in the non-affect condition.”

May combine two claims.

Forbes-Riley and Litman (2011a) describe that the affect-aware version of ITSPOKE led to higher learning than a control condition, although the difference was only statistically significant for a subset of students (those who received the most uncertainty adaptations).

“Our results support Forbes-Riley and Litman (2011a) who describe how the affect-aware version of ITSPOKE led to higher learning than a control condition, although the difference was only statistically significant for a subset of students (those who received the most uncertainty adaptations).”

• cites Forbes-Riley and Litman (2011a)Not checked yet

  Not checked in the free run.

May combine two claims.

The authors were not able to fully hide the experimental condition from the annotators, which could have implicitly biased the annotations.

“we were not able to fully hide the experimental condition from the annotators - the setting and whether or not the students were overtly speaking could have made this obvious and implicitly bias the annotations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The relatively large number of low-interruptive feedback messages in the non-affect condition compared to the affect condition might have had additive effects such that students became 'immune' and systematically disregarded the low-interruptive feedback messages.

“However, the relatively large number of low-interruptive feedback messages in the non-affect condition compared to the affect condition (N=1441 vs N=389) might have had some additive effects such that, after a point, students became 'immune' and systematically disregarded the low-interruptive feedback messages.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Students in both conditions were mainly on task, which might be explained by the nature or novelty of the exploratory learning environment.

“Students in both conditions were mainly on task , which again might be explained by the nature or novelty of the exploratory learning environment, which appeared to engage the students.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The reason low pre-knowledge students benefitted more is presumably that they experienced more negative affective states than high pre-knowledge students, providing more opportunities for the system to move them to a positive affective state.

“This is presumably because these low-attaining students experienced more negative affective states than those students who had high pre-knowledge, and thus there were more opportunities for the system to move them to a positive affective state.”

• AssumptionRests on an assumption: Low-attaining students experienced more negative affective states than high pre-knowledge students.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The authors followed a layered process to evaluate different aspects of the learner model, building it on top of previously collected data, and internally validated its robustness.

“we followed a layered process to evaluate different aspects of the learner model, building it on top of previously collected data (Grawemeyer et al., 2015a,b), and we internally validated its robustness.”

• cites Grawemeyer et al., 2015a,bNot checked yet

  Not checked in the free run.

May combine two claims.

The lower boredom in the affect condition can be explained by the way feedback was adapted: students in the affect condition received feedback that varied more often and the variation was directly in response to student affect.

“The reason for this can be found in the way the feedback was adapted. Students in the affect condition received feedback that varied more often than did those in the non-affect condition. In addition, the variation of the feedback types within the affect condition was directly in response to student affect.”

May combine two claims.

Students in the affect condition were more engaged with the tasks and spent more time on solving the task instead of trying to move quickly on to the next task without finishing it.

“This suggests that students in the affect condition were more engaged with the tasks and spent more time on solving the task instead of trying to move quickly on to the next task without finishing it.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Intrinsic systematic differences between the two conditions, such as differences in student affect, behaviour and/or learning, may have been introduced simply by monitoring and responding to affect rather than necessarily individualising the response to each student's affective states.

“It is worth noting that intrinsic systematic differences between the two conditions, like differences in student affect, behaviour and/or learning, may have been introduced simply by monitoring and responding to affect rather than necessarily individualising the response to each student's affective states.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reason students were mainly in positive affective states might be the nature of the learning environment (the exploratory nature of Fractions Lab).

“Similar to Conati and MacLaren (2009), the reason might be the nature of the learning environment (in our case, the exploratory nature of Fractions Lab ).”

• cites Conati and MacLaren (2009) sourceNot checked yet

  Not checked in the free run.