Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation

Probably not a substantial claim: may not pass S3 (load-bearing).

Group 6 shows a desired (e.g., balanced) interaction according to the goals.

“According to the goals, Group 6 shows a desired (e.g., balanced) interaction.”

• AssumptionRests on an assumption: Group dynamics and interactivity are to a large extent predictive for the collaborative nature of learning processes.

May combine two claims.

In groups 1-5, there are some problems with collaboration because either not all participants are involved or some participants' activities are limited.

“For instance, if a teacher specified a balanced interactivity for a task in which a group member should contribute to active discussion on the analysis results, it is obvious from Figure 8 that in groups 1-5, there are some problems with collaboration because either not all participants are involved or some participants' activities are limited.”

• AssumptionRests on an assumption: Group dynamics and interactivity are to a large extent predictive for the collaborative nature of learning processes.

Classified as probably not a substantial claim.

In the context of feedback research, cognitive and behavioral approaches translate into two major forms: explanations targeted at improving cognitive dimensions of knowledge and guidance to influence a learner's behavior.

“In the context of feedback research, these approaches translate into two major forms: explanations targeted at improving cognitive dimensions of knowledge (e.g., understanding) and guidance to influence a learner's behavior (e.g., engaging in a specific type of activity believed to be related to a successful learning path; Sedrakyan, 2016).”

• cites Sedrakyan, 2016Not checked yet

  Not checked in the free run.

May combine two claims.

Most research on educational dashboards lacks both theoretical support from recent advancements in the learning sciences and an evidence-informed foundation for choosing data to observe and assess learning processes.

“According to Sawyer (2014), most research on educational dashboards lacks both theoretical support from recent advancements in the learning sciences and an evidenceinformed foundation for choosing the data that can assist in observing and assessing learning processes to identify the feedback needs of learners/teachers.”

• cites Sawyer (2014)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Feedback mechanisms that the environment or peers provide can also be considered as a form of co-regulated learning.

“Although self-regulation concerns individual adaptation, feedback mechanisms that the environment or peers provide can also be considered as a form of co-regulated learning (Isohätälä, Järvenoja, & Järvelä, 2017).”

• cites Isohätälä, Järvenoja, & Järvelä, 2017Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Cognitive feedback aims to improve learners' understanding of intermediate solutions allowing them to engage in self-regulatory learning mechanisms.

“This type of feedback aims to improve learners' understanding of intermediate solutions allowing them to engage in self-regulatory learning mechanisms (van Merriënboer & Kirschner, 2012).”

• cites van Merriënboer & Kirschner, 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Learner goal orientations and differences in achievement levels need to be carefully considered in the design of any intervention, as the resulting approaches and tools can affect students' interpretations of their data and subsequent academic success.

“Furthermore, recent studies on the effectiveness of existing dashboards suggest that learner goal orientations (Lonn et al., 2015) and differences in achievement levels (Park & Jo, 2015) need to be carefully considered in the design of any intervention, as the resulting approaches and tools can affect students' interpretations of their data and subsequent academic success.”

• cites Lonn et al., 2015 sourceNot checked yet

  Not checked in the free run.

• cites Park & Jo, 2015Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Self-regulated and strategic learning involves experimenting with, and learning about, effective strategies for regulating aspects of their own, peers', and groups' shared learning processes, including planning, setting goals, organizing, monitoring, and adapting.

“In practice, self-regulated and strategic learning involves experimenting with, and learning about, effective strategies for regulating aspects of their own, peers', and groups' shared learning processes (Winne, Hadwin, & Perry, 2013), including planning, setting goals, organizing, monitoring, and adapting.”

• cites Winne, Hadwin, & Perry, 2013Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Learning may be explained by a combination of two basic approaches: cognitive theories and behavioral theories.

“Even though psychologists and educators are not in complete agreement, most agree that learning may be explained by a combination of two basic approaches: cognitive theories (i.e., cognitivism, which views the learning process as a step-by-step knowledge construction process) and behavioral theories (i.e., behaviorism, in which learning is defined as a change of the behavior of a learner by rein”

• cites Tomic, 1993Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Co-regulated learning occurs when learners' regulatory activities are guided, supported, shaped, or constrained by others, such as peers or teachers, and the social system, including the learning environment.

“Co-regulated learning (CoRL) occurs when learners' regulatory activities are guided, supported, shaped, or constrained by others, such as peers or teachers, and the social system, including the learning environment (Hadwin, Järvelä & Miller, 2017).”

• cites Hadwin, Järvelä & Miller, 2017Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The design artefact presented in the article uses a conceptual model that visualizes the relationships between dashboard design and the learning sciences to provide cognitive and behavioral process-oriented feedback to learners and teachers to support regulation of learning.

“The design artefact presented in this article uses a conceptual model that visualizes the relationships between dashboard design and the learning sciences to provide cognitive and behavioral process-oriented feedback to learners and teachers to support regulation of learning.”

• AssumptionRests on an assumption: The conceptual model presented is a valid guide for designing LAD feedback to support regulation of learning.

Classified as probably not a substantial claim.

Regulation of learning and performance is central to research on feedback.

“The regulation of learning and performance is central to research on feedback (Butler & Winne, 1995).”

• cites Butler & Winne, 1995 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Behavioral change and improved performance were observed when supporting learners in the interpretation of visualizations.

“Empirical studies show that behavioral change and improved performance were observed when supporting learners in the interpretation of visualizations (Sedrakyan, 2016).”

• cites Sedrakyan, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

In terms of data analysis approaches, process (sequence) analytics should be considered as opposed to the statistical and data-mining approaches currently widely applied in research on learning analytics.

“This type of process can be referred to as a sequencing of cognitive and behavioral activities, suggesting that in terms of data analysis approaches, we also need to consider the role of process (sequence) analytics (Sedrakyan, 2016;Sedrakyan, Snoeck & De Weerdt, 2014) as opposed to the statistical and data-mining approaches currently widely applied in research on learning analytics.”

• cites Sedrakyan, 2016Not checked yet

  Not checked in the free run.

• cites Sedrakyan, Snoeck & De Weerdt, 2014Not checked yet

  Not checked in the free run.

May combine two claims.

The process-analytics approach allowed exploring learning process patterns that associate with better or worse learning outcomes, subsequently being used as guidelines to improve teaching.

“The approach allowed exploring learning process patterns that associate themselves with better/worse learning outcomes, subsequently being used as guidelines to improve teaching (Sedrakyan, De Weerdt & Snoeck, 2016;Sedrakyan, 2016).”

• cites Sedrakyan, De Weerdt & Snoeck, 2016Not checked yet

  Not checked in the free run.

• cites Sedrakyan, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Adaptation profiles can use learner behavior following viewing and monitoring activities to show approximations of adaptation effort.

“These profiles can use learner behavior following viewing and monitoring activities to show approximations of adaptation effort.”

May combine two claims.

Depending on the types of goals learners possess, learning outcomes will target different levels of knowledge, skills, competences, or simply task completion, which also determines how learners engage with the regulated learning process.

“Depending on the types of goals learners possess, learning outcomes will target different levels of knowledge, skills, competences, or simply task completion, which also determines how learners engage with the regulated learning process (Winne, Hadwin, & Perry, 2013).”

• cites Winne, Hadwin, & Perry, 2013Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

A planning profile includes activities such as planning the sub-goals to reach, selection of learning strategy (action plan), the materials and resources to use, and allocation of time.

“In general, a planning profile includes activities such as a) planning the sub-goals to reach, b) selection of learning strategy (action plan), c) the materials and resources to use, and c) allocation of time (Pintrich, 2000).”

• cites Pintrich, 2000 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A co-adaptation profile will utilize the concept of peer-oriented feedback both for individual and group learning plans.

“A co-adaptation profile will utilize the concept of peer-oriented feedback (e.g., “Could you give feedback on your peer task results?”) both for individual and group learning plans.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A self-monitoring profile can inform a learner about his or her progress with respect to an action plan and self-defined goals.

“A self-monitoring profile can inform a learner about his or her progress with respect to an action plan and self-defined goals.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Planning profiles will be used both to guide learners and inform teachers about the coherence and alignment between learners' and teacher's specified goals and action plans.

“These planning profiles will be used both to guide learners and inform teachers about the coherence and alignment between learners' and teacher's specified goals and action plans.”

Probably not a substantial claim: may not pass S4 (based).

A learner's attempt in combination with outcomes, utilized time and resources will allow measuring if the self-regulation effort was successful or sufficient.

“whereas a learner's attempt in combination with outcomes, utilized time and resources (as well as feedback received on earlier attempt) will allow measuring if the self-regulation effort was successful or sufficient.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Absence of another trial might suggest a need for behavioral feedback to engage a learner in a regulatory process.

“Absence of another trial might for instance suggest a need for behavioral feedback to engage a learner in a regulatory process,”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

By creating visible artefacts of varying sub-goals related to mastering new knowledge or skills, we can make them observable in the learning process for both instructor and learner.

“That is, by creating visible artefacts of varying sub-goals related to mastering the new knowledge or skills and/or competence, we can make them observable in the learning process for both the instructor and learner.”

Classified as probably not a substantial claim.

Task-specific subgoals (TSSG) include steps that are specific, measurable, action-oriented, realistic, and temporal.

“Task-specific subgoals (TSSG) include steps that are a) specific, b) measurable, c) action-oriented, d) realistic, and e) temporal (Winne & Hadwin, 1998).”

• cites Winne & Hadwin, 1998Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Technology Acceptance Model suggests ease of use and perceived usefulness as important factors contributing to the user acceptance (i.e., actual utility) of LAD feedback.

“Previous studies have identified important variables dealing with user acceptance for computer-assisted learning environments, such as the Technology Acceptance Model (Davis, 1989), which suggests ease of use and perceived usefulness as important factors contributing to the user acceptance (i.e., actual utility) of LAD feedback.”

• cites Davis, 1989 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Backward design uses goal setting as the focal point of lesson design.

“For example, “backward design,” which is a well-known instructional design model, uses goal setting as the focal point of lesson design (Wiggins & McTighe, 1998).”

• cites Wiggins & McTighe, 1998Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Planning profiles can provide additional information about the overall level of preparedness, preferences and difficulties of learners.

“The planning profiles can also provide additional information about the overall level of preparedness, preferences and difficulties of learners.”

Probably not a substantial claim: may not pass S4 (based).

An increased or decreased effort in combination with achievements can be indicative of whether activities had an impact on achievement.

“For instance, an increased or decreased effort in combination with achievements (e.g., score) can be indicative of whether activities, such as attending classes, use of a learning resource, task completion, or received feedback had an impact on achievement.”

Probably not a substantial claim: may not pass S2 (checkable).

Based on learners' goals, LADs should be able to propose personalized learning trajectories (action plans), also taking into consideration the level of preparedness and dependencies between learning (sub-)goals.

“We posit that based on these goals, LADs should be able to propose personalized learning trajectories (action plans), also taking into consideration the level of preparedness and dependencies between learning (sub-)goals.”

• AssumptionRests on an assumption: The learning science concepts reviewed (regulation of learning, feedback typology, goal orientation, efficiency and effectiveness) are the relevant and sufficient concepts for designing LAD feedback.

Classified as probably not a substantial claim.

The conceptual model can serve as a general framework for designing and building information systems as a basis for educational dashboard feedback.

“This model can serve as a general framework for designing and building information systems as a basis for educational dashboard feedback.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Detection of inefficient processes can be useful in determining the timeliness (receive time) of the feedback.

“we posit that the detection of inefficient processes can be useful in determining the timeliness (receive time) of the feedback.”

Probably not a substantial claim: may not pass S4 (based).

Adaptation profiles can inform learners about the level of effort put into learning regulation and the needs for adaptation.

“Adaptation profiles can inform learners about the level of effort put into learning regulation and the needs for adaptation (e.g., how learners perform with learning challenges, the resources they use, how much time they spend, and whether they need additional feedback).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Goal-directed behavior that results from self-goal setting is empowering and proactive, as students take responsibility and ownership.

“Goal-directed behavior that results from self-goal setting is empowering and proactive, as students take responsibility and ownership (Elliot & Fryer, 2008).”

• cites Elliot & Fryer, 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

When using backward design, instructors identify learning goals first, determine acceptable evidence, and plan learning experiences and instruction to achieve those goals.

“When using backward design, instructors identify learning goals for the course first, considering what they want students to know and be able to do when they finish the course, and then determine acceptable evidence regarding whether those goals are met and plan learning experiences and instruction to achieve those learning goals (Wiggins & McTighe, 1998).”

• cites Wiggins & McTighe, 1998Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

A self-adaptation profile will inform a learner about how successful he or she was in addressing a challenge and what actions he or she can take to address a detected challenge.

“A self-adaptation profile will inform a learner about how successful he or she was in addressing a challenge and what actions he or she can take to address a detected challenge.”

May combine two claims.

By complementing the idea of feedback with the concepts of effectiveness and efficiency, it is possible to assess learning progress with respect to potential intervention needs and to determine optimal receive time of feedback by discovering inefficient or ineffective processes during learning.

“In summary, by complementing the idea of feedback with the concepts of effectiveness and efficiency, it is possible to assess a learning progress with respect to potential intervention needs and, more specifically, with respect to determining optimal”

• AssumptionRests on an assumption: The learning science concepts reviewed (regulation of learning, feedback typology, goal orientation, efficiency and effectiveness) are the relevant and sufficient concepts for designing LAD feedback.

May combine two claims.

Outputs of TSSGs and engagement patterns will allow distinguishing the needs for the type of feedback (cognitive or behavioral) and expanding dashboard performance visualizations with textual feedback to learners and teachers.

“In summary, while the outputs of TSSGs and engagement patterns will allow distinguishing the needs for the type of feedback (cognitive or behavioral), these also allow expanding dashboard performance visualizations with textual feedback to learners and teachers.”

Classified as probably not a substantial claim.

The model allows for the generation of process-oriented feedback to different users related to different learning goals.

“The model allows for the generation of process-oriented feedback to different users (e.g., teacher and learners) related to different learning goals.”

Classified as probably not a substantial claim.

In the planning phase using backward design, the task-specific sub-goals necessary to obtain a knowledge or skill are linked with mastering the proposed learning material.

“In the planning phase, using backward design, the task specific sub-goals necessary to obtain a knowledge or skill are linked with mastering the proposed learning material.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

If the expected utility of a specific resource within an agreed time does not lead to achievement of the expected learning goal, it might suggest that the resource contains a certain level of difficulty for that cluster, and vice versa.

“For example, if the expected utility of a specific resource within a meaningful (agreed) time does not lead to the achievement of the expected learning goal, either for a specific learner or group of learners (depending on the observation target), it might suggest that the resource for a specific goal contains a certain level of “difficulty” for that cluster, and vice versa.”

Classified as probably not a substantial claim. May combine two claims.

According to goal-setting theory, goals should be short term rather than long term and challenging, and should be Specific, Measurable, Achievable, Relevant, and Time-framed (SMART).

“According to goal-setting theory, goals should be short term rather than long term and challenging (Latham & Locke, 2002) and “Specific, Measurable, Achievable, Relevant, Time-framed” (SMART; Drucker, 1954).”

• cites Latham & Locke, 2002Not checked yet

  Not checked in the free run.

• cites Drucker, 1954Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The role of the monitoring profile in the context of dashboards is to detect and suggest further adaptation needs for learners and inform teachers about difficulties if the expected performance level is not being achieved within an agreed action plan and timeframe.

“In general, the role of the monitoring profile in the context of dashboards is to detect and suggest further adaptation needs for learners and inform teachers about difficulties if the expected performance level is not being achieved within an agreed action plan and timeframe (see the case example for further details).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Learning resources can serve as artefacts of instructor expectations or planning.

“These resources can serve as artefacts of instructor expectations or planning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A co-monitoring profile can provide information about how well a learner performs with respect to his or her peers if the goal orientation is set to performance.

“A comonitoring profile can also provide information about how well a learner performs with respect to his or her peers if the goal orientation is set to “performance”.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The feedback idea presented in this example was piloted by the analysis of data from the authors' experiment conducted at LeaForum.

“The feedback idea presented in this example was piloted by the analysis of data from the authors' experiment conducted at LeaForum (http://www.oulu.fi/leaf-eng/).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Group standards can be used to provide group-oriented monitoring feedback.

“Group standards can be used to provide group-oriented monitoring feedback (see the discussed theoretical concept in section 3.2.1,3.2.2 and the measurment and presentation layer concepts in sections 4.1 and 4.3).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The proposed design model allows extending dashboard feedback with textual explanations in the form of cognitive and behavioral feedback, which makes learning-process data visualizations easier to interpret for an end user.

“Ultimately, the proposed design model allows extending dashboard feedback with textual explanations in the form of cognitive and behavioral feedback which makes learning-process data visualizations easier to interpret for an end user.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The case example makes use of visualization techniques based on empirical evidence from earlier research.

“as the case example makes use of visualization techniques based on empirical evidence from earlier research, investigating other visualization techniques and testing which techniques can work better in supporting the feedback ideas presented in this paper can yet be another possible future research direction.”

Classified as probably not a substantial claim.

Several researchers argue that showing an aggregated analysis with respect to peer performance is a good approach.

“Several researchers argue that showing an aggregated analysis with respect to peer performance is a good approach (Lonn et al., 2015).”

• cites Lonn et al., 2015 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Such feedback can be constructed based on learner profiles of regulation process phases spanning the planning, monitoring, and adapting activities, thus allowing detecting inefficient processes in learning.

“Such feedback can be constructed based on learner profiles of regulation process phases spanning the planning, monitoring, and adapting activities thus allowing detecting inefficient processes in learning.”

Probably not a substantial claim: may not pass S2 (checkable).

Backward instructional design provides richer opportunities for tracking learning processes due to the use of goal setting as the focal point of lesson design and TSSGs, which combine steps that are measurable, action-oriented, and temporal.

“Further,”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Visual inspection can easily reveal if the interactions are balanced within a group or concentrated among particular members.

“For instance, visual inspection can easily reveal if the interactions are balanced within a group or concentrated among particular members.”