Gamifying education: what is known, what is believed and what remains uncertain: a critical review

Probably not a substantial claim: may not pass S3 (load-bearing).

Hakulinen et al. (2015) present convincing evidence that points, badges and leaderboard incorporated in online Java exercises increase the use of an open learning environment.

“For example, (Hakulinen et al., 2015) present convincing evidence that points, badges and leaderboard incorporated in online Java exercises increases the use of an open learning environment.”

• cites Hakulinen et al., 2015 sourcePartly supported

  The Results report that students offered badges spent significantly more time in TRAKLA2 and had more exercise sessions than controls, supporting increased use of the online learning environment. But the study tested badges, not a combined points–badges–leaderboard intervention: exercise points applied to both groups, and Future Work says the social aspect was missing. Materials and Procedure describes algorithm-simulation exercises, not specifically online Java exercises. (LLM, conf=0.90)

  Experimental — accuracy not yet measured

May combine two claims.

Two reviewed papers report positive outcomes for using single game elements, but one is for badges and the other one for leaderboards.

“For example, two papers (Hakulinen et al., 2015) and (Landers & Landers, 2015) report positive outcomes for using single game elements, but one is for badges and the other one for leaderboards.”

• cites Hakulinen et al., 2015 sourceSupported
• cites Landers & Landers, 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The collection of reviewed papers covers a wide range of academic subjects (32) organized in six categories.

“The collection of papers covers a wide range of academic subjects (32) organized in six categories (see Table 2).”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

The total number of papers included in the previous review covering January 2010-June 2014 was 34.

“For comparison, the total number of papers included in the previous review covering the period January 2010 -June 2014 was 34.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

The end result was a list of 11 theoretical papers appearing in the searched databases between June 30, 2014 and December 31, 2015.

“The end result was a list of 11 theoretical papers appearing in the searched databases between June 30, 2014 and December 31, 2015.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

May combine two claims.

The vast majority of the empirical works (25 studies) report inconclusive outcomes, meaning there is no basis for confidence in the reported results.

“A surprising fact is that the vast majority of the empirical works (25 studies) report inconclusive outcomes, which means that there is no basis for confidence in the reported results.”

May combine two claims.

In the reviewed collection, 11 papers report studies of the effect of a single game element, 8 papers study gamified systems using 2 game elements, 16 papers study gamified systems with 3 game elements, and the remaining 16 papers report results of gamifying systems by incorporating more than three elements.

“In the reviewed collection, 11 papers report studies of the effect of a single game element, 8 papers study gamified systems using 2 game elements, 16 papers study gamified systems with 3 game elements, while the remaining 16 papers report results of gamifying systems by incorporating more than three elements (see Table 4).”

May combine two claims.

The bulk of gamification studies in the survey period were conducted at university level (44 papers), with less attention to K-12 education (7 papers).

“Considering the educational level, the bulk of gamification studies in the survey period were conducted at university level (44 papers), with less attention to K-12 education (7 papers).”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

May combine two claims.

Among the 15 conclusive ABC studies, the findings on the benefits of gamification are mixed: 12 present evidence for positive effects of gamification in educational settings and 3 present evidence for negative effects.

“In those 15 papers, the findings related to the benefits of gamification are mixed: 12 studies present evidence for positive effects of gamification in educational settings, while 3 present evidence for negative effects.”

May combine two claims.

Although the majority of empirical works still examine the impact of gamification on students' engagement, performance, participation or retention, they are widening and deepening the focus of their studies.

“Although the majority of empirical works still examine the impact of the gamification on students ' engagement, performance, participation or retention, they are widening and deepening the focus of their studies.”

May combine two claims.

Although the connection between motivation and gamification design is demonstrated by a number of the reviewed studies, they do not add persuasive evidence confirming the effect of gamification as a motivational tool.

“Although the connection between motivation and gamification design is demonstrated by a number of the reviewed studies, they do not add persuasive evidence confirming the effect of gamification as a motivational tool.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Only 51 studies met the criteria and are reviewed in this article.

“The result was a list of 51 empirical research papers.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

The final number of selected papers (empirical and theoretical) amounted to 63 in total.

“Thus the final number of selected papers (empirical and theoretical) amounted to 63 in total.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

Another part of the papers addresses activities with indirect effect on learning, such as engaging students in more regular interactions with the learning environment (11).

“Another part of the papers addresses activities with indirect effect on learning, such as engaging students in more regular interactions with the learning environment (11).”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

May combine two claims.

Although online learning presumes a higher concentration of studies on gamified online learning, the findings indicate the opposite.

“Although this presumes a higher concentration of studies on gamified online learning our findings indicate the opposite.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

May combine two claims.

The empirical research covers studies on different configurations of game elements, used to gamify different activities and resulting in different outcomes, without any identifiable pattern of distribution.

“It covers studies on different configurations of game elements, used to gamify different activities and resulting in different outcomes, without any identifiable pattern of distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The gamification terminology is still unsettled and various variations of the introduced terms exist.

“However, we note that the gamification terminology is still unsettled and various variations of the introduced above terms exist.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

The search yielded a total of 4998 results.

“This search yielded a total of 4998 results depicted in Table 1.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

May combine two claims.

A significant part of the empirical research was reporting success stories and possibly contributing to inflated expectations.

“In parallel, a significant part of the empirical research was nonetheless reporting success stories and possibly contributing to the ' inflated expectations ' .”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

The papers listed in the platform-centric category do not cover all gamified platforms proposed in the reviewed papers.

“The papers listed in the platform-centric category do not cover all gamified platforms proposed in the reviewed papers.”

Classified as probably not a substantial claim. May combine two claims.

Studies on how distinct categories of learners are affected by gamification, what to measure as an outcome, and how to add a gamified layer to a core activity are emerging.

“Studies on how distinct categories of learners are affected by gamification, what to measure as an outcome, and how to add a gamified layer to a core activity are also emerging.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The question shared between most of the reviewed papers has shifted from 'Is gamification effective?' to a more extended version combined with the questions 'for what?' or 'to whom?'.

“While in our first review the question shared between most of the papers was ' Is gamification effective? ' , now it appears in a more extended version, in combination with the questions ' for what? ' or ' to whom? ' .”

May combine two claims.

The reviewed empirical studies tended to use surveys and quasi-experimental designs, while randomized controlled trials were less common.

“The examination of the selected papers indicated that the empirical studies tended to use surveys and quasi-experimental designs, while the randomized controlled trials were less common.”

The quote only partly states this claim.

The review categorized a set of studies as inconclusive due to methodological limitations such as small population size, short study period, and unclear measurements.

“Table 12 Inconclusive studies with reasons for this categorization”

May combine two claims.

The educational benefits of gamification in terms of increasing student motivation or linking this motivation to learning outcomes are still not well understood.

“As a result, the educational benefits of gamification in terms of increasing student motivation or linking this motivation to learning outcomes are still not well understood.”

May combine two claims.

The majority of the reviewed studies analyze specific educational effects of gamification (on learning, attainment, participation), but their focus is often aside from motivation.

“While the majority of the reviewed studies do analyze specific educational effects of gamification (on learning, attainment, participation), their focus is often aside from motivation.”

May combine two claims.

Research remains inconclusive on the assumptions that gamification is motivating, engaging, and can improve attendance and participation.

“However, research remains inconclusive on these assumptions.”

May combine two claims.

The reported outcome of most reviewed studies often concludes that the gamification produced the pursued learning gains and that the users appreciated the added gamification features.

“The reported outcome often concludes that the gamification produced the pursued learning gains and that the users appreciated the added gamification features.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The majority of the studies target college students.

“Moreover, the majority of the studies target college students.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A number of gamification approaches, driven by specific objectives, have been applied to support learning and related activities in a variety of educational contexts.

“A number of gamification approaches, driven by specific objectives, have been applied to support learning and related activities in a variety of educational contexts.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Several assumptions underlie the usefulness of gamification in educational context, such as gamification is motivating, gamification is engaging, gamification can improve attendance and participation.

“There are several assumptions underlying the usefulness of gamification in educational context, such as gamification is motivating, gamification is engaging, gamification can improve attendance and participation.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Gamification is not just a technology but also a methodology which some organizations adopt as a way to increase motivation.

“We maintain the view that gamification is not just a technology but also a methodology which some organizations adopt as way to increase motivation.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Studies involving different demographic groups are beneficial, as results from one demographic group cannot necessarily be generalized to another.

“Studies involving different demographic groups however are beneficial, as we cannot necessarily generalize the results of a study conducted with one demographic group to another demographic group.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The used game elements, mechanics and dynamics are inherently related to the success of a gamification application.

“We also included the used game elements, mechanics and dynamics since they are inherently related to the success of a gamification application.”

Classified as probably not a substantial claim.

The review was restricted to papers appearing in the searched databases between June 30, 2014 and December 31, 2015.

“The review was restricted to papers appearing in the searched databases between June 30, 2014 and December 31, 2015.”

• AssumptionRests on an assumption: The selected databases and search terms capture the relevant empirical literature on educational gamification.
• AssumptionRests on an assumption: The 51 empirical papers reviewed are representative of the empirical research on educational gamification in the survey period.

Probably not a substantial claim: may not pass S3 (load-bearing).

Pedro et al. (2015b) reported that the game mechanics implemented in a virtual learning environment did not have any effect on motivation and performance of the female students.

“For example, (Pedro et al., 2015b) reported that the game mechanics implemented in a virtual learning environment did not have any effect on motivation and performance of the female students.”

• cites Pedro et al., 2015bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Online learning normally requires stronger motivation, which makes it a somewhat more promising field for applying gamification.

“Online learning normally requires stronger motivation, which makes it a somewhat more promising field for applying gamification.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Motivation as a psychological process that gives behavior purpose and direction is contextual.

“Motivation as a psychological process that gives behavior purpose and direction is contextual.”

Probably not a substantial claim: may not pass S2 (checkable).

More substantial empirical research is needed to determine whether both extrinsic and intrinsic motivation of learners can be actually influenced by gamification.

“Although most of the reviewed 34 papers have been reporting promising results, the review concluded that more substantial empirical research is needed to determine whether both extrinsic and intrinsic motivation of the learners can be actually influenced by gamification.”

• cites Dicheva et al. 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The intrinsic appeal of games is due to their ability to satisfy the basic psychological needs for competence, autonomy, and relatedness.

“More specifically, (Ryan et al., 2006) argue that the intrinsic appeal of games is due to their ability to satisfy the basic psychological needs for competence, autonomy, and relatedness.”

• cites Ryan et al., 2006 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Individuals are motivated in multiple ways, and their motivation varies according to the situation or context of the task.

“Not only are individuals motivated in multiple ways, but also their motivation varies according to the situation or context of the task.”

May combine two claims.

A gamified laboratory simulation can increase both learning outcomes and motivation levels when compared with traditional teaching.

“The results show that a gamified laboratory simulation can increase both learning outcomes and motivation levels when compared with traditional teaching.”

• cites Bonde et al. (2014) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Gamification is not a purely marketing trend but a behavioral/affective design trend that can be applied to different areas, including education.

“In this aspect, gamification is not a purely marketing trend but a behavioral/affective design trend that can be applied to different areas, including education.”

May combine two claims.

Motivation is not a unitary phenomenon; different people may have different types and amounts of motivation, which can be shaped by the activity they are undertaking.

“While gamification strives at its core to increase motivation, yet motivation is not a unitary phenomenon - different people may have different types and amounts of motivation, which can be shaped by the activity they are undertaking (Gooch et al., 2016).”

• cites Gooch et al., 2016 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Women in the female-dominated leaderboard condition demonstrated stronger academic identification than those in the control and male-dominated leaderboard conditions.

“The results of the study found that women in the female-dominated leaderboard condition demonstrated stronger academic identification than those in the control and maledominated leaderboard conditions.”

• cites Christy & Fox, 2014 sourceNot checked — no full text of the source

May combine two claims.

The 12 studies reporting encouraging outcomes cover a range of specific combinations of game elements, specific activities and outcomes, and thus do not support practical generalization.

“Whilst 12 studies report encouraging outcomes, they cover a range of specific combinations of game elements, specific activities and outcomes and thus do not support practical generalization.”

Probably not a substantial claim: may not pass S4 (based).

The verification of the validity of reported results requires availability of relevant statistical information about the studies.

“In addition to appropriate measures, the verification of the validity of reported results requires availability of relevant statistical information about the studies.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Gartner Hype Cycle points out that an emerging technology first climbs the 'peak of inflated expectations', followed by a strong fall into the 'trough of disillusionment', before reaching the 'slop of enlightenment', which marks the stage where its benefits and limitations are understood and demonstrated.

“which points out that an emerging technology first climbs the ' peak of inflated expectations ' followed by a subsequent strong fall down into the ' trough of disillusionment ' , before reaching the ' slop of enlightenment ' , which marks the stage where its benefits and limitations are understood and demonstrated.”

• cites Gartner, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Improving our understanding of motivational aspects of gamification will enable us to predict its effect on the related concepts.

“Improving our understanding of motivational aspects of gamification will enable us to predict its effect on the related concepts.”

• AssumptionRests on an assumption: Motivation is associated with a number of learning related concepts such as engagement, effort, goals, focus of attention, self-efficacy, confidence, achievement, interest, etc.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This suggests a user-centered approach in the design of gamified systems, characterized by a focus on the needs and desires of learners.

“This suggests a user-centered approach in the design of gamified systems, characterized by a focus on the needs and desires of learners.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Gamification increased communication, participation and punctuality.

“Increased communication, participation, punctuality”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Without a theoretical framework backing the design of the studies and the interpretation of their results, it is problematic to select an appropriate gamification structure or to differentiate which of the employed game mechanisms and principles were essential for arriving at successful outcomes.

“Without a theoretical framework backing the design of the studies and the interpretation of their results, it is problematic to select an appropriate gamification structure or to differentiate which of the employed game mechanisms and principles were essential for arriving at successful outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Gamification increased motivation and learning.

“Increased motivation and learning”

Probably not a substantial claim: may not pass S3 (load-bearing).

In several cases the review's judgment of inconclusiveness simply conveys the paper's conclusion where the authors themselves acknowledge that the results of the study should be interpreted with caution.

“In several cases our judgment simply conveys the paper ' s conclusion where the authors themselves acknowledge that the results of the study should be interpreted with caution.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification increased intention to use the PLE.

“Increased intention to use the PLE”

Classified as probably not a substantial claim.

Merely making an activity seem like a game impacts learners' engagement.

“This study reveals an interesting psychological perspective of gamification in educational environments: merely making an activity seem like a game impacts learners ' engagement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most emerging technologies and the accompanying research go through an initial period of hype as described by the Gartner's Hype Cycle, before evolving for a second period of measured popularity, in which it attains maturity and meets the expectations.

“Most emerging technologies and the accompanying research go through an initial period of hype as described by the Gartner ' s Hype Cycle, before evolving for a second period of measured popularity, in which it attains maturity and meets the expectations (Naik, 2015).”

• cites Naik, 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The theory of gamified learning proposed by Landers et al. (2015) provides two specific causal pathways by which gamification can affect learning and a framework for testing these pathways.

“The theory of gamified learning proposed by (Landers et al., 2015) provides two specific causal pathways by which gamification can affect learning and a framework for testing these pathways.”

• cites Landers et al., 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There was mixed acceptance of game elements.

“Mixed acceptance of game elements”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Gamification increased knowledge retention and reduced attrition.

“Increased knowledge retention, reduce attrition”

• cites (Nevin et al., 2014)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Gamification had a positive effect on student collaboration.

“Positive effect on student collaboration”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Gamification reduced undesirable behaviors in the VLE.

“reduced undesirable behaviors in VLE”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Gamification improved retention period and learning performance.

“Improved retention period, and learning performance”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Gamification increased playfulness and engagement in learning.

“Increased playfulness and engagement in learning”

• cites (Perry, 2015)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

The availability of successful gamified platforms will help widen the scope of gamified educational activities and create a ground for broadening experimental studies towards developing evidence based practices.

“The availability of successful gamified platforms will help widen the scope of gamified educational activities and create a ground for broadening experimental studies towards developing evidence based practices.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Gamification motivated continuous learning for the module.

“Motivated continuous learning for module”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There is a relation between badge interaction and engagement.

“Relation between badge interaction and engagement”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the theory of gamified learning, in the first process, learning-related behavior moderates the relationship between instructional quality and learning; in the second, this behavior mediates the relationship between game elements and learning.

“In the first one, this behavior moderates the relationship between instructional quality and learning. In the second, this behavior mediates the relationship between game elements and learning.”

• cites Landers et al., 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification improved learning performance.

“Improved learning performance”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Possible reasons for the inconclusive evidence are the hype to publish on gamification and the addressing of an overly broad research question based on limited supportive evidence.

“Possible reasons for this are from one side the hype to publish on gamification and from another, the addressing of an overly broad research question based on limited supportive evidence.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The process of integrating game design principles within varying educational experiences appears challenging and there are currently no practical guidelines for how to do so in a coherent and efficient manner.

“However, the process of integrating game design principles within varying educational experiences appears challenging and there are currently no practical guidelines for how to do so in a coherent and efficient manner.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Gamification increased engagement and fostered learning.

“Increased engagement, foster learning”

Classified as probably not a substantial claim.

For gamification to be effective, it must successfully alter an intermediary learner behavior or learner attitude.

“For gamification to be effective, it must successfully alter an intermediary learner behavior or learner attitude.”

• cites Landers et al., 2015Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Gamification improved participation and quality of online discussion.

“Improved participation and quality of online discussion”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification increased usage of the open learning environment.

“Increased usage of the open learning environment”

May combine two claims.

An emerging shared message particularly relevant to instructional designers recognizes that what one learner values, another may not, and what one learner believes is achievable, another may not.

“an emerging shared message particularly relevant to instructional designers recognizes that what one learner values, another may not, what one learner believes is achievable, another may not.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification increased intrinsic motivation.

“Increased intrinsic motivation”

Probably not a substantial claim: may not pass S3 (load-bearing).

Leaderboards create stereotype threat.

“Leaderboards create stereotype threat”

• cites (Christy & Fox, 2014)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Reported positive effects of gamification based on a two-week study could be attributed to the novelty effect of the used tool or approach rather than to the added gamification features.

“For example, reported positive effects of gamification based on a two-week study could be attributed to the ' novelty effect ' of the used tool or approach rather than to the added gamification features.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Gamification increased competence need (negative) and performance (positive).

“Increased competence need (negative) and performance (positive)”

Probably not a substantial claim: may not pass S2 (checkable).

The observed emphasis on points, badges, and leaderboards is too narrow to address the relevant motivational factors.

“The observed emphasis on points, badges, and leaderboards is too narrow to address the relevant motivational factors.”

May combine two claims.

There is still insufficient evidence that gamification produces reliable, valid and long-lasting educational outcomes, or that it does so better than traditional educational models.

“While it seems apparent that gamification has the potential to create enhanced learning environments, there is still insufficient evidence that it (1) produces reliable, valid and long-lasting educational outcomes, or (2) does so better than traditional educational models.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The review's judgment in studying inconclusiveness can be viewed as rather subjective.

“We note that our judgment in studying inconclusiveness can be viewed as rather subjective.”

Classified as probably not a substantial claim. May combine two claims.

The review's decision on the validity of the gamification studies was guided by the sample size, the number of study groups, the length of the study, how the data was collected, how the variables were controlled, how and by what statistical procedures the data was analyzed, how well the conclusions are supported by the data, and whether the study gives enough information to convince the reader of

“Our decision on the validity of the gamification studies was guided by the following factors: the sample size, the number of study groups, the length of the study, how the data was collected, how the variables were controlled, how and by what statistical procedures the data was analyzed, how well the conclusions are supported by the data, and does the study give enough information to convince the”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The rise and fall of expectations for applying gamification in educational contexts is nothing out of the ordinary.

“The rise and fall of expectations for applying gamification in educational contexts is nothing out of the ordinary.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification improved time on task.

“Improved time on task”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a positive correlation between XPs, badges and learning.

“Correlation between XPs, badges and learning - positive”

• cites (Paiva et al., 2015)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification increased disposition to the experience flow.

“Increased disposition to the experience flow”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Gamification does not improve learning.

“Does not improve learning”

• cites Long & Aleven, 2014 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There is a relation between learner gamification typology and learner interactive behavior.

“Relation between learner gamification typology and learner interactive”

May combine two claims.

Lieberoth (2015) found that engagement and enjoyment increased significantly due to the psychological effects of framing the task as a game, and no actual increased interest or enjoyment was measured by adding actual game mechanics to the task when it was already framed as a game.

“The outcomes indicate that engagement and enjoyment increased significantly due to the psychological effects of framing the task as a game. Furthermore, no actual increased interest or enjoyment was measured by adding actual game mechanics to the task, when it was already framed as a game.”

• cites Lieberoth (2015) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

Gamification improved motivation.

“Improved motivation”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Tomaselli et al. (2015) found that although there is support for the importance of competition against peers, the challenge of overcoming the game's obstacles and mastering them is what matters the most to players, regardless of the type of the game.

“The results show that although there is support for the importance of competition against peers (contrary to the current prevailing understanding), the challenge of overcoming the game ' s obstacles and mastering them is what matters the most to players, regardless of the type of the game.”

• cites Tomaselli et al., 2015Not checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Gamification fostered learning activities.

“Fostered learning activities”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There is a relation between gaming personality and game dynamics.

“Relation between gaming personality and game dynamics”