CoChat
Sign Up
Cited by 584Educational Games and GamificationCognitive Abilities and TestingCognitive Functions and Memory

Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy

Open access
Lumsden, Jim, Edwards, Elizabeth A, Lawrence, Natalia S, Coyle, David, Munafò, Marcus R·2016·JMIR Serious Games
DOI 10.2196/games.5888PMC PMC4967181
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Cognitive tasks are typically viewed as effortful, frustrating, and repetitive, which often leads to participant disengagement. This, in turn, may negatively impact data quality and/or reduce intervention effects. However, gamification may provide a possible solution. If game design features can be incorporated into cognitive tasks without undermining their scientific value, then data quality, intervention effects, and participant engagement may be improved. OBJECTIVES: This systematic review aims to explore and evaluate the ways in which gamification has already been used for cognitive training and assessment purposes. We hope to answer 3 questions: (1) Why have researchers opted to use gamification? (2) What domains has gamification been applied in? (3) How successful has gamification been in cognitive research thus far? METHODS: We systematically searched several Web-based databases, searching the titles, abstracts, and keywords of database entries using the search strategy (gamif* OR game OR games) AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*). Searches included papers published in English between January 2007 and October 2015. RESULTS: Our review identified 33 relevant studies, covering 31 gamified cognitive tasks used across a range of disorders and cognitive domains. We identified 7 reasons for researchers opting to gamify their cognitive training and testing. We found that working memory and general executive functions were common targets for both gamified assessment and training. Gamified tests were typically validated successfully, although mixed-domain measurement was a problem. Gamified training appears to be highly engaging and does boost participant motivation, but mixed effects of gamification on task performance were reported. CONCLUSIONS: Heterogeneous study designs and typically small sample sizes highlight the need for further research in both gamified training and testing. Nevertheless, careful application of gamification can provide a way to develop engaging and yet scientifically valid cognitive assessments, and it is likely worthwhile to continue to develop gamified cognitive tasks in the future.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review identified 33 studies covering 31 gamified cognitive tasks and found 7 reasons researchers gamify cognitive assessment or training. Gamified tasks generally improved motivation and engagement, but evidence for improved data quality was lacking, and effects on task performance were mixed.

Key findings

  • The review included 33 relevant studies and 31 gamified cognitive tasks; working memory and general executive functions were common targets for both gamified assessment and training.
  • Gamified assessments were typically validated successfully, with sample intertask correlations of 0.45–0.60 in some simpler validation studies, but mixed-domain measurement was a problem.
  • Gamified training appeared highly engaging and boosted participant motivation, but mixed effects on task performance were reported; 21 of 33 studies compared a gamelike task directly against a nongamified counterpart.
  • The authors found no evidence that gamified tests improve data quality and some indications they may worsen it, while positive training findings were tentative due to methodological problems.

Methods

  • Systematic search of PsycInfo, Medline, ETHOS, EMBASE, PubMed, IBSS, Francis, Web of Science, and Scopus using `(gamif* OR game OR games) AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*)`, limited to English papers from January 2007 to October 2015.
  • Inclusion required primary empirical research on a novel gamelike task designed to assess or train cognition and either validated or piloted; non–peer-reviewed papers, education/health-promotion/rehabilitation gamification, and non-gamelike VR/3D environments were excluded.
  • Papers were screened by one reviewer (JL) in three stages, with a second reviewer (EE) rescreening 20% from the title stage onward; inter-rater agreement was moderate (k=.526, 95% CI 0.416–0.633, P<.001).

Limitations

  • The paper states that the review required a narrow scope focused on cognitive training and testing, which led to some papers being excluded on the subjective basis of not being “cognitive research,” and that many included studies were preliminary, making findings tentative.
  • The paper states that sample sizes were small in nearly all reviewed studies and only 5 of 33 papers described a power calculation.
  • Evident from the text: many training games used only a gamified task, so the effect of gamification could not be dissociated from the intervention effect, and validation standards were heterogeneous.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/games.5888

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

3 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

0 of 82 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Gamified tests were typically validated successfully.

    “Gamified tests were typically validated successfully”
    Quote in the paperSubstantial claim
  • Findingown data

    Working memory and general executive functions were common targets for both gamified assessment and training.

    “We found that working memory and general executive functions were common targets for both gamified assessment and training.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    Mixed-domain measurement was a problem for gamified tests.

    “although mixed-domain measurement was a problem”
    Quote in the paperNot a substantial claim
  • Findingown data

    Mixed effects of gamification on task performance were reported.

    “mixed effects of gamification on task performance were reported”
    Quote in the paperSubstantial claim
  • Findingown data

    The review identified 7 reasons for researchers opting to gamify their cognitive training and testing.

    “We identified 7 reasons for researchers opting to gamify their cognitive training and testing.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 44
  • Classified as probably not a substantial claim. 19

  • May combine two claims.

    Findingnone given

    Gamified training appears to be highly engaging and boosts participant motivation.

    “Gamified training appears to be highly engaging and does boost participant motivation”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The review identified 33 relevant studies, covering 31 gamified cognitive tasks used across a range of disorders and cognitive domains.

    “Our review identified 33 relevant studies, covering 31 gamified cognitive tasks used across a range of disorders and cognitive domains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Sixteen studies explicitly used gamified tasks to measure or train cognition in a more motivating manner, and the majority (10 of 16) of these studies were assessment studies.

    “we found 16 studies that explicitly used gamified tasks to measure or train cognition in a more motivating manner, and the majority (10 of 16) of these studies were assessment studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Included studies had heterogeneous study designs and typically small sample sizes.

    “Heterogeneous study designs and typically small sample sizes”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Many of these games were used in an unobserved, nonlaboratory setting, and the tasks made use of motivational features that made training and testing intrinsically more appealing and less of a burden to perform regularly.

    “Many of these games were used in an unobserved, nonlaboratory setting; therefore, the tasks made use of motivational features that made the training and testing intrinsically more appealing and less of a burden to perform on a regular basis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Watermons was designed using Self Determination Theory to maximize participant intrinsic motivation and make any motivational effects of game mechanics as apparent as possible.

    “One game, Watermons [38], was designed, using Self Determination Theory [50,51], to maximize participant intrinsic motivation and make any motivational effects of game mechanics as apparent as possible.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [38] source

      Not checked in the free run.

    • cites [50] source

      Not checked in the free run.

    • cites [51] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Working memory was a popular target for training, closely followed by executive function and inhibitory control training.

    “Again, working memory was a popular target, closely followed by EF and inhibitory control training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Many games tested several cognitive domains and/or general executive function, highlighting the difficulty of examining gamification effects on specific cognitive domains.

    “Many games tested several cognitive domains and/or general EF, highlighting the difficulty of examining gamification effects on specific cognitive domains (and therefore comparing performance to standard cognitive tasks).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The numbers of studies investigating cognitive testing (17) and training (15) were very similar.

    “It is also worth noting that the numbers of studies investigating cognitive testing (17) and training (15) were very similar”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    General executive function, attention, and inhibition were also commonly tested domains.

    “General executive function (EF), attention, and inhibition were also commonly tested domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The Great Brain Experiment aimed to keep task sessions below 5 minutes and found that the shortest task, the Stop Signal Task, was the most popular mini game.

    “The Great Brain Experiment aimed to keep task sessions below 5 minutes in duration and found that the shortest task, the Stop Signal Task, was the most popular mini game.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Comparable numbers of games were used for cognitive testing (17) and training (13), with one game that can be used for both testing and training.

    “We found comparable numbers of games used for cognitive testing (17) and training (13), with one game that can be used for both testing and training”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    These games tended to build simple game-archetypes around an existing cognitive task, with the goal of encouraging self-motivation, improving participant enjoyment, and reducing test anxiety.

    “these games tended to build simple game-archetypes, such as space invaders or whack-a-mole, around an existing cognitive task; with the goal of encouraging self-motivation, improving participant enjoyment, and even reducing test anxiety [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] McPherson, J, 2007, Behav Res Methods source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Whack-a-Mole's primary output measure was correlated with inhibition (r(22)=.60, P<.001) and updating (r(22)=.35, P<.05).

    “For example, use of exploratory factor analysis showed that Whack-a-Mole’s primary output measure was correlated with 2 of the 3 EFs of interest: inhibition (r (22)=.60, P<.001) and updating (r (22) = .35, P<.05) [33].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • May combine two claims.

    Findingreasoning

    Validation studies varied in design, and some reported complex correlations between gamified and nongamified tasks with multiple outputs.

    “Validation studies varied in their design, and some studies reported complex correlations between gamified and nongamified tasks with multiple outputs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There is a smaller overlap in domains covered by cognitive training tasks; test batteries tend to test a wide range of domains, whereas training focuses on 1 or 2 domains exclusively.

    “There is a smaller overlap in domains covered by cognitive training tasks; test batteries tend to test a wide range of domains, whereas training focuses on 1 or 2 domains exclusively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The initial study was successful, with Space Code's output measure correlating well with a conventional measure of processing speed (r(58)=.55, P<.001).

    “The initial study was successful, with Space Code’s output measure correlating well with a conventional measure of processing speed (r (58)=.55, P<.001).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Continuous Performance Task-II detected more commission errors from ADHD adults as expected (M=56, SD=13 vs M=46, SD=10), but Retirement Party did not (M=14.4, SD=5.8 vs M=13.2, SD=4.3).

    “The Continuous Performance Task-II detected more commission errors from the ADHD adults as expected (mean [M]=56, standard deviation [SD]=13 vs M=46, SD=10), but Retirement Party did not (M=14.4, SD=5.8 vs M=13.2, SD=4.3)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    We saw several games aimed at training and testing people with ADHD, and overall, these games appear highly engaging to users, in some cases, even increasing the time spent training.

    “We saw several games aimed at training and testing people with ADHD, and overall, these games appear highly engaging to users, in some cases, even increasing the time spent training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Gamification was reported as both motivational and positive by study authors and participants alike.

    “Indeed, gamification was reported as both motivational and positive by study authors and participants alike.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Our review found no evidence to support that gamified tests can be used to improve data quality, either by reducing between-subject noise or by improving participant performance.

    “Our review found no evidence to support that gamified tests can be used to improve data quality, either by reducing between-subject noise or by improving participant performance”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    3D graphics were quite prevalent in training games, as was the use of avatars, points, levels, and dynamically growing game worlds.

    “3D graphics were quite prevalent, as was the use of avatars, points, levels, and dynamically growing game worlds”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Assessment games were simpler, predominantly using 2D graphics, sound effects, score, and theme to create the appearance of a game.

    “In contrast, assessment games were simpler, predominantly using 2D graphics, sound effects, score, and theme to create the appearance of a game.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Training games typically contained many features and were similar in appearance to commercial video games.

    “Training games typically contained many features and were similar in appearance to commercial video games.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Cognitive training normally requires several sessions to be effective.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Papers were included in the review only on the agreement of both reviewers.

    “Papers were only included in the review on the agreement of both JL and EE.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Inter-rater reliability of paper inclusion at the 20% data check stage was moderate (k=.526, 95% CI 0.416 to 0.633, P<.001).

    “We used Cohen K to assess inter-rater reliability of paper inclusion at the 20% data check stage (7590 papers checked). There was moderate agreement between the 2 reviewers (k=.526, 95% CI, 0.416 to 0.633, P<.001).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Video games appear to have positive effects on working memory, attentional capacity, problem solving, motivation, emotional control, and prosocial behaviors.

    “Furthermore, using games as a vehicle to deliver cognitive training may also be advantageous simply because video games appear to have positive effects on a number of outcomes, including working memory, attentional capacity, problem solving, motivation, emotional control, and prosocial behaviors [6].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A second reviewer rescreened 20% of the papers from the title stage onward to ensure that no relevant papers were missed.

    “A second reviewer (EE) rescreened 20% of the papers from the title stage onward to ensure that no relevant papers were missed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Cognitive assessment has been accused of being ecologically invalid.

    “In a similar vein, cognitive assessment has been accused of being ecologically invalid [60].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [60] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Patients with ADHD react differently from controls to rewards and feedback: they prefer strong reinforcement and immediate feedback, as well as clear goals and objectives.

    “Furthermore, patients with ADHD react differently from controls to rewards and feedback: they prefer strong reinforcement and immediate feedback, as well as clear goals and objectives, all of which can easily be delivered in a gamelike environment [22,24,62].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [22] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

    • cites [62] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Video game players outperform nongamers on tests of working memory, visual attention, and processing speed.

    “There is good evidence that video game players outperform nongamers on tests of working memory [52] visual attention [53,54] and processing speed [55,56].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [52] source

      Not checked in the free run.

    • cites [53] Durlach, P, 2009, Mil Psychol source

      Not checked in the free run.

    • cites [54] source

      Not checked in the free run.

    • cites [55] source

      Not checked in the free run.

    • cites [56] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Working memory was the most commonly tested domain likely because it is easy to test and because working memory deficit is a common symptom in many disorders.

    “This is likely due to its ease of testing and also because working memory deficit is a common symptom in many disorders.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Working memory deficit is a common symptom in many disorders.

  • Classified as probably not a substantial claim.

    Premisecitation

    Games are uniquely suited to some forms of cognitive training because they give the player freedom to make choices and experience feedback, providing opportunities for experiential learning.

    “As Dunbar et al explain [8], games are uniquely suited to some forms of cognitive training as they give the player freedom to make choices and experience feedback on the effects of those choices; in other words, they provide opportunities for experiential learning [61].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [8] source

      Not checked in the free run.

    • cites [61] Kolb, D, 1984, Experiential learning: experience as the source of learning and development

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The slightly smaller proportion of games used to deliver cognitive training is likely explained by the relative recency of the field.

    “the slightly smaller proportion of games used to deliver cognitive training is likely explained by the relative recency of the field.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Gamification aims to increase engagement with tasks that might otherwise be perceived as demotivating.

    “Gamification aims to increase engagement with tasks that might otherwise be perceived as demotivating [48]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] ErikssonBMusialikMWagnerJStudent essay20120862016-07-08Gamification - Engaging the Future https://gupea.ub.gu.se/handle/2077/30037

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Commercial gamification is often used to create long-term interest around a user experience, product, or event.

    “Commercial gamification is often used to create long-term interest around a user experience, product, or event [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Broadly speaking, these were well-designed and well-powered studies, and together they provide encouraging evidence that cognitive tests can be gamified and still be useful as a research tool.

    “Broadly speaking, these were well-designed and well-powered studies, and together, they provide encouraging evidence that cognitive tests can be gamified and still be useful as a research tool.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    A performance boost resulting from gamelike elements is a disadvantage when performing cognitive assessment but is likely desirable in a cognitive training scenario.

    “Such a performance boost resulting from gamelike elements is a disadvantage when performing cognitive assessment but is likely to be desirable in a cognitive training scenario.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Designing gamified cognitive tasks is difficult, and multiple, well-powered validation studies may be required to ensure a task is measuring what is intended.

    “The fact that Space Code was thought to be a fairly pure measure in one study and then was shown to be mixed-domain in the next highlights the fact that designing gamified cognitive tasks is difficult, and multiple, well-powered validation studies may be required to ensure a task is measuring what is intended.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Given the relatively large combined sample size of these studies (N=200), they provide good evidence that game mechanics can be included in cognitive tests without invalidating the data and with the desired effect of increasing motivation.

    “Given the relatively large combined sample size of these studies (N=200), they provide good evidence that game mechanics can be included in cognitive tests without invalidating the data and with the desired effect of increasing motivation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The dopaminergic system is thought to be abnormal in ADHD.

    “The dopaminergic system is thought to be abnormal in ADHD [66,67].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [66] source

      Not checked in the free run.

    • cites [67] 1996, Mol Psychiatry

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Gamified tasks may be valuable for assessing ADHD patients because computer games are particularly appealing to them, with rapid rewards, immediate feedback, and time-pressure being exactly the type of stimulus the ADHD brain craves.

    “Gamified tasks may be valuable for assessing ADHD patients as computer games are particularly appealing to them: with rapid rewards, immediate feedback, and time-pressure being exactly the type of stimulus the ADHD brain craves [64,65].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [64] Kok, A, 2002, Autism source

      Not checked in the free run.

    • cites [65] WebMD20122016-04-19Video Games and ADHD: What's the Link? http://www.webmd.com/add-adhd/childhood-adhd/features/adhd-and-video-games-is-there-a-link?print=true

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The Great Brain Experiment and GAME both showed gamelike tasks to be appropriate for measuring and training working memory.

    “The Great Brain Experiment [39] and GAME [46] both showed gamelike tasks to be appropriate for measuring and training working memory.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [39] source

      Not checked in the free run.

    • cites [46] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Data collected from two very different gamified and nongamified tasks could fit similar models of working memory capacity.

    “McNab and Dolan showing that data collected from 2 very different gamified and nongamified tasks could fit similar models of working memory capacity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Children in the gamelike condition were more engaged and enjoyed the training more, as measured by absence time.

    “Children in the gamelike condition were also more engaged and enjoyed the training more, as measured by “absence time.””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Watermons gamelike features increased the effect of training, reducing reaction times and switch costs, compared against a non-gamelike version of the training.

    “They found that these gamelike features increased the effect of training, reducing reaction times and switch costs, compared against a non–gamelike version of the training.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    Many of the studies reviewed were of a preliminary nature, and as such, the findings reported here should be considered tentative.

    “An additional consideration is that many of the studies reviewed were of a preliminary nature, and as such, the findings reported here should be considered tentative.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Gamified cognitive tasks make the participant experience less effortful and potentially reduce drop-outs in longitudinal studies.

    “thus making the participant experience less effortful and potentially reducing drop-outs in longitudinal studies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Tasks of this nature represent gamification at its simplest, but they were well received by users, implying that minimal gamification is better than no gamification.

    “Tasks of this nature represent gamification at its simplest, but they were well received by users, implying that minimal gamification is better than no gamification.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The simplicity of gamification employed is likely due to the constant tension between creating an engaging task and the risk of undermining the task's scientific validity: including unknown game mechanics might have deleterious effects on the data collected.

    “The simplicity of gamification employed is likely due to the constant tension between creating an engaging task and the risk of undermining the task’s scientific validity: including unknown game mechanics might have deleterious effects on the data collected.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Long-term goals served to sustain engagement over a long period.

    “and served to sustain engagement over a long period.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Papers were only included in the review on consensus from both reviewers (JL and EE), and a 20% selection check was performed by EE on papers from the title-screening stage onward.

    “papers were only included in the review on consensus from both reviewers (JL and EE), and a 20% selection check was performed by EE on papers from the title-screening stage onward.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Katz et al and Miranda and Palmer highlight that balancing motivation and performance can be difficult, with several game mechanics having unforeseen deleterious effects on performance.

    “However, Katz et al [34] and Miranda and Palmer [36] highlight that this can be difficult balancing act, with several game mechanics having unforeseen deleterious effects on performance.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [34] source

      Not checked in the free run.

    • cites [36] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Results of references [9,17,18,38] show that gamified tasks can be used to improve motivation, while still maintaining a scientifically valid task.

    “The results of references [9,17,18,38], show that gamified tasks can be used to improve motivation, while still maintaining a scientifically valid task.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [9] source

      Not checked in the free run.

    • cites [17] McPherson, J, 2007, Behav Res Methods source

      Not checked in the free run.

    • cites [18] source

      Not checked in the free run.

    • cites [38] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Many training games used a gamified task only, meaning the effect of gamification cannot be dissociated from the effect of the intervention.

    “Many training games used a gamified task only, meaning the effect of gamification cannot be dissociated from the effect of the intervention.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Playing video games may improve player performance and motivation.

    “this may improve player performance and motivation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Gamelike tasks may also reduce feelings of test anxiety and allow alternative interfaces to cognitive tests that would otherwise be difficult to deliver in certain populations.

    “Gamelike tasks may also reduce feelings of test anxiety and allow alternative interfaces to cognitive tests that would otherwise be difficult to deliver in certain populations.”
    Quote in the paperNot a substantial claim
57 of 57

  • The paper relies on this without support.

    Cognitive tasks are typically viewed as effortful, frustrating, and repetitive.

    “Cognitive tasks are typically viewed as effortful, frustrating, and repetitive, which often leads to participant disengagement.”

  • The paper relies on this without support.

    To the authors' knowledge, there have been no systematic reviews of gamification in cognitive research.

    “To our knowledge, there have been no systematic reviews of gamification in cognitive research.”

  • The paper relies on this without support.

    The highly repetitive nature of cognitive tasks means they are ripe for improvement.

    “the highly repetitive nature of cognitive tasks means they are ripe for improvement.”

  • The paper relies on this without support.

    Cognitive tests are typically a one-off measure, and replayability is not a requirement.

    “Cognitive tests are typically a one off measure, and replayability is not a requirement.”

  • The paper relies on this without support.

    Working memory deficit is a common symptom in many disorders.

    “working memory deficit is a common symptom in many disorders.”

  • The paper relies on this without support.

    Cognitive training normally requires several sessions to be effective.

    “Cognitive training normally requires several sessions to be effective”

  • The paper relies on this without support.

    Engagement will be the key to collecting high-quality data as cognitive research moves out of the laboratory and onto personal computers and mobile devices.

    “As cognitive research begins to move out of the laboratory and onto personal computers and mobile devices, engagement will be the key to collecting high-quality data.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Serious game interventions have been found to improve time management and planning/organization and reduce hyperactivity symptoms in students with ADHD.

    “Those authors found improvements in time management and planning/organization, and a reduction in hyperactivity symptoms in a group of students with ADHD who received an SG intervention.”

    From Serious Games and Their Effect Improving Attention in Students with Learning Disabilities (2019) · cites this paper as [16]

  • The current results are consistent with previous research stating that playing videogames can increase concentration and other cognitive variables, such as processing speed or visual discrimination.

    “These results are in line with previous research stating that playing videogames can increase concentration and other cognitive variables, such as processing speed or visual discrimination [12,13,14,15,16].”

    From Serious Games and Their Effect Improving Attention in Students with Learning Disabilities (2019) · cites this paper as [16]

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • p-value fits an unstated one-tailed test: r (22) = .35, p < .05

    Minor

    The paper reports p < .05 here; recomputing from the test statistic and its degrees of freedom gives p = .094 (two-tailed). The reported value is what a one-tailed test gives, but the paper does not say near this result that a one-tailed test was used.

    Reported p < .05 · recomputed p = .094

  • p-value fits an unstated one-tailed test: r (22)=.60, p < .001

    Minor

    The paper reports p < .001 here; recomputing from the test statistic and its degrees of freedom gives p = .002 (two-tailed). The reported value is what a one-tailed test gives, but the paper does not say near this result that a one-tailed test was used.

    Reported p < .001 · recomputed p = .002

2 of 2

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upr(22) = 0.6reported p < .001 · recomputed p = .002

    “r (22)=.60, p < .001”

  • Doesn’t add upr(22) = 0.35reported p < .05 · recomputed p = .094

    “r (22) = .35, p < .05”

  • Consistentr(58) = 0.55reported p < .001 · recomputed p < .001

    “r (58)=.55, p < .001”

3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Trapp study has conflicting publication years

    Minor

    Table 1 dates the Trapp et al study to 2008, while its matching reference lists BMC Psychiatry 2013.

  • Ninaus finding points to a different reference

    Minor

    The text attributes the working-memory performance finding to Ninaus et al but cites reference 48, which is an essay by Eriksson and colleagues. The Ninaus study is listed as reference 46.

2 of 2

Cites

Flagged first, then by citations

1 of the 71 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Gamberini, L, 2010, Stud Health Technol Inform, 154, 29
  2. 1996, Mol Psychiatry, 1, 121

Cited by

Sorted by citations

Related

Authors
Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R.
Affiliations
  • University of Bristol (GB)
  • Queen Mary University of London (GB)
  • University of Exeter (GB)
  • University College Dublin (IE)
Corresponding author
Lumsden, Jim
ORCID
Venue
JMIR Serious Games
Publisher
JMIR Publications Inc.
Published
2016-07-15
Volume
4
Issue
2
Pages
e11
ISSN
2291-9279 (ISSN-L)
DOI
10.2196/games.5888
PMC
PMC4967181
OpenAlex
W2465353856
Semantic Scholar
4f2c6684a94d57d5dc731e41341a6b0109778428
Cited by
555
References
71
Citations by year
2026: 57 · 2025: 77 · 2024: 74 · 2023: 52 · 2022: 61 · 2021: 56 · 2020: 70 · 2019: 43 · 2018: 37 · 2017: 22 · 2016: 6
FWCI
6.76
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Serious Games (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Open Research Exeter (University of Exeter) (acceptedVersion)
  • Arrow@dit (Dublin Institute of Technology) (submittedVersion)
  • Bristol Research (University of Bristol) (submittedVersion, other-oa)
  • Research Repository UCD (University College Dublin) (submittedVersion, cc-by-nc-nd)
Funders
  • Wellcome Trust (092731)
  • United Kingdom Clinical Research Collaboration
  • Cancer Research UK
  • National Institute for Health and Care Research (NIHR-IPF-16-10-04)
  • British Heart Foundation
  • University of Bristol (092731, MC_UU_12013/6)
  • Medical Research Council (MC_UU_12013, MR/K023195/1, 092731, MC_UU_12013/6)
  • Economic and Social Research Council (ES/G007489/1, 1325188)
Topics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
Keywords
gamification, cognitive assessment, cognitive training, participant engagement, working memory, executive function, game elements, task validation, motivation, neuropsychological assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review