Developing a serious game for cognitive assessment: choosing settings and measuring performance
Abstract
Gamification and serious games are becoming increasingly important for training, wellness, and other applications. How can games be developed for non-traditional gaming populations such as the elderly, and how can gaming be applied in non-traditional areas such as cognitive assessment? The application that we were interested in is detection of cognitive impairment in the elderly. Example use cases where gamified cognitive assessment might be useful are: prediction of delirium onset risk in emergency departments and postoperative hospital wards; evaluation of recovery from stroke in neuro-rehabilitation; monitoring of transitions from mild cognitive impairment to dementia in long-term care. With the rapid increase in cognitive disorders in many countries, inexpensive methods of measuring cognitive status on an ongoing basis, and to large numbers of people, are needed. In order to address this challenge we have developed a novel game-based method of cognitive assessment. In this paper, we present findings from a usability study conducted on the game that we developed for measuring changes in cognitive status. We report on the game's ability to predict cognitive status under varying game parameters, and we introduce a method to calibrate the game that takes into account differences in speed and accuracy, and in motor coordination. Recommendations concerning the development of serious games for cognitive assessment are made, and detailed recommendations concerning future development of the whack-a-mole game are also provided.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Whack-a-Mole's primary output measure was correlated with inhibition (r(22)=.60, P<.001) and updating (r(22)=.35, P<.05).
“For example, use of exploratory factor analysis showed that Whack-a-Mole’s primary output measure was correlated with 2 of the 3 EFs of interest: inhibition (r (22)=.60, P<.001) and updating (r (22) = .35, P<.05) [33].”
From Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy (2016) · cites this paper as [33]
Cites
The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment
Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I., Cummings, J. L., & Chertkow, H. (2005) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2005.53221.x · Cited by 25,553 · see citing works
(2011) · Cited by 14,317 · see citing works
Working memory: Looking back and looking forward
Baddeley, A. (2003) · Nature Reviews Neuroscience · doi:10.1038/nrn1201 · Cited by 5,200 · see citing works
Miyake, A., & Friedman, N. P. (2012) · Current Directions in Psychological Science · doi:10.1177/0963721411429458 · Cited by 3,944 · see citing works
Robbins, T. W., James, M., Owen, A. M., Sahakian, B. J., McInnes, L., & Rabbitt, P. (1994) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000106735 · Cited by 1,078 · see citing works
Executive functions and their disorders
Elliott, R. (2003) · British Medical Bulletin · doi:10.1093/bmb/65.1.49 · Cited by 792 · see citing works
What cognitive abilities are involved in trail-making performance?
Salthouse, T. A. (2011) · Intelligence · doi:10.1016/j.intell.2011.03.001 · Cited by 508 · see citing works
Cognitive assessment in the elderly: a review of clinical methods
Woodford, H. J., & George, J. (2007) · QJM · doi:10.1093/qjmed/hcm051 · Cited by 341 · see citing works
Johnson, J. K., Lui, L.-Y., & Yaffe, K. (2007) · The Journals of Gerontology Series A: Biological Sciences and Medical Sciences · doi:10.1093/gerona/62.10.1134 · Cited by 339 · see citing works
Nutley, S. B., Söderqvist, S., Bryde, S., Thorell, L. B., Humphreys, K., & Klingberg, T. (2011) · Developmental Science · doi:10.1111/j.1467-7687.2010.01022.x · Cited by 284 · see citing works
Working memory training and transfer in older adults.
Richmond, L. L., Morrison, A. B., Chein, J. M., & Olson, I. R. (2011) · Psychology and Aging · doi:10.1037/a0023631 · Cited by 257 · see citing works
Delirium in the elderly: a clinical review
Saxena, S. A., & Lawley, D. (2009) · Postgraduate Medical Journal · doi:10.1136/pgmj.2008.072025 · Cited by 186 · see citing works
Neuropsychological testing and assessment for dementia
Jacova, C., Kertesz, A., Blair, M., Fisk, J. D., & Feldman, H. (2007) · Alzheimer s & Dementia · doi:10.1016/j.jalz.2007.07.011 · Cited by 142 · see citing works
Executive Function and Depression as Independent Risk Factors for Postoperative Delirium
Smith, P. J., Attix, D. K., Weldon, B. C., Greene, N. H., & Monk, T. G. (2009) · Anesthesiology · doi:10.1097/aln.0b013e31819b5bc2 · Cited by 137 · see citing works
Inhibitory Control Across the Life Span
Christ, S. E., White, D. A., Mandernach, T., & Keys, B. A. (2001) · Developmental Neuropsychology · doi:10.1207/s15326942dn2003_7 · Cited by 132 · see citing works
Mechanisms of speed–accuracy tradeoff: evidence from covert motor processes
Osman, A. M., Lou, L., Müller‐Gethmann, H., Rinkenauer, G., Mattes, S., & Ulrich, R. (2000) · Biological Psychology · doi:10.1016/s0301-0511(99)00045-9 · Cited by 131 · see citing works
An empirical investigation of the novice experience with soft keyboards
MacKenzie, I. S., & Zhang, S. X. (2001) · Behaviour and Information Technology · doi:10.1080/01449290110089561 · Cited by 95 · see citing works
Gamberini, L., Martino, F., Seraglia, B., Spagnolli, A., Fabregat, M. E., Ibáñez, F. S., Alcañíz, M., & Andrés, J. M. (2009) · Conference on Human System Interactions · doi:10.1109/hsi.2009.5090973 · Cited by 74 · see citing works
Strömberg, L. E., Lindgren, U., Nordin, C., Öhlén, G., & Svensson, O. (1997) · Scandinavian Journal of Caring Sciences · doi:10.1111/j.1471-6712.1997.tb00450.x · Cited by 71 · see citing works
Cognitive impairment in medical inpatients. II: Do physicians miss cognitive impairment?
Harwood, D., Hope, T., & Jacoby, R. J. (1997) · Age and Ageing · doi:10.1093/ageing/26.1.37 · Cited by 63 · see citing works
In Search of Excellence in Anesthesiology
Smith, A. F. (2008) · Anesthesiology · doi:10.1097/aln.0b013e318190b263 · Cited by 42 · see citing works
Cognitive impairment in medical inpatients
Torisson, G. (2015) · Lund University Publications (Lund University) · Cited by 4 · see citing works
(1931) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1931.tb52849.x · Cited by 2 · see citing works
(2007) · Alzheimer s & Dementia · doi:10.1016/s1552-5260(06)04904-1 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016) · JMIR Serious Games · doi:10.2196/games.5888 · Cited by 555 · see citing works
“Testing Whack-a-mole Developing a serious game for cognitive assessment: choosing settings and measuring performance Tong and Chignell, 2014 [33]”
BackgroundMethodAn application of adaptive games-based learning based on learning style to teach SQL
Soflano, M., Connolly, T. M., & Hainey, T. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.03.015 · Cited by 198 · see citing works
A Serious Game for Clinical Assessment of Cognitive Status: Validation Study
Tong, T., Chignell, M., Tierney, M. C., & Lee, J. (2016) · JMIR Serious Games · doi:10.2196/games.5006 · Cited by 132 · see citing works
Self-citationTrends on the application of serious games to neuropsychological evaluation: A scoping review
Valladares‐Rodríguez, S., Pérez-Rodríguez, R., Anido, L., & Iglesias, M. J. F. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.019 · Cited by 112 · see citing works
A systematic literature review on user centered design and participatory design with older people
Duque, E., Fonseca, G., Vieira, H., Gontijo, G., & Ishitani, L. (2019) · Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems · doi:10.1145/3357155.3358471 · Cited by 101 · see citing works
A Gamification Framework for Cognitive Assessment and Cognitive Training: Qualitative Study
Khaleghi, A., Aghaei, Z., & Mahdavi, M. A. (2021) · JMIR Serious Games · doi:10.2196/21900 · Cited by 106 · see citing works
Academic methods for usability evaluation of serious games: a systematic review
Yáñez-Gómez, R., Cascado-Caballero, D., & Sevillano, J. L. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3845-9 · Cited by 83 · see citing works
Ford, E. M., Milne, R., & Curlewis, K. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1492 · Cited by 80 · see citing works
Lumsden, J., Skinner, A., Woods, A. T., Lawrence, N., & Munafo, M. R. (2016) · PeerJ · doi:10.7717/peerj.2184 · Cited by 61 · see citing works
Touchscreen games to detect cognitive impairment in senior adults. A user-interaction pilot study
Valladares-Rodriguez, S., Fernández-Iglesias, M. J., Anido-Rifón, L., Facal, D., Rivas-Costa, C., & Pérez-Rodríguez, R. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.04.012 · Cited by 55 · see citing works
Lumsden, J., Skinner, A., Coyle, D., Lawrence, N., & Munafo, M. R. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8473 · Cited by 56 · see citing works
Digital Cognitive Assessment Tests for Older Adults: Systematic Literature Review
Cubillos, C., & Rienzo, A. (2023) · JMIR Mental Health · doi:10.2196/47487 · Cited by 45 · see citing works
Tong, T., Chan, J. H., & Chignell, M. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054930 · Cited by 39 · see citing works
Self-citationUsing a serious game to measure executive functioning: Response inhibition ability
Tong, T., Chignell, M., & DeGuzman, C. A. (2019) · Applied Neuropsychology Adult · doi:10.1080/23279095.2019.1683561 · Cited by 35 · see citing works
Self-citationDesigning Serious Games for Cognitive Assessment of the Elderly
Tong, T., Chignell, M., Lam, P., Tierney, M. C., & Lee, J. S. (2014) · Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · doi:10.1177/2327857914031004 · Cited by 34 · see citing works
Self-citationA Review of Research on Gamification Approach in Education
Kocakoyun, S., & Özdamlı, F. (2018) · InTech eBooks · doi:10.5772/intechopen.74131 · Cited by 35 · see citing works
The Making and Evaluation of Digital Games Used for the Assessment of Attention: Systematic Review
Wiley, K., Robinson, R., & Mandryk, R. L. (2021) · JMIR Serious Games · doi:10.2196/26449 · Cited by 34 · see citing works
Episodix: a serious game to detect cognitive impairment in senior adults. A psychometric study
Valladares‐Rodríguez, S., Iglesias, M. J. F., Anido, L., Facal-Mayo, D., & Pérez-Rodríguez, R. (2018) · PeerJ · doi:10.7717/peerj.5478 · Cited by 29 · see citing works
How Points and Theme Affect Performance and Experience in a Gamified Cognitive Task
Wiley, K., Vedress, S., & Mandryk, R. L. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376697 · Cited by 25 · see citing works
Test-Retest Reliability of a Serious Game for Delirium Screening in the Emergency Department
Tong, T., Chignell, M., Tierney, M. C., & Lee, J. S. (2016) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2016.00258 · Cited by 25 · see citing works
Self-citationScreening for Mild Cognitive Impairment in the Preoperative Setting: A Narrative Review
Pas, M. T., Rikkert, M. G. M. O., Bouwman, R. A., Kessels, R. P. C., & Buise, M. P. (2022) · Healthcare · doi:10.3390/healthcare10061112 · Cited by 24 · see citing works
FISHERMAN: A Serious Game for Executive Function Assessment of Older Adults
Wang, P., Fang, Y., Qi, J.-Y., & Li, H. (2022) · Assessment · doi:10.1177/10731911221105648 · Cited by 23 · see citing works
Sánchez-Morales, A., Durand-Rivera, J. A., & Martínez‐González, C. L. (2020) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2020.0110661 · Cited by 19 · see citing works
Costa, C. R., Iglesias, M. J. F., Anido, L., Carballa, M. G., & Valladares‐Rodríguez, S. (2017) · PeerJ · doi:10.7717/peerj.2845 · Cited by 20 · see citing works
Dutra, T. C., Felipe, Ð., Gasparini, I., & Maschio, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78092-0_4 · Cited by 18 · see citing works
Valladares‐Rodríguez, S., Iglesias, M. J. F., Anido, L., & Pacheco-Lorenzo, M. R. (2022) · Electronics · doi:10.3390/electronics11213424 · Cited by 16 · see citing works
Pallavicini, F., Pepe, A., & Minissi, M. E. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23563-5_12 · Cited by 13 · see citing works
Ahmad, F., Ahmed, Z., Shaheen, M., Muneeb, S., & Riasat, R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15100-x · Cited by 12 · see citing works
Criteria of Mobile Psychotherapy Games for Memory Disorder: A Systematic Literature Review
Sie-Yi, L. L., & ChePa, N. (2020) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v14i05.13345 · Cited by 10 · see citing works
Monitoring Cognitive Performance with a Serious Game
Urakami, J., Hu, Y. Z., & Chignell, M. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3443431 · Cited by 10 · see citing works
Self-citationTAG-ME again: A serious game for measuring working memory
Hu, Y. Z., Parimoo, S., Chignell, M., Lowe, C. J., & Morton, J. B. (2023) · Applied Neuropsychology Adult · doi:10.1080/23279095.2023.2183361 · Cited by 10 · see citing works
Self-citationAnalysis of Voice Biomarkers for the Detection of Cognitive Impairment
Pacheco-Lorenzo, M. R., Christensen, H., Anido, L., Iglesias, M. J. F., & Valladares‐Rodríguez, S. (2024) · IEEE Access · doi:10.1109/access.2024.3442431 · Cited by 7 · see citing works
Improving the Ergonomics of Cognitive Assessment with Serious Games
Tong, T., Yeung, J., Sandrakumar, J., Chignell, M., Tierney, M. C., & Lee, J. S. (2015) · Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · doi:10.1177/2327857915041001 · Cited by 7 · see citing works
Self-citationRienzo, A., Cubillos, C., & Soto, G. (2021) · IEEE International Conference on Automatica · doi:10.1109/icaacca51523.2021.9465207 · Cited by 6 · see citing works
ChePa, N., Sie-Yi, L. L., & Adetunmbi, S. (2022) · Journal of Advanced Research in Applied Sciences and Engineering Technology · doi:10.37934/araset.28.2.162180 · Cited by 5 · see citing works
Picture-Word Interference Effects Are Robust With Covert Retrieval, With and Without Gamification
Wei, H. T., Hu, Y. Z., Chignell, M., & Meltzer, J. A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.825020 · Cited by 3 · see citing works
Self-citationGame-based Experiments on Human Visual Attention
Abeele, V. V., Tierens, P., Schutter, B. D., de‐Wit, L., & Geurts, L. (2015) · Foundations of Digital Games · Cited by 4 · see citing works
Attention and Motor skill improvements in Mild Cognitive Impairment patients using COSMA Application
Christogianni, A., Bojan, K., Mukaetova-Ladinska, E. B., Sriramm, V. T., Murth, G., & Kumarpilla, G. (2022) · International Journal of Serious Games · doi:10.17083/ijsg.v9i4.517 · Cited by 3 · see citing works
Ajana, K., Everard, G. J., Lejeune, T. M., & Edwards, M. (2025) · Technology and Health Care · doi:10.1177/09287329241296380 · Cited by 2 · see citing works
A Gamified Approach to Cognitive Assessment with Machine Learning Based Predictions
Simpson, A. I. F., An, Y., Estep, J., Saraf, A., & Raiti, J. (2022) · IEEE Global Humanitarian Technology Conference (GHTC) · doi:10.1109/ghtc55712.2022.9910615 · Cited by 2 · see citing works
PANORAMA DA ERGONOMIA E DA USABILIDADE NO DESENVOLVIMENTO DE JOGOS SÉRIOS
Faust, F. G., Sierra, B. I. D. S., Merino, E. A. D., Nickel, E. M., & Ferreira, M. G. G. (2017) · doi:10.5151/16ergodesign-0003
No effect of gamification on attrition from a longitudinal cognitive testing studyPreprint
Lumsden, J., Skinner, A., Coyle, D., Lawrence, N., & Munafo, M. R. (2017) · doi:10.31234/osf.io/ya4sn
Christogianni, A., Bojan, K., Mukaetova-Ladinska, E. B., Sriramm, V. T., Murthy, G., & Kumarpillai, G. (2023) · Augmented Human Research · doi:10.1007/s41133-023-00061-5
Accuracy of Speech-to-Text Transcription in a Digital Cognitive Assessment for Older Adults
Gordon, A. M., & Wais, P. E. (2025) · Brain Sciences · doi:10.3390/brainsci15101090
Alignment in Gamified Cognitive Assessment
Bauge, K., & Babić, A. (2026) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-032-37739-5_29
- Authors
- Tong, T., & Chignell, M.
- Affiliations
- University of Toronto (CA)
- Venue
- Proceedings of the Second International Symposium of Chinese CHI
- Publisher
- ACM
- Published
- 2014-04-26
- Pages
- 70-79
- OpenAlex
- W1964622906
- Cited by
- 48
- References
- 25
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 5 · 2022: 7 · 2021: 5 · 2020: 3 · 2019: 4 · 2018: 3 · 2017: 5 · 2016: 8 · 2015: 3 · 2014: 1
- FWCI
- 12.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium of Chinese CHI (Chinese CHI) (publishedVersion)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
- Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
- Keywords
- serious games, gamification, cognitive assessment, cognitive impairment detection, older adults, usability testing, delirium prediction, stroke recovery, mild cognitive impairment, dementia, long-term care, motor coordination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
