Immersive Virtual Reality-Based Methods for Assessing Executive Functioning: Systematic Review
Open accessPublished version of a preprint
The DOI matches its Crossref record; 108 of the 108 references it cites were found in a registry or an index.
Abstract
Background: Neuropsychological assessments traditionally include tests of executive functioning (EF) because of its critical role in daily activities and link to mental disorders. Established traditional EF assessments, although robust, lack ecological validity and are limited to single cognitive processes. These methods, which are suitable for clinical populations, are less informative regarding EF in healthy individuals. With these limitations in mind, immersive virtual reality (VR)–based assessments of EF have garnered interest because of their potential to increase test sensitivity, ecological validity, and neuropsychological assessment accessibility. Objective: This systematic review aims to explore the literature on immersive VR assessments of EF focusing on (1) EF components being assessed, (2) how these assessments are validated, and (3) strategies for monitoring potential adverse (cybersickness) and beneficial (immersion) effects. Methods: EBSCOhost, Scopus, and Web of Science were searched in July 2022 using keywords that reflected the main themes of VR, neuropsychological tests, and EF. Articles had to be peer-reviewed manuscripts written in English and published after 2013 that detailed empirical, clinical, or proof-of-concept studies in which a virtual environment using a head-mounted display was used to assess EF in an adult population. A tabular synthesis method was used in which validation details from each study, including comparative assessments and scores, were systematically organized in a table. The results were summed and qualitatively analyzed to provide a comprehensive overview of the findings. Results: The search retrieved 555 unique articles, of which 19 (3.4%) met the inclusion criteria. The reviewed studies encompassed EF and associated higher-order cognitive functions such as inhibitory control, cognitive flexibility, working memory, planning, and attention. VR assessments commonly underwent validation against gold-standard traditional tasks. However, discrepancies were observed, with some studies lacking reported a priori planned correlations, omitting detailed descriptions of the EF constructs evaluated using the VR paradigms, and frequently reporting incomplete results. Notably, only 4 of the 19 (21%) studies evaluated cybersickness, and 5 of the 19 (26%) studies included user experience assessments. Conclusions: Although it acknowledges the potential of VR paradigms for assessing EF, the evidence has limitations. The methodological and psychometric properties of the included studies were inconsistently addressed, raising concerns about their validity and reliability. Infrequent monitoring of adverse effects such as cybersickness and considerable variability in sample sizes may limit interpretation and hinder psychometric evaluation. Several recommendations are proposed to improve the theory and practice of immersive VR assessments of EF. Future studies should explore the integration of biosensors with VR systems and the capabilities of VR in the context of spatial navigation assessments. Despite considerable promise, the systematic and validated implementation of VR assessments is essential for ensuring their practical utility in real-world applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, their accuracy in assessing functional deficits in ADHD may be limited.”
From Enhancing Ecological Validity: Virtual Reality Assessment of Executive Functioning in Children and Adolescents with ADHD (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Typically conducted in clinical or laboratory settings, these tests often evaluate specific skills in isolation, potentially overlooking interactions between different EFs and their relevance to daily activities [4,5].”
From Enhancing Ecological Validity: Virtual Reality Assessment of Executive Functioning in Children and Adolescents with ADHD (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Typically conducted in clinical or laboratory settings, these tests often evaluate specific skills in isolation, potentially overlooking interactions between different EFs and their relevance to daily activities [4,5].”
From Enhancing Ecological Validity: Virtual Reality Assessment of Executive Functioning in Children and Adolescents with ADHD (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Typically conducted in clinical or laboratory settings, these tests often evaluate specific skills in isolation, potentially overlooking interactions between different EFs and their relevance to daily activities [4,5].”
From Enhancing Ecological Validity: Virtual Reality Assessment of Executive Functioning in Children and Adolescents with ADHD (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent advancements in virtual reality (VR) technology have provided promising avenues for creating immersive digital environments that simulate real-life scenarios [5].”
From Enhancing Ecological Validity: Virtual Reality Assessment of Executive Functioning in Children and Adolescents with ADHD (2024) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000) · Cognitive Psychology · doi:10.1006/cogp.1999.0734 · Cited by 15,798
Diamond, A. (2013) · Annual Review of Psychology · doi:10.1146/annurev-psych-113011-143750 · Cited by 13,268
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2010) · International Journal of Surgery · doi:10.1016/j.ijsu.2010.02.007 · Cited by 12,156 · cited 1×
“Our review methodology followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement [57].”
MethodThe journal coverage of Web of Science and Scopus: A comparative analysis
Mongeon, P., & Paul-Hus, A. (2016) · Scientometrics · doi:10.1007/s11192-015-1765-5 · Cited by 5,274 · cited 1×
“To compensate for the bias toward engineering and natural science articles found through Scopus and WoS [59], EBSCOhost was searched for articles published in fields such as (clinical) psychology and medicine.”
MethodDelis-Kaplan Executive Function System
Delis, D. C., Kaplan, E., & Kramer, J. H. (2012) · PsycTESTS Dataset · doi:10.1037/t15082-000 · Cited by 2,683
Assessment and Development of Executive Function (EF) During Childhood
Anderson, P. (2002) · Child Neuropsychology · doi:10.1076/chin.8.2.71.8724 · Cited by 2,218
DEFICITS IN STRATEGY APPLICATION FOLLOWING FRONTAL LOBE DAMAGE IN MAN
SHALLICE, T., & BURGESS, P. W. (1991) · Brain · doi:10.1093/brain/114.2.727 · Cited by 2,017 · cited 1×
“For example, the Multiple Errands Test (MET) [31] requires individuals to run errands in a real environment (eg, a shopping center).”
BackgroundEssentials of Psychological Testing
Urbina, S. (2014) · doi:10.1002/9781394259458 · Cited by 1,845
Snyder, H. R. (2013) · Psychological Bulletin · doi:10.1037/a0028727 · Cited by 1,667 · cited 1×
“Although traditional methods used to assess EF are popular [11,12] and well validated [13], they have been criticized for their lack of ecological validity [14,15].”
MethodA tale of two databases: the use of Web of Science and Scopus in academic papers
Zhu, J., & Liu, W. (2020) · Scientometrics · doi:10.1007/s11192-020-03387-8 · Cited by 1,556 · cited 1×
“Scopus and WoS were chosen because of their prominence as citation databases [58].”
BackgroundNigg, J. T. (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12675 · Cited by 1,518 · cited 1×
““Inhibitory control” encompasses the inhibition of goal-irrelevant stimuli, cognitions, and behavioral responses [6,84].”
BackgroundAssessment of executive functions: Review of instruments and identification of critical issues.
CHAN, R., SHUM, D., TOULOPOULOU, T., & CHEN, E. (2008) · Archives of Clinical Neuropsychology · doi:10.1016/j.acn.2007.08.010 · Cited by 1,401 · cited 2×
“Outside the testing environment, the process of making a decision or planning and eliciting goal-directed behavior in everyday life is often highly dynamic and influenced by numerous internal and external factors [13,14].”
BackgroundMethodDimensions of executive functioning: evidence from children
Lehto, J. E., Juujärvi, P., Kooistra, L., & Pulkkinen, L. (2003) · British Journal of Developmental Psychology · doi:10.1348/026151003321164627 · Cited by 1,226 · cited 3×
“As the literature [1,2,4,6] does not relate these broad domains to EF, they are not discussed further but are presented in Tables 5-6.”
BackgroundMethodBehavioural Assessment of the Dysexecutive Syndrome
(2004) · Methodology Of Frontal And Executive Function · doi:10.4324/9780203344187-15 · Cited by 1,149
The Psychology Experiment Building Language (PEBL) and PEBL Test Battery
Mueller, S. T., & Piper, B. J. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2013.10.024 · Cited by 1,085
Review on cybersickness in applications and visual displays
Rebenitsch, L., & Owen, C. (2016) · Virtual Reality · doi:10.1007/s10055-016-0285-9 · Cited by 971
Snyder, H. R., Miyake, A., & Hankin, B. L. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00328 · Cited by 946 · cited 2×
“As EF impairment has been linked to a variety of mental disorders [9], it is often considered a transdiagnostic risk factor [10].”
BackgroundKrath, J., Schürmann, L., & von Korflesch, H. F. O. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106963 · Cited by 943 · cited 1×
““Serious games,” defined as games intended for a variety of serious purposes, such as training, learning, stimulation, or cognitive assessment [42], have been shown to be more engaging than nongamified tasks [43-45].”
BackgroundThe ecological validity of tests of executive function
BURGESS, P. W., ALDERMAN, N., EVANS, J., EMSLIE, H., & WILSON, B. A. (1998) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617798466037 · Cited by 932 · cited 1×
“Although traditional methods used to assess EF are popular [11,12] and well validated [13], they have been criticized for their lack of ecological validity [14,15].”
MethodAutism spectrum disorders: a meta-analysis of executive function
Demetriou, E. A., Lampit, A., Quintana, D. S., Naismith, S. L., Song, Y. J. C., Pye, J. E., Hickie, I., & Guastella, A. J. (2018) · Molecular Psychiatry · doi:10.1038/mp.2017.75 · Cited by 841
Linguistic Features Identify Alzheimer’s Disease in Narrative Speech
Fraser, K. C., Meltzer, J. A., & Rudzicz, F. (2015) · Journal of Alzheimer's Disease · doi:10.3233/jad-150520 · Cited by 797 · cited 1×
“Eye tracking during reading aids Alzheimer disease identification [96], and linguistic attributes (eg, formation and fluency of sentences, syntax, and grammar) distinguish patients with Alzheimer disease from those with MCI [97].”
BackgroundExecutive functions and their disorders
Elliott, R. (2003) · British Medical Bulletin · doi:10.1093/bmb/65.1.49 · Cited by 792 · cited 1×
“As EF impairment has been linked to a variety of mental disorders [9], it is often considered a transdiagnostic risk factor [10].”
BackgroundProduct review: Covidence (systematic review software)
Babineau, J. (2014) · Journal of the Canadian Health Libraries Association / Journal de l'Association des bibliothèques de la santé du Canada · doi:10.5596/c14-016 · Cited by 721 · cited 2×
“Through WoS, EBSCOhost, and Scopus, 892 items were identified, from which the Covidence systematic review management platform [60] filtered 337 (37.8”
MethodInfluentialBURGESS, P. W., ALDERMAN, N., FORBES, C., COSTELLO, A., M-A.COATES, L., DAWSON, D. R., ANDERSON, N. D., GILBERT, S. J., DUMONTHEIL, I., & CHANNON, S. (2006) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617706060310 · Cited by 663 · cited 2×
“Outside the testing environment, the process of making a decision or planning and eliciting goal-directed behavior in everyday life is often highly dynamic and influenced by numerous internal and external factors [13,14].”
BackgroundMethodAnxiety and Impulsiveness Related to Psychomotor Efficiency
Barratt, E. S. (1959) · Perceptual and Motor Skills · doi:10.2466/pms.1959.9.3.191 · Cited by 631
The Trajectory of Gait Speed Preceding Mild Cognitive Impairment
Buracchio, T., Dodge, H. H., Howieson, D., Wasserman, D., & Kaye, J. (2010) · Archives of Neurology · doi:10.1001/archneurol.2010.159 · Cited by 563 · cited 1×
“For example, sensitive detection can identify the prodromal stages of Alzheimer disease in seemingly healthy individuals [26] and mild cognitive decline up to 12 years before clinical diagnosis [27].”
BackgroundReasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making
Collins, A., & Koechlin, E. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001293 · Cited by 558
Learning in virtual reality: Effects on performance, emotion and engagement
Allcoat, D., & von Mühlenen, A. (2018) · Research in Learning Technology · doi:10.25304/rlt.v26.2140 · Cited by 556 · cited 1×
“In addition, VR could increase engagement [28,29], reduce test time, and better integrate data from biosensors with in-task events that facilitate assessment.”
BackgroundLumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016) · JMIR Serious Games · doi:10.2196/games.5888 · Cited by 555 · cited 1×
“Recognized as a crucial aspect of cognitive assessment, engagement can be improved through gamification, thereby improving task performance [41].”
BackgroundUnreliability of the dot probe task
Schmukle, S. C. (2005) · European Journal of Personality · doi:10.1002/per.554 · Cited by 538 · cited 1×
“Furthermore, the DPT, which is typically used to assess selective attention [88], was used to assess inhibition, although its own psychometric properties have been the subject of controversy [89,90].”
MethodImproving the ecological validity of executive functioning assessment
CHAYTOR, N., SCHMITTEREDGECOMBE, M., & BURR, R. (2006) · Archives of Clinical Neuropsychology · doi:10.1016/j.acn.2005.12.002 · Cited by 462 · cited 2×
“For example, Chaytor et al [19] noted that traditional EF tests accounted for only 18% to 20% of the variance in the everyday executive ability of participants.”
BackgroundBagetta (2016) · doi:10.1111/mbe.12100 · Cited by 433 · cited 1×
“This issue, longstanding in EF research (see the study by Suchy [5]; for a review, see the study by Baggetta and Alexander [56]), presents challenges to VR in the EF assessment field.”
BackgroundEcological Validity of Neuropsychological Testing
Long, C. J., & Sbordone, R. J. (1996) · Cited by 425
Executive Functioning: Overview, Assessment, and Research Issues for Non-Neuropsychologists
Suchy, Y. (2009) · Annals of Behavioral Medicine · doi:10.1007/s12160-009-9097-4 · Cited by 408 · cited 3×
“This issue, longstanding in EF research (see the study by Suchy [5]; for a review, see the study by Baggetta and Alexander [56]), presents challenges to VR in the EF assessment field.”
BackgroundCharacteristic changes in the physiological components of cybersickness
Kim, Y. Y., Kim, H. J., Kim, E. N., Ko, H. D., & Kim, H. T. (2005) · Psychophysiology · doi:10.1111/j.1469-8986.2005.00349.x · Cited by 389 · cited 1×
“This is concerning as cybersickness presents a substantial confound for valid VR assessment and has been shown to negatively affect task performance [92,93].”
BackgroundSPOONER, D., & PACHANA, N. (2006) · Archives of Clinical Neuropsychology · doi:10.1016/j.acn.2006.04.004 · Cited by 323 · cited 2×
“…test mirrors the demands of a person’s daily living activities that it aims to evaluate [17], sometimes referred to as verisimilitude [18]—and generalizability—the extent to which test performance predicts an individual’s functioning in their daily living activities [17], also known…”
BackgroundKappenman, E. S., Farrens, J. L., Luck, S. J., & Proudfit, G. H. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01368 · Cited by 266 · cited 1×
“Furthermore, the DPT, which is typically used to assess selective attention [88], was used to assess inhibition, although its own psychometric properties have been the subject of controversy [89,90].”
MethodAbma, I. L., Rovers, M., & van der Wees, P. J. (2016) · BMC Research Notes · doi:10.1186/s13104-016-2034-2 · Cited by 245 · cited 1×
“…the traditional comparator task used to validate the VR paradigm needs to have sound psychometric properties in its own right to assess the respective construct; when two tasks are compared with one another, it is unclear which task may be responsible for discrepancies in the outcome [91].”
MethodA Hierarchical Model of Inhibitory Control
Tiego, J., Testa, R., Bellgrove, M. A., Pantelis, C., & Whittle, S. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01339 · Cited by 240 · cited 2×
“Although these terms are used in the literature [85], its readability and synthesis would be improved through agreement on a particular term for the same construct.”
BackgroundYücel, M., Oldenhof, E., Ahmed, S. H., Belin, D., Billieux, J., Bowden‐Jones, H., Carter, A., Chamberlain, S. R., Clark, L., Connor, J., et al. (2019) · Addiction · doi:10.1111/add.14424 · Cited by 229 · cited 1×
“Alternatively, researchers could engage in a Delphi study to generate expert-informed consensus on the key constructs of EF that merit investigation using VR paradigms (eg, see the study by Yücel et al [87] for a Delphi study on neuropsychological assessment for addiction).”
BackgroundSelf-citationThe assessment of executive functions: coming out of the office
Manchester, D., Priestley, N., & Jackson, H. (2004) · Brain Injury · doi:10.1080/02699050410001672387 · Cited by 219
Huang, W., Roscoe, R. D., Johnson‐Glenberg, M. C., & Craig, S. D. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12520 · Cited by 218 · cited 1×
“In addition, VR could increase engagement [28,29], reduce test time, and better integrate data from biosensors with in-task events that facilitate assessment.”
BackgroundVirtual reality measures in neuropsychological assessment: a meta-analytic review
Neguț, A., Matu, S.-A., Sava, F. A., & David, D. (2016) · The Clinical Neuropsychologist · doi:10.1080/13854046.2016.1144793 · Cited by 199
Nalivaiko, E., Davis, S. L., Blackmore, K. L., Vakulin, A., & Nesbitt, K. V. (2015) · Physiology & Behavior · doi:10.1016/j.physbeh.2015.08.043 · Cited by 195 · cited 2×
“Cybersickness (ie, dizziness and vertigo) is akin to motion sickness but occurs in response to exposure to VR [47].”
BackgroundInfluentialResearch Domain Criteria (RDoC): Progress and Potential
Cuthbert, B. N. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214211051363 · Cited by 194 · cited 1×
“The RDoC was originally created to consolidate the research conducted in various fields of mental health [86].”
MethodPrefrontal cortex dysfunction and ‘Jumping to Conclusions’: Bias or deficit?
Lunt, L., Bramham, J., Morris, R. G., Bullock, P. R., Selway, R. P., Xenitidis, K., & David, A. S. (2012) · Journal of Neuropsychology · doi:10.1111/j.1748-6653.2011.02005.x · Cited by 181 · cited 2×
“In line with the literature, EF was defined as a set of executive functions, including inhibition, cognitive flexibility, and working memory [2,4,6], that support other higher-order executive functions, such as reasoning, planning, and problem-solving [6,8].”
BackgroundMethodDecoding fMRI brain states in real-time
LaConte, S. M. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.052 · Cited by 175 · cited 1×
“Similar to the gaming industry, VR paradigms could be adaptive and performance driven so that the level of challenge adjusts according to real-time individual responses [100,101].”
BackgroundFleeting reliability in the dot-probe task
Chapman, A., Devue, C., & Grimshaw, G. M. (2019) · Psychological Research · doi:10.1007/s00426-017-0947-6 · Cited by 172 · cited 1×
“Furthermore, the DPT, which is typically used to assess selective attention [88], was used to assess inhibition, although its own psychometric properties have been the subject of controversy [89,90].”
MethodThe effects of gamification on computerized cognitive training: Systematic review and meta-analysis
Vermeir, J. F., White, M. J., Johnson, D., Crombez, G., & Van Ryckeghem, D. M. L. (2020) · JMIR Serious Games · doi:10.2196/18644 · Cited by 172 · cited 1×
““Serious games,” defined as games intended for a variety of serious purposes, such as training, learning, stimulation, or cognitive assessment [42], have been shown to be more engaging than nongamified tasks [43-45].”
BackgroundVideo Games for Neuro-Cognitive Optimization
Mishra, J., Anguera, J. A., & Gazzaley, A. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.010 · Cited by 161 · cited 1×
“Similar to the gaming industry, VR paradigms could be adaptive and performance driven so that the level of challenge adjusts according to real-time individual responses [100,101].”
BackgroundCogné, M., Taillade, M., N’Kaoua, B., Tarruella, A., Klinger, E., Larrue, F., Sauzéon, H., Joseph, P.-A., & Sorita, E. (2017) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2015.12.004 · Cited by 150 · cited 1×
“The environment could also be systematically manipulated to match the needs of the assessment [104] and tailored to specific populations.”
BackgroundDAWSON, D. R., Anderson, N. D., Burgess, P. W., Cooper, E., Krpan, K. M., & Stuss, D. T. (2009) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2009.07.012 · Cited by 135 · cited 1×
“Empirical assessment of the MET has demonstrated its generalizability to daily functioning [32] and carer reports of daily functioning [33].”
BackgroundAvailable Virtual Reality-Based Tools for Executive Functions: A Systematic Review
Borgnis, F., Baglio, F., Pedroli, E., Rossetto, F., Uccellatore, L., Oliveira, J., Riva, G., & Cipresso, P. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.833136 · Cited by 112 · cited 2×
“The works by Negu et al [49] and Borgnis et al [50] highlight the utility of VR assessment tools to detect dysfunction and present the various tools in the literature created to investigate EF.”
BackgroundMethodCipresso, P., Albani, G., Serino, S., Pedroli, E., Pallavicini, F., Mauro, A., & Riva, G. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00405 · Cited by 111
Staged decline of visual processing capacity in mild cognitive impairment and Alzheimer's disease
Bublak, P., Redel, P., Sorg, C., Kurz, A., Förstl, H., Müller, H. J., Schneider, W. X., & Finke, K. (2009) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2009.07.012 · Cited by 110 · cited 1×
“For example, sensitive detection can identify the prodromal stages of Alzheimer disease in seemingly healthy individuals [26] and mild cognitive decline up to 12 years before clinical diagnosis [27].”
BackgroundKourtesis, P., Collina, S., Doumas, L. A. A., & MacPherson, S. E. (2021) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617720000764 · Cited by 107 · cited 8×
“Both the studies by Kourtesis et al [76] and Kourtesis and MacPherson [75] used the same VR environment based on a variety of everyday tasks.”
BackgroundMethodInfluentialEnhanced Attention Using Head-mounted Virtual Reality
Li, G., Anguera, J. A., Javed, S., Khan, M. A., Wang, G., & Gazzaley, A. H. (2020) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01560 · Cited by 105 · cited 2×
“Notably, recent studies using electroencephalography (EEG)-based metrics have shown greater attention elicited in immersive VR paradigms than in 2D computerized assessments [46].”
BackgroundEyetracking Metrics in Young Onset Alzheimer’s Disease: A Window into Cognitive Visual Functions
Pavisic, I. M., Firth, N. C., Parsons, S., Rego, D. M., Shakespeare, T. J., Yong, K. X. X., Slattery, C. F., Paterson, R. W., Foulkes, A. J. M., Macpherson, K., et al. (2017) · Frontiers in Neurology · doi:10.3389/fneur.2017.00377 · Cited by 96 · cited 1×
“For example, pupillometry can offer insights into brain injury prognosis [94] and differentiate between participants with Alzheimer disease and healthy controls [95].”
BackgroundHow effective is the Trail Making Test (Parts A and B) in identifying cognitively impaired drivers?
Dobbs, B., & Shergill, S. S. (2013) · Age and Ageing · doi:10.1093/ageing/aft073 · Cited by 89
Ranchet, M., Morgan, J. C., Akinwuntan, A., & Devos, H. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.07.001 · Cited by 85 · cited 2×
“EEG, pupillometry, and cardiovascular measures are also sensitive to cognitive load capacities [99].”
BackgroundRiva, G., Gaggioli, A., Grassi, A., Raspelli, S., Cipresso, P., Pallavicini, F., Cinzia, V., Andrea, G., Stefano, G., & Donvito, G. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-706-2-493 · Cited by 80 · cited 1×
“Several studies [37-39] have investigated and developed platforms for this purpose, commonly known as the virtual MET.”
BackgroundBimodal Virtual Reality Stroop for Assessing Distractor Inhibition in Autism Spectrum Disorders
Parsons, T. D., & Carlew, A. R. (2015) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-015-2663-7 · Cited by 79 · cited 2×
“[72,73] set out to assess multiple constructs.”
BackgroundVirtual reality for psychological assessment in clinical practice.
Parsons, T. D., & Phillips, A. S. (2016) · Practice Innovations · doi:10.1037/pri0000028 · Cited by 76 · cited 1×
“There is an increasing emphasis on conducting EF assessments using tasks that resemble situations experienced in everyday life [30].”
BackgroundIndividual Differences in Inhibitory Control: A latent Variable Analysis
Gärtner, A., & Strobel, A. (2021) · Journal of Cognition · doi:10.5334/joc.150 · Cited by 75 · cited 1×
“A related methodological issue, known as the “task impurity problem” [4,20], indicates that the score on an EF task usually reflects not only the systematic variance attributable to the specific aspect of EF targeted by that task but also the (1) systematic variance across multiple types of EF…”
BackgroundThe impact of task workload on cybersickness
Sepich, N. C., Jasper, A., Fieffer, S. J., Gilbert, S. B., Dorneich, M. C., & Kelly, J. W. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.943409 · Cited by 73 · cited 1×
“Similarly, Sepich et al [48] found that participants’ accuracy on n-back task performance was weakly to moderately negatively correlated ( r =−0.32; P =.002) with subjective cybersickness ratings.”
BackgroundInfluentialThe Clinical Utility of the Trail Making Tests As Predictors of Driving Performance in Older Adults
Papandonatos, G., Ott, B. L., Davis, J. D., Barco, P., & Carr, D. (2015) · Journal of The American Geriatrics Society · doi:10.1111/jgs.13776 · Cited by 71 · cited 1×
“Although some studies have demonstrated a relationship between the two [21,22], others have shown no relationship [23], particularly in nonclinical populations [24,25].”
ResultThe Cambridge Prospective Memory Test (CAMPROMPT)
Thöne-Otto, A. I. T., & Streubel, B. (2006) · Zeitschrift für Neuropsychologie · doi:10.1024/1016-264x.17.3.201 · Cited by 68
Learner immersion engagement in the 3D virtual world: principles emerging from the DELVE project
Mount, N. J., Chambers, C., Weaver, D., & Priestnall, G. (2009) · Innovation in Teaching and Learning in Information and Computer Sciences · doi:10.11120/ital.2009.08030040 · Cited by 65 · cited 1×
“VR has the potential to enhance individual engagement more effectively than traditional pencil-and-paper or computerized tasks by offering a fully immersive experience [40].”
BackgroundMiskowiak, K. W., Jespersen, A. E., Kessing, L. V., Aggestrup, A. S., Glenthøj, L. B., Nordentoft, M., Ott, C. V., & Lumbye, A. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.12.002 · Cited by 64 · cited 5×
“Miskowiak et al [63] assessed executive functions by letting participants complete the TMT-B, One Touch Stockings of Cambridge mean choices to correct, and verbal fluency test versions S and D.”
BackgroundMethodInfluentialExploring the contribution of spatial navigation to cognitive functioning in older adults
Laczó, J., Andel, R., Nedelská, Z., Vyhnálek, M., Vlček, K., Crutch, S. J., Harrison, J., & Hort, J. (2016) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2016.12.003 · Cited by 60 · cited 1×
“For example, spatial navigation is a cognitive marker used to detect early attention deficit [102,103] and offers additional relevant information beyond the traditional neuropsychological tests [103].”
BackgroundVaucher, P., Herzig, D., Cardoso, I., Herzog, M. H., Mangin, P. J., & Favrat, B. (2014) · BMC Geriatrics · doi:10.1186/1471-2318-14-123 · Cited by 58 · cited 1×
“Although some studies have demonstrated a relationship between the two [21,22], others have shown no relationship [23], particularly in nonclinical populations [24,25].”
ResultDavison, S. M. C., Deeprose, C., & Terbeck, S. (2017) · Acta Neuropsychiatrica · doi:10.1017/neu.2017.14 · Cited by 56 · cited 6×
“[61,62,68,69] reported only statistically significant correlations, leaving unanswered questions because of the omission of nonsignificant results.”
BackgroundMethodResultInfluentialBenke, T., Karner, E., Petermichl, S., Prantner, V., & Kemmler, G. (2013) · Alzheimer Disease & Associated Disorders · doi:10.1097/wad.0000000000000009 · Cited by 55 · cited 1×
“For example, spatial navigation is a cognitive marker used to detect early attention deficit [102,103] and offers additional relevant information beyond the traditional neuropsychological tests [103].”
BackgroundRobitaille, N., Jackson, P. L., Hébert, L. J., Mercier, C., Bouyer, L. J., Fecteau, S., Richards, C. L., & McFadyen, B. J. (2016) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2016.1229048 · Cited by 55 · cited 3×
“Working memory was investigated in 21% (4/19) of the studies [63,65,70,74].”
BackgroundMethodInfluentialVoinescu, A., Petrini, K., Fraser, D. S., Lazarovicz, R.-A., Papavă, I., Fodor, L. A., & David, D. (2021) · Virtual Reality · doi:10.1007/s10055-021-00520-7 · Cited by 54 · cited 3×
“Voinescu et al [71] immersed participants in a virtual aquarium where they had to perform a variety of tasks.”
BackgroundMethodInfluentialTsai, C.-F., Chen, C.-C., Wu, E. H.-K., Chung, C.-R., Huang, C.-Y., Tsai, P.-Y., & Yeh, S.-C. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3118918 · Cited by 49 · cited 2×
“Tsai et al [67] immersed 2 participant groups in a virtual shopping environment: one group with mild cognitive impairment (MCI) and one control group.”
BackgroundAgeing and spatial reversal learning in humans: Findings from a virtual water maze
Schoenfeld, R., Foreman, N. K., & Leplow, B. (2014) · Behavioural Brain Research · doi:10.1016/j.bbr.2014.04.036 · Cited by 48
Pöhlmann, K. M. T., Li, G., McGill, M., Markoff, R., & Brewster, S. A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580966 · Cited by 42 · cited 1×
“This is concerning as cybersickness presents a substantial confound for valid VR assessment and has been shown to negatively affect task performance [92,93].”
BackgroundAn initial validation of the Virtual Environment Grocery Store
Parsons, T. D., & McMahan, T. (2017) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2017.07.027 · Cited by 41 · cited 4×
“…immersion and XSL • FO RenderX engagement details, and comparative measures for validation purposes were extracted (Multimedia Appendix 1 [53-55,61-76 A qualitative evaluation of the studies included in the review was performed, meaning that the content of each manuscript was assessed…”
BackgroundMethodInfluentialIliadou, P., Paliokas, I., Zygouris, S., Lazarou, E., Votis, K., Tzovaras, D. K., & Tsolaki, M. (2021) · Journal of Alzheimer s Disease · doi:10.3233/jad-201300 · Cited by 41 · cited 1×
“The combination of these biosensor metrics and real-time function-led VR performance could increase the sensitivity of tests, enabling the detection of subtle differences such as between MCI and subjective memory complaints [98].”
BackgroundKourtesis, P., & MacPherson, S. E. (2023) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2021.2008983 · Cited by 41 · cited 3×
“In total, 26% (5/19) of the studies [62,68,69,75,76] identified planning as a target construct in their VR paradigms.”
BackgroundMethodFernández, G., Manes, F. F., Politi, L. E., Orozco, D., Schumacher, M., Castro, L., Agamennoni, O. E., & Rotstein, N. P. (2016) · Journal of Alzheimer s Disease · doi:10.3233/jad-150265 · Cited by 40 · cited 1×
“Eye tracking during reading aids Alzheimer disease identification [96], and linguistic attributes (eg, formation and fluency of sentences, syntax, and grammar) distinguish patients with Alzheimer disease from those with MCI [97].”
BackgroundA Novel Virtual Reality Assessment of Functional Cognition: Validation Study
Porffy, L., Mehta, M. A., Patchitt, J., Boussebaa, C., Brett, J., D’Oliveira, T. C., Mouchlianitis, E. D., & Shergill, S. S. (2021) · Journal of Medical Internet Research · doi:10.2196/27641 · Cited by 40 · cited 5×
“Porffy et al [65] asked participants to operate a virtual store in which the working memory component was assessed at the “Pay” step, where participants had to select and pay for their items at a self-checkout machine providing the exact amount.”
BackgroundMethodInfluentialDesign Considerations for Supporting Mindfulness in Virtual Reality
Kelly, R., Seabrook, E., Foley, F., Thomas, N., Nedeljkovic, M., & Wadley, G. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2021.672556 · Cited by 38
Kim, E., Han, J., Choi, H., Prié, Y., Vigier, T., Bulteau, S., & Kwon, G. H. (2021) · JMIR Serious Games · doi:10.2196/30249 · Cited by 36 · cited 1×
“Kim et al [51] provided an overview of the research trends using VR for neuropsychological tests and documented the cognitive functions assessed in each study.”
Current and future applications of virtual reality technology for cardiac interventions
Mahtab, E. A. F., & Egorova, A. D. (2022) · Nature Reviews Cardiology · doi:10.1038/s41569-022-00789-4 · Cited by 35 · cited 1×
““Serious games,” defined as games intended for a variety of serious purposes, such as training, learning, stimulation, or cognitive assessment [42], have been shown to be more engaging than nongamified tasks [43-45].”
BackgroundGiglioli, I. A. C., de-Juan-Ripoll, C., Parra, E., & Alcañíz, M. (2019) · Applied Neuropsychology Adult · doi:10.1080/23279095.2019.1607735 · Cited by 34
Tan, N. C., Lim, J. E., Allen, J. C., Wong, W. T., Shen, M. L. W., Quah, J. H. M., Muthulakshmi, P., Teh, T. A., Lim, S. H., & Malhotra, R. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.847590 · Cited by 33 · cited 2×
“%) reported on verbal learning [63], associate learning pattern recognition [65], perceptual motor, social cognition, language [66], and calculation [67].”
BackgroundVirtual Apartment-Based Stroop for assessing distractor inhibition in healthy aging
Parsons, T. D., & Barnett, M. D. (2017) · Applied Neuropsychology Adult · doi:10.1080/23279095.2017.1373281 · Cited by 30 · cited 2×
“Similarly, Parsons and Barnett [73] had participants perform the ST in a virtual apartment as well as complete a computerized and paper-and-pencil version of the task.”
BackgroundRemote virtual reality as a tool for increasing external validity
Draschkow, D. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-022-00082-8 · Cited by 30 · cited 1×
“VR can mitigate these difficulties by maintaining the real-world environment without requiring travel while enabling fine-tuned control and uniform presentation of environmental characteristics [36].”
BackgroundMaeir, A., Krauss, S., & Katz, N. (2011) · OTJR: Occupational Therapy Journal of Research · doi:10.3928/15394492-20101108-07 · Cited by 29 · cited 1×
“Empirical assessment of the MET has demonstrated its generalizability to daily functioning [32] and carer reports of daily functioning [33].”
BackgroundGiglioli, I. A. C., Gálvez, B. P., Granados, A. G., & Alcañíz, M. (2021) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2020.0560 · Cited by 29
Impact of Catastrophic Health Expenditures on Chinese Household Consumption
Wei, N., Huang, W., & Zhou, L. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.646494 · Cited by 25
Cut-off point for the trail making test to predict unsafe driving after stroke
Choi, S.-Y., Lee, J., & Oh, Y.-J. (2016) · Journal of Physical Therapy Science · doi:10.1589/jpts.28.2110 · Cited by 22 · cited 1×
“Although some studies have demonstrated a relationship between the two [21,22], others have shown no relationship [23], particularly in nonclinical populations [24,25].”
ResultErrors versus speed on the trail making test: Relevance to driving performance
Duncanson, H., Hollis, A. M., & O’Connor, M. G. (2018) · Accident Analysis & Prevention · doi:10.1016/j.aap.2018.01.004 · Cited by 22 · cited 1×
“Although some studies have demonstrated a relationship between the two [21,22], others have shown no relationship [23], particularly in nonclinical populations [24,25].”
ResultVandormael, A. M., Adam, M., Greuel, M., & Bärnighausen, T. (2020) · Trials · doi:10.1186/s13063-020-04449-1 · Cited by 20
Williams, A., Alikhademi, K., & Gilbert, J. E. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102734 · Cited by 18 · cited 4×
“…immersion and XSL • FO RenderX engagement details, and comparative measures for validation purposes were extracted (Multimedia Appendix 1 [53-55,61-76 A qualitative evaluation of the studies included in the review was performed, meaning that the content of each manuscript was assessed…”
BackgroundMethodInfluentialNalder, E. J., Clark, A. J., Anderson, N. D., & DAWSON, D. R. (2015) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2015.1067628 · Cited by 17 · cited 1×
“However, given that the MET is designed to be performed in real-life locations, it is impractical for routine administration by clinicians [34,35] and susceptible to the variable features of real-world environments that are outside experimental control.”
BackgroundVirtual Multiple Errands Test: Reliability, Usability and Possible Applications
Pedroli, E., Cipresso, P., Serino, S., Pallavicini, F., Albani, G., & Riva, G. (2013) · Studies in health technology and informatics · doi:10.3233/978-1-61499-282-0-38 · Cited by 15 · cited 1×
“Several studies [37-39] have investigated and developed platforms for this purpose, commonly known as the virtual MET.”
BackgroundPallavicini, F., Pepe, A., & Minissi, M. E. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23563-5_12 · Cited by 15 · cited 1×
“Pallavicini et al [64] had participants play the Audioshield dance game, which the authors hypothesized could be closely related to EF constructs such as inhibition and working memory.”
BackgroundDeveloping and Validating a Big-Store Multiple Errands Test
Antoniak, K., Clores, J., Jensen, D., Nalder, E. J., Rotenberg, S., & DAWSON, D. R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02575 · Cited by 15 · cited 1×
“However, given that the MET is designed to be performed in real-life locations, it is impractical for routine administration by clinicians [34,35] and susceptible to the variable features of real-world environments that are outside experimental control.”
BackgroundAnnual Update in Intensive Care and Emergency Medicine 2019
(2019) · Annual update in intensive care and emergency medicine · doi:10.1007/978-3-030-06067-1 · Cited by 13
Playing the mirror game in virtual reality with an autonomous character
Llobera, J., Jacquat, V., Calabrese, C., & Charbonnier, C. (2022) · Scientific Reports · doi:10.1038/s41598-022-25197-z · Cited by 11
Baños, J. H. (2002) · Archives of Clinical Neuropsychology · doi:10.1016/s0887-6177(01)00125-1 · Cited by 11
Applying machine learning to a virtual serious game for neuropsychological assessment
Marín‐Morales, J., Carrasco-Ribelles, L. A., Alcañíz, M., & Giglioli, I. A. C. (2021) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon46332.2021.9454138 · Cited by 6 · cited 4×
“Working memory was investigated in 21% (4/19) of the studies [63,65,70,74].”
BackgroundMethodInfluentialValidation of a Sorting Task Implemented in the Virtual Multitasking Task-2 and Effect of Aging
Banville, F., Lussier, C., Massicotte, E., Verhulst, E., Couture, J.-F., Allain, P., & Richard, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-92046-7_4 · Cited by 4 · cited 6×
“[61,62,68,69] reported only statistically significant correlations, leaving unanswered questions because of the omission of nonsignificant results.”
BackgroundMethodResultInfluentialWilf, M., Yair, N. B., Wright, W. G., & Plotnik, M. (2022) · Applied Neuropsychology Adult · doi:10.1080/23279095.2022.2065204 · Cited by 3 · cited 1×
“Sequentially, many VR tasks are being created from a “function-led” perspective but validated against “construct-led” tasks [53,54].”
Background
Research works printed but not linked: 6
- Kvavilashvili, L, 2004, History Philos Psychol, 6, 59
- Pedroli, E, 2013, Stud Health Technol Inform, 191, 38
- Riva, G, 2011, Stud Health Technol Inform, 163, 493
- Delis, DC, 2000, California Verbal Learning Test-Second Edition (CVLT-II)
- Wilson, BA, 2008, Rivermead Behavioural Memory Test - Third Edition
- D'Elia, LF, 1996, Color Trails Test
Cited by
Jespersen, A. E., Lumbye, A., Schandorff, J. M., Damgaard, V., Glenthøj, L. B., Nordentoft, M., Mikkelsen, C., Didriksen, M., Ostrowski, S. R., Vinberg, M., et al. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.10.095 · Cited by 19
“…consider verisimilitude , i.e., designing simulations representative of daily-life situations, they often neglect the veridicality of the results, i.e., the investigation of how well the tasks predict real-world performance in these situations (Franzen and Wilhelm, 1996; Kirkham et al., 2024).”
Caporusso, E., Melillo, A., Perrottelli, A., Giuliani, L., Marzocchi, F. F., Pezzella, P., & Giordano, G. M. (2025) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2025.1543005 · Cited by 21
Romero-Ayuso, D., del Pino-González, A., Torres-Jiménez, A., Juan-González, J., Celdrán, F. J., Franchella, M. C., Ortega-López, N., Triviño-Juárez, J. M., Garach-Gómez, A., Arrabal-Fernández, L., et al. (2024) · Children · doi:10.3390/children11080986 · Cited by 20
“Recent advancements in virtual reality (VR) technology have provided promising avenues for creating immersive digital environments that simulate real-life scenarios [5].”
BackgroundHoran, W. P., Moore, R. C., Belanger, H. G., & Harvey, P. D. (2024) · Schizophrenia Bulletin Open · doi:10.1093/schizbullopen/sgae025 · Cited by 17
Llana, T., Garcés-Arilla, S., García-Navarra, S., Méndez-López, M., Juan, M. ‐., & Méndez, M. (2025) · Virtual Reality · doi:10.1007/s10055-025-01244-8 · Cited by 7
“One of the most sig - nificant obstacles to the widespread adoption of VR is cybersickness, a common human factor that may occur in virtual environments (VEs), which can reduce the validity of assessments (Kirkham et al. 2024; Rebenitsch and Owen 2016 ).”
Metadata
- Authors
- Kirkham, R., Kooijman, L., Albertella, L., Myles, D., Yücel, M., & Rotaru, K.
- Affiliations
- Monash University (AU)
- Deakin University (AU)
- QIMR Berghofer Medical Research Institute (AU)
- Corresponding author
- Kirkham, Rebecca
- Venue
- JMIR Serious Games
- Publisher
- JMIR Publications Inc.
- Published
- 2024-02-26
- Volume
- 12
- Pages
- e50282
- ISSN
- 2291-9279 (ISSN-L)
- PMC
- PMC10928525
- OpenAlex
- W4390419184
- Cited by
- 40
- References
- 108
- Citations by year
- 2026: 17 · 2025: 16 · 2024: 7
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Serious Games (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (acceptedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Wilson Foundation
- Department of Science and Technology, Ministry of Science and Technology, India
- Universiteit van Amsterdam
- Department of Industry, Innovation and Science, Australian Government
- National Institutes of Health
- National Health and Medical Research Council
- Defence Science and Technology Group
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- executive function, immersive virtual reality, neuropsychological assessment, cybersickness, ecological validity, test validation, inhibitory control, cognitive flexibility, working memory, planning, attention, user experience, head-mounted displays, psychometric properties, healthy adults, clinical populations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
