Development and validation of a simulation workload measure: the simulation task load index (SIM-TLX)
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Virtual reality (VR) simulation offers significant potential for human factors training as it provides a novel approach which enables training in environments that are otherwise dangerous, impractical or expensive to simulate. While VR training has been adopted in many environments, such as heavy industry, surgery and aviation, there remains an inadequate understanding of how virtual simulations impact cognitive factors. One such factor, which needs careful consideration during the design of VR simulations, is the degree of mental or cognitive load experienced during training. This study aimed to validate a newly developed measure of workload, based on existing instruments (e.g. the NASA-TLX), but tailored to the specific demands placed on users of simulated environments. While participants completed a VR puzzle game, a series of experimental manipulations of workload were used to assess the sensitivity of the new instrument. The manipulations affected the questionnaire subscales (mental demands; physical demands; temporal demands; frustration; task complexity; situational stress; distraction; perceptual strain; task control; presence) as predicted in all cases (p s < .05), except for presence, which displayed little relationship with other aspects of task load. The scale was also found to have good convergent validity with an alternate index of task load. The findings support the sensitivity of the new instrument for assessing task load in virtual reality. Overall, this study contributes to the understanding of mental workload in simulated environments and provides a practical tool for use in both future research and applications in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Three papers referred to Witmer and Singer's presence questionnaire [34], and three followed the Igroup presence questionnaire (IPQ) [35], Nichols et al.'s measurements of presence [75] and an adapted form of the immersive tendencies questionnaire (ITQ) [34], one for each paper.”
From A systematic review of immersive virtual reality for industrial skills training (2021) · cites this paper as [35]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“User workload in a simulation environment was evaluated using the SIM-TLX developed by Harris et al. [19] and translated by Jeong et al. [30].”
From Development and pilot evaluation of an AI-based pediatric medication safety virtual simulation program for nursing students: a quasi-experimental study in South Korea (2026) · cites this paper as Harris et al. [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
mice: Multivariate Imputation by Chained Equations in R
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · cited 1×
“Missing pairwise weighting data for 1 participant was imputed based on predictive mean matching, using the R package ‘mice’ (van Buuren and Groothuis-Oudshoorn 2011).”
MethodMcAuley, E., Duncan, T., & Tammen, V. V. (1989) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1989.10607413 · Cited by 2,694
Burdea, G., & Coiffet, P. (2003) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474603322955950 · Cited by 2,315
Guadagnoli, M. A., & Lee, T. D. (2004) · Journal of Motor Behavior · doi:10.3200/jmbr.36.2.212-224 · Cited by 1,534
Using Presence Questionnaires in Reality
Usoh, M., Catena, E., Arman, S., & Slater, M. (2000) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474600566989 · Cited by 980
(1999) · Cited by 944
Cognitive load theory: implications of cognitive load theory on the design of learning
Kirschner, P. A. (2002) · Learning and Instruction · doi:10.1016/s0959-4752(01)00014-7 · Cited by 917 · cited 1×
“As cognitive load plays an important role in informational learning and skill acquisition (Kirschner 2002; Renkl and Atkinson 2003; van Gog et al.”
BackgroundDirect Measurement of Cognitive Load in Multimedia Learning
Brunken, R., Plass, J. L., & Leutner, D. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_7 · Cited by 896
Shin, D. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2017.09.012 · Cited by 738
Assimilating complex information
Pollock, E., Chandler, P., & Sweller, J. (2002) · Learning and Instruction · doi:10.1016/s0959-4752(01)00016-0 · Cited by 692
Renkl, A., & Atkinson, R. K. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_3 · Cited by 540
Surgical Simulation: A Systematic Review
Sutherland, L. M., Middleton, P. F., Anthony, A., Hamdorf, J., Cregan, P., Scott, D., & Maddern, G. J. (2006) · Annals of Surgery · doi:10.1097/01.sla.0000200839.93965.26 · Cited by 536
Sensorimotor training in virtual reality: A review
Adamovich, S. V., Fluet, G. G., Tunik, E., & Merians, A. S. (2009) · NeuroRehabilitation · doi:10.3233/nre-2009-0497 · Cited by 507
Virtual reality exposure therapy of anxiety disorders: A review
Krijn, M., Emmelkamp, P. M. G., Olafsson, R. P., & Biemond, R. (2004) · Clinical Psychology Review · doi:10.1016/j.cpr.2004.04.001 · Cited by 492
Development and Validation of a Surgical Workload Measure: The Surgery Task Load Index (SURG‐TLX)
Wilson, M. R., Poolton, J. M., Malhotra, N., Ngo, K., Bright, E., & Masters, R. S. W. (2011) · World Journal of Surgery · doi:10.1007/s00268-011-1141-4 · Cited by 414
Self-citationTraining in virtual environments: transfer to real world tasks and equivalence to real task training
Rose, F. D., Attree, E. A., Brooks, B. M., Parslow, D. M., & Penn, P. R. (2000) · Ergonomics · doi:10.1080/001401300184378 · Cited by 372
Efficiency in work behaviour: A design approach for modern tools
Zijlstra, F. (1993) · Research Repository (Delft University of Technology) · Cited by 356
A systematic review of the application of interactive virtual reality to sport
Neumann, D. L., Moffitt, R. L., Thomas, P. R., Loveday, K., Watling, D. P., Lombard, C. L., Antonova, S., & Tremeer, M. A. (2018) · Virtual Reality · doi:10.1007/s10055-017-0320-5 · Cited by 351
Frederiksen, J. G., Sørensen, S. M. D., Konge, L., Svendsen, M. B. S., Nobel-Jørgensen, M., Bjerrum, F., & Andersen, S. A. W. (2020) · Surgical Endoscopy · doi:10.1007/s00464-019-06887-8 · Cited by 294
A method to assess the driver mental workload: The driving activity load index (DALI)
Pauzié, A. (2008) · IET Intelligent Transport Systems · doi:10.1049/iet-its:20080023 · Cited by 246
van Gog, T., Ericsson, K. A., Rikers, R. M. J. P., & Paas, F. (2005) · Educational Technology Research and Development · doi:10.1007/bf02504799 · Cited by 209
The impact of a self-avatar on cognitive load in immersive virtual reality
Steed, A., Pan, Y., Zisch, F., & Steptoe, W. (2016) · 2016 IEEE Virtual Reality (VR) · doi:10.1109/vr.2016.7504689 · Cited by 187
Stanney, K. M., Mollaghasemi, M., Reeves, L., Breaux, R., & Graeber, D. A. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00015-6 · Cited by 184
Transfer of Training from Virtual to Real Baseball Batting
Gray, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.02183 · Cited by 180
Interactive effects of physical and mental workload on subjective workload assessment
DiDomenico, A., & Nussbaum, M. A. (2008) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2008.01.012 · Cited by 178
Wickens, C. D. (2003) · [Proceedings] 1992 IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1992.271688 · Cited by 171
Limitations of subjective cognitive load measures in simulation‐based procedural training
Naismith, L. M., Cheung, J. J. H., Ringsted, C., & Cavalcanti, R. B. (2015) · Medical Education · doi:10.1111/medu.12732 · Cited by 167
Estimating VR Sickness and user experience using different HMD technologies: An evaluation study
Somrak, A., Humar, I., Hossain, M. S., Alhamid, M. F., Hossain, M. A., & Guna, J. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.11.041 · Cited by 166
The Relationship between Presence and Performance in Virtual Simulation Training
Stevens, J. A., & Kincaid, J. P. (2015) · Open Journal of Modelling and Simulation · doi:10.4236/ojmsi.2015.32005 · Cited by 138
The stress and workload of virtual reality training: the effects of presence, immersion and flow
Lackey, S. J., Salcedo, J. N., Szalma, J. L., & Hancock, P. A. (2016) · Ergonomics · doi:10.1080/00140139.2015.1122234 · Cited by 118
Estimating Mental Effort Using Heart Rate and Heart Rate Variability
Mulder, L. (., de Waard, D., & Brookhuis, K. (2004) · CRC Press eBooks · doi:10.1201/9780203489925-30 · Cited by 94
Bharathan, R., Vali, S., Setchell, T. E., Miskry, T. S., Darzi, A. W., & Aggarwal, R. (2013) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2013.03.017 · Cited by 56
Hashimoto, D. A., Petrusa, E., Phitayakorn, R., Valle, C., Casey, B., & Gee, D. (2018) · Surgical Endoscopy · doi:10.1007/s00464-017-5821-5 · Cited by 46
Challenges in evaluating skill transfer from construction equipment simulators
Dunston, P. S., Proctor, R. W., & Wang, X. (2011) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2011.624647 · Cited by 32
Research works printed but not linked: 2
- Bates D, Mächler M, Bolker B, Walker S (2014) Fitting linear mixed-effects models using lme4. Retrieved from http://arxiv.org/abs/1406.5823 · arXiv
- Mestre D, Fuchs P, Berthoz A, Vercher JL (2006) Immersion et présence. In: Fuchs P, Moreau G, Berthoz A, Vercher JL (eds) Le Traite de la realite Virtuelle. Ecole Des Mines de Paris, Paris, pp 309–338
Cited by
Skulmowski, A., & Xu, K. M. (2022) · Educational Psychology Review · doi:10.1007/s10648-021-09624-7 · Cited by 678
Anwar, M. S., Yang, J. C., Frnda, J., Choi, A., Baghaei, N., & Ali, M. (2025) · Virtual Reality · doi:10.1007/s10055-025-01126-z · Cited by 204
A systematic review of immersive virtual reality for industrial skills training
Radhakrishnan, U., Koumaditis, K., & Chinello, F. (2021) · Behaviour & Information Technology · doi:10.1080/0144929x.2021.1954693 · Cited by 200
Vlahović, S., Sužnjević, M., & Skorin‐Kapov, L. (2022) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-022-00388-0 · Cited by 142
Learning in immersed collaborative virtual environments: design and implementation
de Back, T. T., Tinga, A. M., & Louwerse, M. M. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2021.2006238 · Cited by 134
Metadata
- Authors
- Harris, D., Wilson, M., & Vine, S.
- Affiliations
- University of Exeter (GB)
- Corresponding author
- Harris, David
- Venue
- Virtual Reality
- Published
- 2019-12-21
- Volume
- 24
- Issue
- 4
- Pages
- 557-566
- ISSN
- 1359-4338 (ISSN-L)
- 1434-9957
- OpenAlex
- W2994780247
- Cited by
- 146
- References
- 36
- Citations by year
- 2026: 18 · 2025: 40 · 2024: 25 · 2023: 24 · 2022: 21 · 2021: 14 · 2020: 4
- FWCI
- 9.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Virtual Reality (publishedVersion, cc-by)
- Figshare (submittedVersion, cc-by)
- Open Research Exeter (University of Exeter) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- Royal Academy of Engineering
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Engineering › Control and Systems Engineering › Simulation and Modeling Applications
- Keywords
- virtual reality simulation, mental workload, NASA-TLX, cognitive load, human factors training, scale validation, convergent validity, presence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.