The DOI matches its Crossref record; 266 of the 271 references it cites were found in a registry or an index, and 4 are not checked yet; 16 of the 76 sources its claims cite were checked for support.
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Open accessAbstract
The ever-increasing number of computing devices around us results in more and more systems competing for our attention, making cognitive workload a crucial factor for the user experience of human-computer interfaces. Research in Human-Computer Interaction (HCI) has used various metrics to determine users’ mental demands. However, there needs to be a systematic way to choose an appropriate and effective measure for cognitive workload in experimental setups, posing a challenge to their reproducibility. We present a literature survey of past and current metrics for cognitive workload used throughout HCI literature to address this challenge. By initially exploring what cognitive workload resembles in the HCI context, we derive a categorization supporting researchers and practitioners in selecting cognitive workload metrics for system design and evaluation. We conclude with three following research gaps: (1) defining and interpreting cognitive workload in HCI, (2) the hidden cost of the NASA-TLX, and (3) HCI research as a catalyst for workload-aware systems, highlighting that HCI research has to deepen and conceptualize the understanding of cognitive workload in the context of interactive computing systems.
Verified 2026-10-05 · 4 references not checked yet
Overview
CoChat Summary
TL;DR
This ACM Computing Surveys paper systematically reviews how cognitive workload has been measured in HCI, drawing on a final corpus of 579 records. It categorizes subjective, physiological, behavioral, and performance-based metrics and proposes a step-by-step procedure for selecting suitable measures, while identifying three research gaps around definition, NASA-TLX reliance, and workload-aware systems.
Key findings
- The review’s final corpus contains 579 records: 482 from 26 conferences and 97 from 11 journals, derived from an initial corpus of 6,414 papers (4,408 conference, 2,006 journal).
- Questionnaires are the most used metric, with the NASA-TLX described as the most frequently used questionnaire in the corpus; the paper argues its dominance may reflect community convention and an “academic tradition” rather than demonstrated HCI-specific validity.
- The authors identify three research gaps: (1) defining and interpreting cognitive workload in HCI, (2) the hidden cost of the NASA-TLX, and (3) HCI research as a catalyst for workload-aware systems.
- They propose a four-step procedure for choosing cognitive workload metrics, covering whether cognitive workload is the right concept, whether it varies during use, whether the system must adapt to it, and which modality to choose.
Methods
- Systematic literature review following PRISMA procedures, with a keyword search in HCI-related proceedings and journals; venues included the top 20 conferences plus six conferences in the top 40, and all top 10 journals, confirmed via CORE 4 rankings, with a final cut-off date of 30 April 2020.
- The search used a regular expression matching terms such as “cognitive/mental load/workload” and “working memory”; the initial corpus was 6,414 papers.
- Filtering used paper eligibility (archival, peer-reviewed papers only; 904 excluded) and relevance via a double decision scheme with four researchers; 4,931 papers were excluded at relevance, yielding 579 records. Categorization used qualitative coding, a naming scheme agreed by four researchers, and affinity diagramming to build a taxonomy of measures.
Limitations
- The paper states that its qualitative analysis cannot guarantee the proposed taxonomy is the only correct one, though it aims for comprehensiveness.
- The paper states that the final choice of cognitive workload modality is almost always a trade-off between ecological validity and the requirements of the measurement modality, and that NASA-TLX validity in HCI is unclear because no empirical evidence determines the range of artifacts or experiences to which it applies.
- Evident from the text: the search is limited to selected top HCI venues and a cut-off of 30 April 2020, and many irrelevant papers were excluded due to ambiguity of the term “working memory,” especially in computer science venues.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
6 Valid · 16 Not valid · 23 Not checked · 86 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Previous research has shown that the size and speed of saccades are highly discriminatory parameters for an increased cognitive workload: Here, an increased cognitive workload is indicated by larger or faster saccades [47].”
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.3.2 Saccades.
What the sources say
- Disputedcites [47] 47. Siyuan Chen, Julien Epps, Natalie Ruiz, and Fang Chen. 2011. Eye activity as a measure of human mental effort in HCI. In Proceedings of the 16th International Conference on Intelligent User Interfaces (IUI'11). ACM, New York, NY, USA, 315-318. https://doi.org/10.1145/1943403.1943454 sourceExperimental — accuracy not yet measured
The Discussion says saccade speed and especially size were highly discriminatory, but reports that both decreased as the task demanded more working memory and attention. The cited paper therefore supports their usefulness as measures, but gives the opposite direction from the claim’s ‘larger or faster’ saccades. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Similarly, touch input [39] -especially finger trajectories [173] -were used to measure cognitive workload.”
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.7 Haptic Interaction
What the sources say
- Not checked — no full text of the sourcecites [39] 39. Davide Maria Calandra, Antonio Caso, Francesco Cutugno, Antonio Origlia, and Silvia Rossi. 2013. CoWME: A gen- eral framework to evaluate cognitive workload during multimodal interaction. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI'13). ACM, Sydney, Australia, 111-118. https://doi.org/10. 1145/2522848.2522867 source
- Disputedcites [173] 173. E. M. Peck, E. Carlin, and R. Jacob. 2015. Designing brain-computer interfaces for attention-aware systems. Computer 48, 10 (Oct. 2015), 34-42. https://doi.org/10.1109/MC.2015.315 sourceExperimental — accuracy not yet measured
The citation [173] is attached specifically to “finger trajectories.” In “Training CARSON to Read the Brain” and Figure 3 (page not provided), the paper instead describes predicting cognitive workload from fNIRS brain-sensing signals, not finger trajectories or touch input. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Similarly, beta power is widely used as a metric for mental workload, including alertness, arousal, frustration, engagement, and workload states [141].”
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.2.1 Electroencephalography (EEG).
What the sources say
- Could not verifycites [141] 141. Kristiyan Lukanov, Horia A. Maior, and Max L. Wilson. 2016. Using fNIRS in Usability Testing: Understanding the Effect of Web Form Layout on Mental Workload. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16). ACM, Santa Clara, California, USA, 4011-4016. https://doi.org/10.1145/2858036.2858236 sourceExperimental — accuracy not yet measured
The Measuring Mental Workload and Objective Measures sections discuss psychophysiological measures and fNIRS haemoglobin measurements, but do not address beta power. Searches for beta power and EEG spectral power found no passage linking beta power to alertness, arousal, frustration, engagement, or workload states. The available passages have no page numbers. (LLM, conf=0.76)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Previous research shows that the rate and duration of fixation are indicators of an attention shift due to increasing task complexity [47].”
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.3.4 Fixations.
What the sources say
- Supportedcites [47] 47. Siyuan Chen, Julien Epps, Natalie Ruiz, and Fang Chen. 2011. Eye activity as a measure of human mental effort in HCI. In Proceedings of the 16th International Conference on Intelligent User Interfaces (IUI'11). ACM, New York, NY, USA, 315-318. https://doi.org/10.1145/1943403.1943454 source
The Discussion reports that fixation duration and fixation rate changed with task complexity and interprets those results as indicating that significantly more attention was needed for the more complex task. It also specifies that fixation duration increased while fixation rate decreased as attentional demands rose. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous research has shown that pupils dilate with increasing task difficulty [47] across tasks like reading, problem-solving, or visual search activities [208], where the cognitive demand can be estimated in real-time using machine learning [131].”
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.3.1 Pupil Diameter.
What the sources say
- Partly supportedcites [47] 47. Siyuan Chen, Julien Epps, Natalie Ruiz, and Fang Chen. 2011. Eye activity as a measure of human mental effort in HCI. In Proceedings of the 16th International Conference on Intelligent User Interfaces (IUI'11). ACM, New York, NY, USA, 315-318. https://doi.org/10.1145/1943403.1943454 sourceExperimental — accuracy not yet measured
The Discussion reports increased pupil size when participants performed the more demanding visual training task, supporting the link between pupil dilation and task difficulty. The paper does not establish that pattern across the cited examples of reading, problem-solving, and visual search; its own task involved recalling player positions in a basketball video. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [208] 208. Namrata Srivastava, Eduardo Velloso, Jason M. Lodge, Sarah Erfani, and James Bailey. 2019. Continuous evaluation of video lectures from real-time difficulty self-report. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI'19). Association for Computing Machinery, Glasgow, Scotland, UK, 1-12. https://doi.org/ 10.1145/3290605.3300816 source
- Could not verifycites [131] 131. Moira LeMay and Eric Hird. 1986. Operator work load: When is enough enough? Commun. ACM 29, 7 (July 1986), 638-642. https://doi.org/10.1145/6138.6147 sourceExperimental — accuracy not yet measured
The Results report that time-stress ratings increased with workload, and the introduction lists physiological measures as a category, but neither addresses pupil dilation. Searches for pupil diameter, task difficulty, and pupillary response found no passage supporting the claim attributed to this paper; the task examples are attached to a different citation. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
52 more claims · 4 with their cited sources checked
Create a free account to see all 131 claims and every source verdict.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 49
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.1 Questionnaires
“Questionnaires lack real-time capabilities since they are employed after an experimental condition.”
§ A Survey on Measuring Cognitive Workload in Human-Computer Interaction > 4.1.3 Instantaneous Self-Assessment.
“Participants rate their perceived workload in predefined intervals during the execution of a task -allowing for real-time assessment of cognitive workload.”
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · cited 1×
“The first phase of our review was a systematic keyword search in the digital library databases relevant to the HCI field.”
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 5×
“The NASA Task Load Index (NASA-TLX) [90, 91] is a multi-scale questionnaire to assess participants’ perceived subjective workload.”
InfluentialCognitive Load During Problem Solving: Effects on Learning
Sweller, J. (1988) · Cognitive Science · doi:10.1016/0364-0213(88)90023-7 · Cited by 11,048 · cited 3×
“Sweller developed the cognitive load theory [231–233], a theoretical framework that explains how learning is limited through instructional design.”
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · cited 1×
“Participants rate their perceived work-load in predefined intervals during the execution of a task – allowing for real-time assessment of cognitive workload.”
Cognitive Architecture and Instructional Design
Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998) · Educational Psychology Review · doi:10.1023/a:1022193728205 · Cited by 6,023 · cited 3×
“…workload helps interface designers in two ways: using a constructive approach (e.g., during early design phases), where cognitive workload is measured to improve the design of novel interactive systems, or using a summative approach (e.g., using existing systems) to compare the cognitive…”
(2005) · Cited by 5,977
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 1×
“We followed the PRISMA procedures [121, 174] for systematic reviews and meta-analyses.”
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
NASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297 · cited 3×
“The NASA-TLX [90, 91] questionnaire has become a household name in HCI research, and it is widely taught in HCI education.”
Research works printed but not linked: 9
- 10.5555/2993778.2993801 · doi:10.5555/2993778.2993801
- S. G. Hart, M. E. Childress, and J. R. Hauser. 1982. Individual definitions of the term “workload”. In Eighth Symposium on Psychology in the Department of Defense. 478–485.
- Sandra G. Hart. 1986. Theory and measurement of human workload. Human Productivity Enhancement 1 (1986), 396–456.
- A. V. Korshakov, A. A. Frolov, and P. D. Bobrov. 2010. On-line automatic suppression of artifacts in multi-dimensional signals using ICA. In Abstr. 10th Europ. Conf. on Non-Destructive Testing. 370.
- Thomas Kosch and Lewis L. Chuang. 2019. Investigating the influence of RSVP display parameters on working memory load using electroencephalography. In 2nd International Conference on Neuroadaptive Technology.
- Gerald Matthews, James Szalma, April Rose Panganiban, Catherine Neubauer, and Joel S. Warm. 2013. Profiling task stress with the Dundee Stress State Questionnaire. Psychology of Stress: New Research 1 (2013), 49–90.
- Christopher D. Wickens. 1984. Processing resources in attention. Varieties of Attention (1984).
- 10.5555/2993778.2993814 · doi:10.5555/2993778.2993814
- F. R. H. Zijlstra and L. Van Doorn. 1985. The Construction of a Scale to Measure Perceived Effort. University of Technology.
Cited by
Digital deception: generative artificial intelligence in social engineering and phishing
Schmitt, M., & Flechais, I. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10973-2 · Cited by 198
“The expectation that individuals can scrutinize every email they receive is impractical and counterproductive, especially in today’s fast-paced digital environment, that comes with heavy cognitive workload (Kosch et al. 2023 ).”
Industrial applications of large language models
Raza, M., Jahangir, Z., Riaz, M. B., Saeed, M. J., & Sattar, M. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-98483-1 · Cited by 163
Workplace Well-Being in Industry 5.0: A Worker-Centered Systematic Review
Antonaci, F. G., Olivetti, E. C., Marcolin, F., Jiménez, I. A. C., Eynard, B., Vezzetti, E., & Moos, S. (2024) · Sensors · doi:10.3390/s24175473 · Cited by 92
“In recent years, there has been a proliferation of studies seeking the best physiological parameters to characterize and detect these emotional states [89,90], but most of these have been carried out in the laboratory under controlled conditions.”
BackgroundChiossi, F., Turgut, Y., Welsch, R., & Mayer, S. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3604243 · Cited by 59
“The idea is that physiological arousal can be used as a proxy for cognitive load [21, 64].”
BackgroundChiossi, F., Ou, C., Gerhardt, C., Putze, F., & Mayer, S. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103433 · Cited by 73
Metadata
- Authors
- Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W.
- Affiliations
- Humboldt-Universität zu Berlin (DE)
- German Research Centre for Artificial Intelligence (DE)
- University of Konstanz (DE)
- Ludwig-Maximilians-Universität München (DE)
- Chalmers University of Technology (SE)
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2023-07-13
- Volume
- 55
- Issue
- 13s
- Pages
- 1-39
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4318618520
- Cited by
- 304
- References
- 259
- Citations by year
- 2026: 120 · 2025: 95 · 2024: 64 · 2023: 25
- FWCI
- 89.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Funders
- Deutsche Forschungsgemeinschaft (251654672 – TRR 161 and 414984028 – CRC 1404 FONDA)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- mental workload, workload metrics, NASA-TLX, mental demand, user experience, reproducibility, interactive systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey on Measuring Cognitive Workload in Human-Computer Interaction. https://app.cochat.ai/papers/10.1145/3582272
- The paper
- Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023). A Survey on Measuring Cognitive Workload in Human-Computer Interaction. ACM Computing Surveys. https://doi.org/10.1145/3582272
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.




