Automatic Cognitive Load Detection from Face, Physiology, Task Performance and Fusion During Affective Interference
The DOI matches its Crossref record; 52 of the 53 references it cites were found in a registry or an index.
Abstract
Cognitive load (CL) is experienced during critical tasks and also while engaged emotional states are induced either by the task itself or by extraneous experiences. Emotions irrelevant to the working memory representation may interfere with the processing of relevant tasks and can influence task performance and behavior, making the accurate detection of CL from nonverbal information challenging. This paper investigates automatic CL detection from facial features, physiology and task performance under affective interference. Data were collected from participants (n=20) solving mental arithmetic tasks with emotional stimuli in the background, and a combined classifier was used for detecting CL levels. Results indicate that the face modality for CL detection was more accurate under affective interference, whereas physiology and task performance were more accurate without the affective interference. Multimodal fusion improved detection accuracies, but it was less accurate under affective interferences. More specifically, the accuracy decreased with an increasing intensity of emotional arousal.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Increased cognitive workload can lead to higher and lower respiration rate variability [99].”
From A Survey on Measuring Cognitive Workload in Human-Computer Interaction (2023) · cites this paper as [99]
Citations
Cites
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 4×
“The classifiers were implemented using MATLABArsenal7, a wrapper for the classifiers in Weka (Hall et al., 2009).”
MethodInfluentialRussell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
The relation of strength of stimulus to rapidity of habit formation
Yerkes, R. M., & Dodson, J. D. (1908) · Journal of Comparative Neurology and Psychology · doi:10.1002/cne.920180503 · Cited by 6,911
The magical number 4 in short-term memory: A reconsideration of mental storage capacity
Cowan, N. (2001) · Behavioral and Brain Sciences · doi:10.1017/s0140525x01003922 · Cited by 6,779
The Psychology of Human-Computer Interaction
Card, S. K., Moran, T. P., & Newell, A. (2018) · CRC Press eBooks · doi:10.1201/9780203736166 · Cited by 4,103
(1998) · Cited by 3,476
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
International Affective Picture System (IAPS) : Technical Manual and Affective Ratings
Lang, P. J. (1995) · Medical Entomology and Zoology · Cited by 2,870
Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_8 · Cited by 2,527
Research works printed but not linked: 5
- 2007, Aviation, Space, and Environmental Medicine, 78, 231
- 2012, ACM TRANS INTERACT INTELL SYST, 2, 221
- 2011, NEW PERSPECTIVES ON AFFECT AND LEARNING TECHNOLOGIES, 3, 203
- 2010, Psychophysiology, 48, 323
- 1986, HANDBOOK PERCEPT HUM PERFORM, 2, 42
Cited by
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“…21, 27, 44–47, 76, 87, 99, 165, 184, 193, 228, 236, 258] Eye Blinks [4, 5, 27, 43–47, 76, 87, 99, 175, 182, 184, 236, 268] Fixations [1, 10, 54, 68, 99, 140, 153, 184, 219, 228] Smooth Pursuits and Ocular Movements [133] Cardiovascular Heart Rate [4, 10, 25, 27, 40, 41, 49, 50, 76, 78, 82, 85, 87,…”
InfluentialMultimodal Fusion for Objective Assessment of Cognitive Workload: A Review
Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2939399 · Cited by 184
“The same authors in a subsequent study [85] investigated the impact of affective interference that emerge due to the nature of the task or personal feelings (i.e., emotional arousal, frustration, and confusion) on the performance of single models (based on single modality) and hybrid fusion model…”
BackgroundCognitive Load Measurement in a Virtual Reality-Based Driving System for Autism Intervention
Zhang, L., Wade, J., Bian, D., Fan, J., Swanson, A., Weitlauf, A., Warren, Z., & Sarkar, N. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2582490 · Cited by 141
“Different kernel functions of support vector machine (SVM), including linear kernel [31] and Gaussian kernel [32], were used in cognitive load measurement.”
BackgroundMethodInfluentialBarua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 152
“[98], [100], [104], [116], [125], [127], [185], [203], [237], [319], [357], [386], [402], Image & Numerical [62], [75], [119], [126], [69], [306], [323], [345], [396], [401], Audio & Text & Sensor [375], Audio & Text [178], [276], [369], [382], [383], Text & Signal [109], Text & Numerical [297], [341], Sensor & Signal…”
Inside Out: Detecting Learners’ Confusion to Improve Interactive Digital Learning Environments
Arguel, A., Lockyer, L., Lipp, O. V., Lodge, J. M., & Kennedy, G. E. (2016) · Journal of Educational Computing Research · doi:10.1177/0735633116674732 · Cited by 118
“For this reason, a multimodal approach consisting of the integration of several indicators may be preferred to detect confusion (Hussain, Calvo, & Chen, 2014; Hussain, Monkaresi, & Calvo, 2012).”
Fitbit for learning: Towards capturing the learning experience using wearable sensing
Giannakos, M. N., Sharma, K., Papavlasopoulou, S., Pappas, I. O., & Kostakos, V. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.102384 · Cited by 77
“Such responses have been used to infer various learningrelated constructs, including cognitive load (Hussain et al., 2013), perceived difficulty (Pham and Wang, 2016), and learning performance (Cowley et al.”
Workload Estimation in Physical Human–Robot Interaction Using Physiological Measurements
Novak, D., Beyeler, B., Omlin, X., & Riener, R. (2015) · Interacting with Computers · doi:10.1093/iwc/iwu021 · Cited by 57
Predicting Cognitive Load in an Emergency Simulation Based on Behavioral and Physiological Measures
Appel, T., Sevcenko, N., Wortha, F., Tsarava, K., Moeller, K., Ninaus, M., Kasneci, E., & Gerjets, P. (2019) · 2019 International Conference on Multimodal Interaction · doi:10.1145/3340555.3353735 · Cited by 52
“The advantage of jointly considering several (physiological) data streams for the assessment of cognitive load was previously demonstrated in several studies [13, 16, 17, 28].”
BackgroundADABase: A Multimodal Dataset for Cognitive Load Estimation
Oppelt, M. P., Foltyn, A., Deuschel, J., Lang, N. R., Holzer, N., Eskofier, B. M., & Yang, S. H. (2022) · Sensors · doi:10.3390/s23010340 · Cited by 64
“compared different modalities (videos of the face and electro-physiological measurements) for different cognitive load tasks and proposed a fusion technique of inputs [45].”
BackgroundCan EEG Be Adopted as a Neuroscience Reference for Assessing Software Programmers’ Cognitive Load?
Medeiros, J., Couceiro, R., Duarte, G., Duraes, J. A., Castelhano, J., Duarte, I. C., Castelo-Branco, M. S., Madeira, H., Carvalho, P., & Teixeira, C. (2021) · Sensors · doi:10.3390/s21072338 · Cited by 50
“However, to the best of our knowledge, most studies exploring face analysis to assess cognitive states use it alongside other biosignasl [24].”
BackgroundCross-Task and Cross-Participant Classification of Cognitive Load in an Emergency Simulation Game
Appel, T., Gerjets, P. H., Hoffmann, S., Moeller, K., Ninaus, M., Scharinger, C., Sevcenko, N., Wortha, F., & Kasneci, E. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3098237 · Cited by 52
“The important impact of users’ cognitive load on their affective states has been demonstrated repeatedly (e.g., [1], [2], [3]).”
BackgroundBozkir, E., Geisler, D., & Kasneci, E. (2019) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr.2019.8797758 · Cited by 42
“…load of a subject, such as by means of N-back tasks (e.g., Appel et al. [2]), through the analysis of electroencephalography (EEG) signals (e.g., Zarjam et al. [22], Walter et al. [20]), by means of eye movements studies or through assessment of facial expressions (e.g., Hussain et al. [11]).”
BackgroundMa, Y., Liu, Q., & Yang, L. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.117346 · Cited by 32
Enhancing Online Learning: A Multimodal Approach for Cognitive Load Assessment
Xue, Y., Wang, K., & Qiu, Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2327198 · Cited by 31
“This may lead to under-or overloading of cognitive load, both of which are detrimental to task performance, especially in the case of overloading (Hussain et al., 2013).”
The Automatic Detection of Cognition Using EEG and Facial Expressions
Kerdawy, M. E., Halaby, M. E., Hassan, A., Maher, M., Fayed, H. A., Shawky, D., & Badawi, A. (2020) · Sensors · doi:10.3390/s20123516 · Cited by 23
“For instance, in [42] the authors examined facial characteristics to detect the affective states (emotions) that accompany the conceptual material’s deep-level learning.”
BackgroundPham, P. T., & Wang, J. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243002 · Cited by 20
Zhang, L., Wade, J. W., Bian, D., Fan, J., Swanson, A. R., Weitlauf, A. S., Warren, Z. E., & Sarkar, N. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20684-4_68 · Cited by 23
Hybrid System for Engagement Recognition During Cognitive Tasks Using a CFS + KNN Algorithm
Zennifa, F., Ageno, S., Hatano, S., & Iramina, K. (2018) · Sensors · doi:10.3390/s18113691 · Cited by 13
“[6] investigated the activity of physiological signals and facial responses to cognitive load under an emotional stimulus and collected participant ratings from a self-assessment manikin to find the normative ratings in the collection.”
BackgroundEniyandunmo, D., Shin, M., Lee, C., Anwar, A., Kim, E., Kim, K., Kim, Y. H., & Lee, C. (2024) · Ergonomics · doi:10.1080/00140139.2024.2379949 · Cited by 12
Zennifa, F., & Iramina, K. (2019) · IEEE Access · doi:10.1109/access.2019.2929596 · Cited by 10
“participant ratings from a self-assessment manikin to find the normative ratings in the collection by [3].”
BackgroundStress Assessment for Augmented Reality Applications Based on Head Movement Features
Ferrarotti, A., Baldoni, S., Carli, M., & Battisti, F. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3385637 · Cited by 10
“Different implementations of the MA test have been proposed in the literature [16], [17], [18].”
MethodWolf, E., Martínez, M., Roitberg, A., Stiefelhagen, R., & Deml, B. (2018) · doi:10.1145/3279810.3279852 · Cited by 9
Evaluating cognitive task result through heart rate pattern analysis
Yu, J., Liu, G. Y., Wen, W., & Chen, C. W. (2020) · Healthcare Technology Letters · doi:10.1049/htl.2018.5068 · Cited by 7
A Deep Learning Approach for Classifying Emotions from Physiological Data
AlZoubi, O., AlMakhadmeh, B., Tawalbeh, S. K., Yassein, M. B., & Hmeidi, I. (2020) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/icics49469.2020.239534 · Cited by 9
“Similarly, physiological signals, ECG, SC, and respiration, and facial videos were collected from twenty participants while they performed arithmetic tasks under an effective background simulated by IAPS [24].”
MethodInteractive Multimodal Photobook Co-Creation in Virtual Reality
Bittner, S.-J., Leist, R. A., Kopácsi, L., Bhatti, O. S., Selim, A. M., Barz, M., & Sonntag, D. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708557.3716355 · Cited by 5
“…objective and subjective data will be measured: The objective data collection will include metrics to evaluate efficiency and effectiveness (e.g., completion time and error rate) and, implicit human behaviour (e.g., interaction logs and multimodal data like eye and facial tracking [24, 33]).”
Xu, G., Wang, Z., Xu, T., Zhou, T., & Hu, H. (2023) · Applied Sciences · doi:10.3390/app132111924 · Cited by 5
Darejeh, A., Marcus, N., Mohammadi, G., & Sweller, J. (2026) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208261427867 · Cited by 58
Baseline User Calibration for Cold-Start Model Personalization in Mental State Estimation
Tervonen, J., Nath, R. K., Pettersson, K., Närväinen, J., & Mäntyjärvi, J. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-59717-6_3 · Cited by 2
The Potential of Cognitive Circles to Measure Mental Load
Méndez, G. G., Galárraga, L. A., Quijije, R., & Nacenta, M. A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747716 · Cited by 2
“…measurements include a large number of bodily signals, including cardiovascular (such as heart rate, ECG and heart rate variability [9]), brain-based (EEG [25], fMRI [41], fNIRS [28]), respiration [14], skin electrical and temperature changes [32], and eye-based [24] (gaze movement and pupil size).”
Seafarer fatigue identification based on multimodal data fusion
Lv, M., Shi, G., Liu, Z., & Wang, X. (2026) · Ocean Engineering · doi:10.1016/j.oceaneng.2026.124999 · Cited by 1
Diffusion Probabilistic Model Based Face Anonymization in Embedded Environments
Morvay, B. T., Béres, B., Torma, S., & Szegletes, L. (2023) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom59411.2023.10397492 · Cited by 1
Dong, Y., Fan, L., & Yan, B. (2025) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/cyberscitech68397.2025.00069
“Participants scoring below 24 (for lower education levels) or 27 (for higher education levels) were categorized as having MCI[18], [19], resulting in 79 participants in the MCI group and 53 in the Healthy Control (HC) group.”
Perez, E. S. (2026) · International Conference on Mobile Computing and Sustainable Informatics (ICMCSI) · doi:10.1109/icmcsi67283.2026.11412922
Towards Memory-Aware Adaptive Agents
van Sinttruije, D. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3828959
Metadata
- Authors
- Hussain, M. S., Calvo, R. A., & Chen, F.
- Affiliations
- The University of Sydney (AU)
- Data61 (AU)
- Corresponding author
- Hussain, M. S.
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-06-06
- Volume
- 26
- Issue
- 3
- Pages
- 256-268
- ISSN
- 0953-5438 (ISSN-L)
- 1873-7951
- OpenAlex
- W2059430380
- Cited by
- 34
- References
- 53
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 5 · 2023: 4 · 2022: 1 · 2021: 2 · 2020: 3 · 2019: 5 · 2018: 3 · 2016: 2 · 2015: 1 · 2014: 1
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Interacting with Computers (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- cognitive load detection, emotional interference, emotional arousal, facial expressions, physiological signals, task performance, multimodal fusion, mental arithmetic, emotion and cognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.