Automatic Cognitive Load Detection from Face, Physiology, Task Performance and Fusion During Affective Interference
The DOI matches its Crossref record; 52 of the 53 references it cites were found in a registry or an index.
Abstract
Cognitive load (CL) is experienced during critical tasks and also while engaged emotional states are induced either by the task itself or by extraneous experiences. Emotions irrelevant to the working memory representation may interfere with the processing of relevant tasks and can influence task performance and behavior, making the accurate detection of CL from nonverbal information challenging. This paper investigates automatic CL detection from facial features, physiology and task performance under affective interference. Data were collected from participants (n=20) solving mental arithmetic tasks with emotional stimuli in the background, and a combined classifier was used for detecting CL levels. Results indicate that the face modality for CL detection was more accurate under affective interference, whereas physiology and task performance were more accurate without the affective interference. Multimodal fusion improved detection accuracies, but it was less accurate under affective interferences. More specifically, the accuracy decreased with an increasing intensity of emotional arousal.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Increased cognitive workload can lead to higher and lower respiration rate variability [99].”
From A Survey on Measuring Cognitive Workload in Human-Computer Interaction (2023) · cites this paper as [99]
Citations
Cites
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 4×
“The classifiers were implemented using MATLABArsenal7, a wrapper for the classifiers in Weka (Hall et al., 2009).”
MethodInfluentialRussell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
The relation of strength of stimulus to rapidity of habit formation
Yerkes, R. M., & Dodson, J. D. (1908) · Journal of Comparative Neurology and Psychology · doi:10.1002/cne.920180503 · Cited by 6,911
The magical number 4 in short-term memory: A reconsideration of mental storage capacity
Cowan, N. (2001) · Behavioral and Brain Sciences · doi:10.1017/s0140525x01003922 · Cited by 6,779
The Psychology of Human-Computer Interaction
Card, S. K., Moran, T. P., & Newell, A. (2018) · CRC Press eBooks · doi:10.1201/9780203736166 · Cited by 4,103
(1998) · Cited by 3,476
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
International Affective Picture System (IAPS) : Technical Manual and Affective Ratings
Lang, P. J. (1995) · Medical Entomology and Zoology · Cited by 2,870
Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_8 · Cited by 2,527
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · cited 2×
“, 2007), and even influence modalities such as physiological responses (Picard et al., 2001; Bradley et al., 2008; Hussain et al., 2011).”
BackgroundThe pupil as a measure of emotional arousal and autonomic activation
Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008) · Psychophysiology · doi:10.1111/j.1469-8986.2008.00654.x · Cited by 1,919
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 4×
“Behavioral and physiological expressions of feelings, along with factors such as moods, attitudes, affective styles and temperament, have been described as affective phenomena (Calvo and D’Mello, 2010).”
BackgroundMethodInfluentialSelf-citationWe Feel, Therefore We Learn: The Relevance of Affective and Social Neuroscience to Education
Immordino‐Yang, M. H., & Damasio, A. (2007) · Mind, Brain, and Education · doi:10.1111/j.1751-228x.2007.00004.x · Cited by 1,463 · cited 2×
“Cognition and affect are regarded as two interrelated aspects of human functioning and cannot be seen as completely separate phenomena (Immordino-Yang and Damasio, 2007; Huang and Tettegah, 2010).”
BackgroundPaas, F. G. W. C., & Van Merriënboer, J. J. G. (1994) · Journal of Educational Psychology · doi:10.1037/0022-0663.86.1.122 · Cited by 1,198
Cognitive load theory, educational research, and instructional design: some food for thought
de Jong, T. (2010) · Instructional Science · doi:10.1007/s11251-009-9110-0 · Cited by 1,180 · cited 1×
“CL is a factor that users experience, while mental effort is a unit that users actually exert in response (Jong, 2010).”
BackgroundEEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks.
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., Olmstead, R. E., Tremoulet, P. D., & Craven, P. L. (2007) · PubMed · Cited by 1,050
Toward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Workload assessment methodology.
O’Donnell, R. D., & Eggemeier, F. T. (1986) · Cited by 598
Physiological metrics of mental workload: A review of recent progress
Kramer, A. F. (2020) · Multiple-task performance · doi:10.1201/9781003069447-14 · Cited by 459
Galvanic skin response (GSR) as an index of cognitive load
Shi, Y., Ruiz, N., Taib, R., Choi, E., & Chen, F. (2007) · CHI '07 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1240866.1241057 · Cited by 423
Self-citationModulation of emotion by cognition and cognition by emotion
Blair, K. S., Smith, B. W., Mitchell, D. G. V., Morton, J., Vythilingam, M., Pessoa, L., Fridberg, D., Zametkin, A., Nelson, E. E., Drevets, W. C., et al. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.11.048 · Cited by 405
D’Esposito, M., Postle, B. R., Jonides, J., & Smith, E. E. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.13.7514 · Cited by 329
Integrating cognitive load theory and concepts of human–computer interaction
Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010) · Computers in Human Behavior · doi:10.1016/j.chb.2010.05.031 · Cited by 328
Toward multimodal human-computer interface
Sharma, R., Pavlovic, V. I., & Huang, T. S. (1998) · Proceedings of the IEEE · doi:10.1109/5.664275 · Cited by 302
A Survey of Decision Fusion and Feature Fusion Strategies for Pattern Classification
Mangai, U., Samanta, S., Das, S., & Chowdhury, P. (2010) · IETE Technical Review · doi:10.4103/0256-4602.64604 · Cited by 302
On the Psychophysics of Workload: Why Bother with Subjective Measures?
Gopher, D., & Braune, R. (1984) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872088402600504 · Cited by 280
Klingner, J., Tversky, B., & Hanrahan, P. (2011) · Psychophysiology · doi:10.1111/j.1469-8986.2010.01069.x · Cited by 255
Tian, Y.-L., Kanade, T., & Cohn, J. F. (2003) · Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition · doi:10.1109/afgr.2002.1004159 · Cited by 223
Load-dependent modulation of affective picture processing
Erthal, F. S., De Oliveira, L., Mocaiber, I., Pereira, M. G., Machado-Pinheiro, W., Volchan, E., & Pessoa, L. (2005) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.5.4.388 · Cited by 194
Duric, Z., Gray, W. D., Heishman, R., Li, F., Rosenfeld, A., Schoelles, M. J., Schunn, C., & Wechsler, H. (2002) · Proceedings of the IEEE · doi:10.1109/jproc.2002.801449 · Cited by 185 · cited 4×
“Studies have also suggested that interdependences between cognition, affective states and other human functioning (e.g. behavioral, perception) are essential for building adaptive intelligent HCI systems (Duric et al., 2002).”
BackgroundInfluentialContemporary cognitive load theory research: The good, the bad and the ugly
Kirschner, P. A., Ayres, P., & Chandler, P. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2010.06.025 · Cited by 144 · cited 1×
“Research in cognitive load theory has focused on identifying instructional designs that create unnecessary working memory load, such as extraneous load, in order to provide more effective alternatives (Ayres and Gog, 2009; Kirschner et al., 2011).”
BackgroundMultimodal behavior and interaction as indicators of cognitive load
Chen, F., Ruiz, N., Choi, E., Epps, J., Khawaja, M. A., Taib, R., Yin, B., & Wang, Y. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2395123.2395127 · Cited by 141
Self-citationThe Regulation of Working Methods as a Function of Work-load among Air Traffic Controllers
SPERANDIO, J.-C. (1978) · Ergonomics · doi:10.1080/00140137808931713 · Cited by 138
Automatic classification of eye activity for cognitive load measurement with emotion interference
Chen, S., & Epps, J. (2013) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2012.10.021 · Cited by 125
Emotion processing effects on interference resolution in working memory.
Levens, S. M., & Phelps, E. A. (2008) · Emotion · doi:10.1037/1528-3542.8.2.267 · Cited by 119
Affective Interference: An Explanation for Negative Attention Biases in Dysphoria?
Siegle, G. J., Ingram, R. E., & Matt, G. E. (2002) · Cognitive Therapy and Research · doi:10.1023/a:1013893705009 · Cited by 117
The pupillary response as an indicator of arousal and cognition
Stanners, R. F., Coulter, M., Sweet, A. W., & Murphy, P. (1979) · Motivation and Emotion · doi:10.1007/bf00994048 · Cited by 116
Ark, W., Dryer, D. C., & Lu, D. J. (1999) · International Conference on Human-Computer Interaction · Cited by 114
Speech-based cognitive load monitoring system
Yin, B., Chen, F., Ruiz, N., & Ambikairajah, E. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518041 · Cited by 87
Self-citationInvestigation of spectral centroid features for cognitive load classification
Le, P. N., Eliathamby Ambikairajah, A., Epps, J., Sethu, V., & Choi, E. H. C. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.005 · Cited by 77
State of the art research into Cognitive Load Theory
Ayres, P., & Gog, T. V. (2009) · Computers in Human Behavior · doi:10.1016/j.chb.2008.12.007 · Cited by 72
Gütl, C., Pivec, M., Trummer, C., García-Barrios, V. M., Mödritscher, F., Pripfl, J., & Umgeher, M. (2005) · European Journal of Open Distance and E-Learning · Cited by 63
de Tjerk, E. G., Henryk, F. R. A., & Neerincx, M. A. (2010) · Journal of Cognitive Engineering and Decision Making · doi:10.1518/155534310x522851 · Cited by 52
Investigation of Combining SVM and Decision Tree for Emotion Classification
Nguyen, T., Li, M., Bass, I., & Sethi, I. K. (2006) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2005.72 · Cited by 33
Cognitive Load and Empathy in Serious Games: A Conceptual Framework
Huang, W. D., & Tettegah, S. Y. (2013) · Advances in human and social aspects of technology book series · doi:10.4018/978-1-4666-5071-8.ch002 · Cited by 17
Cognitive Load in Adaptive Multimedia Learning
Kalyuga, S. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1_15 · Cited by 13
A Study of Voice Source and Vocal Tract Filter Based Features in Cognitive Load Classification
Le, P. N., Epps, J., Choi, E. H. C., & Eliathamby Ambikairajah, A. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.1097 · Cited by 13
Automatic natural expression recognition using head movement and skin color features
Monkaresi, H., Calvo, R. A., & Hussain, M. S. (2012) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/2254556.2254678 · Cited by 12
Self-citationTowards a theoretical framework for ensemble classification
Seewald, A. K. (2003) · Cited by 11
Towards Detecting Cognitive Load and Emotions in Usability Studies Using the RealEYES Framework
Schultz, R., Peter, C., Blech, M., Voskamp, J., & Urban, B. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73287-7_50 · Cited by 6
(2000)
Research works printed but not linked: 5
- 2007, Aviation, Space, and Environmental Medicine, 78, 231
- 2012, ACM TRANS INTERACT INTELL SYST, 2, 221
- 2011, NEW PERSPECTIVES ON AFFECT AND LEARNING TECHNOLOGIES, 3, 203
- 2010, Psychophysiology, 48, 323
- 1986, HANDBOOK PERCEPT HUM PERFORM, 2, 42
Cited by
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“…21, 27, 44–47, 76, 87, 99, 165, 184, 193, 228, 236, 258] Eye Blinks [4, 5, 27, 43–47, 76, 87, 99, 175, 182, 184, 236, 268] Fixations [1, 10, 54, 68, 99, 140, 153, 184, 219, 228] Smooth Pursuits and Ocular Movements [133] Cardiovascular Heart Rate [4, 10, 25, 27, 40, 41, 49, 50, 76, 78, 82, 85, 87,…”
InfluentialMultimodal Fusion for Objective Assessment of Cognitive Workload: A Review
Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2939399 · Cited by 184
“The same authors in a subsequent study [85] investigated the impact of affective interference that emerge due to the nature of the task or personal feelings (i.e., emotional arousal, frustration, and confusion) on the performance of single models (based on single modality) and hybrid fusion model…”
BackgroundCognitive Load Measurement in a Virtual Reality-Based Driving System for Autism Intervention
Zhang, L., Wade, J., Bian, D., Fan, J., Swanson, A., Weitlauf, A., Warren, Z., & Sarkar, N. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2582490 · Cited by 141
“Different kernel functions of support vector machine (SVM), including linear kernel [31] and Gaussian kernel [32], were used in cognitive load measurement.”
BackgroundMethodInfluentialBarua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 152
“[98], [100], [104], [116], [125], [127], [185], [203], [237], [319], [357], [386], [402], Image & Numerical [62], [75], [119], [126], [69], [306], [323], [345], [396], [401], Audio & Text & Sensor [375], Audio & Text [178], [276], [369], [382], [383], Text & Signal [109], Text & Numerical [297], [341], Sensor & Signal…”
Inside Out: Detecting Learners’ Confusion to Improve Interactive Digital Learning Environments
Arguel, A., Lockyer, L., Lipp, O. V., Lodge, J. M., & Kennedy, G. E. (2016) · Journal of Educational Computing Research · doi:10.1177/0735633116674732 · Cited by 118
“For this reason, a multimodal approach consisting of the integration of several indicators may be preferred to detect confusion (Hussain, Calvo, & Chen, 2014; Hussain, Monkaresi, & Calvo, 2012).”
Metadata
- Authors
- Hussain, M. S., Calvo, R. A., & Chen, F.
- Affiliations
- The University of Sydney (AU)
- Data61 (AU)
- Corresponding author
- Hussain, M. S.
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-06-06
- Volume
- 26
- Issue
- 3
- Pages
- 256-268
- ISSN
- 0953-5438 (ISSN-L)
- 1873-7951
- OpenAlex
- W2059430380
- Cited by
- 34
- References
- 53
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 5 · 2023: 4 · 2022: 1 · 2021: 2 · 2020: 3 · 2019: 5 · 2018: 3 · 2016: 2 · 2015: 1 · 2014: 1
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Interacting with Computers (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- cognitive load detection, emotional interference, emotional arousal, facial expressions, physiological signals, task performance, multimodal fusion, mental arithmetic, emotion and cognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.