The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review
Open accessAbstract
Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Multiple resources and performance prediction
Wickens, C. D. (2002) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220210123806 · Cited by 2,621 · cited 1×
“Among cognitive workload definitions and explanations proposed in the literature, the definitions proposed by Wickens [1], Cain [2], and Abbass et al. [5] explain the different components and external factors of cognitive workload.”
BackgroundBoucsein, W. (2012) · doi:10.1007/978-1-4614-1126-0 · Cited by 2,526
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Optical brain monitoring for operator training and mental workload assessment
Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., & Onaral, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.023 · Cited by 650 · cited 1×
“Different studies have reported that increased levels of HbO in the prefrontal cortex correlates with increased task engagement which is used to indicate the increased cognitive workload [27], [28].”
BackgroundToward an Affect-Sensitive AutoTutor
D'Mello, S., Picard, R. W., & Graesser, A. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.79 · Cited by 386 · cited 1×
“Seated posture has shown to correlate with task engagement and boredom, increased pressure on the seat was associated with increased engagement, while increased pressure on the back of the chair was associated with boredom [48].”
BackgroundTask-evoked pupillary response to mental workload in human-computer interaction
Iqbal, S. T., Zheng, X. S., & Bailey, B. P. (2004) · CHI '04 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/985921.986094 · Cited by 317
Hogervorst, M. A., Brouwer, A.-M., & van Erp, J. B. F. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00322 · Cited by 292 · cited 1×
“Hogervorst et al. [20] performed extensive studies to analyze the impact of different types of fusions on the performance of workload classification.”
BackgroundXie, B., & Salvendy, G. (2000) · Work & Stress · doi:10.1080/026783700417249 · Cited by 247
Research works printed but not linked: 85
- bulling, 2008, Eyemote—Towards context-aware gaming using eye movements recorded from wearable electrooculography, Fun and Games, 33, 10.1007/978-3-540-88322-7_4 · doi:10.1007/978-3-540-88322-7_4
- 10.1109/TCYB.2014.2306916 · DOI
- 10.1016/0301-0511(92)90018-P · DOI · Paper page
- 10.1007/978-3-319-41947-3_13 · DOI
- 10.1016/j.bbr.2010.03.011 · DOI
- 10.1080/17470210600673818 · DOI · Paper page
- suoja, 2017, Application for pre-processing and visualization of electrodermal activity wearable data, Proc EMBEC NBC, 93
- 10.3389/fnhum.2013.00935 · DOI · Paper page
- 10.1016/0167-8760(89)90258-4 · DOI
- 10.1093/cercor/7.4.374 · DOI · Paper page
- 10.3389/fnhum.2017.00264 · DOI
- 10.1109/TCYB.2015.2479240 · DOI
- 10.1177/1541931214581048 · DOI
- rojas, 2017, Toward a functional near-infrared spectroscopy-based monitoring of pain assessment for nonverbal patients, J Biomed Opt, 22
- sciarini, 2009, Assessing Cognitive State With Multiple Physiological Measures A Modular Approach, 533
- 10.1177/0018720812437275 · DOI
- rojas, 2017, Physiological fluctuations show frequency-specific networks in FNIRS signals during resting state, Proc IEEE 39th Annu Int Conf Eng Med Biol Soc (EMBC), 2550
- 10.1177/027836498800700608 · DOI · Paper page
- 10.1109/CCECE.1996.548247 · DOI
- 10.1002/rob.4620070305 · DOI
- 10.1016/j.biopsycho.2013.11.004 · DOI
- 10.1177/001872089303500205 · DOI · Paper page
- rojas, 2017, FNIRS approach to pain assessment for non-verbal patients, Proc Int Conf Neural Inf Process, 778
- 10.1371/journal.pone.0164627 · DOI
- 10.1080/00140130802120267 · DOI · Paper page
- fernandez, 2015, Combining EEG with pupillometry to improve cognitive workload detection, Computer, 48, 18, 10.1109/MC.2015.314 · DOI
- 10.1097/00003643-200805001-00040 · doi:10.1097/00003643-200805001-00040
- abbass, 2011, Towards a code of best practice for evaluating air traffic control interfaces, Proc Aust Transp Res Forum, 469
- 10.1109/THMS.2017.2782483 · DOI
- frank, 2007, Monitoring Seated Postural Responses to Assess Cognitive State
- 10.1080/00140139.2014.956151 · DOI · Paper page
- hirshfield, 2009, Combining electroencephalograph and functional near infrared spectroscopy to explore users’ mental workload, Proc Int Conf Found Augmented Cogn, 239
- 10.1016/j.ijhcs.2009.07.005 · DOI · Paper page
- 10.1007/978-3-642-23768-3_43 · DOI
- 10.1145/971478.971495 · DOI
- 10.1126/science.628837 · DOI · Paper page
- 10.1016/j.neubiorev.2012.10.003 · DOI · Paper page
- 10.1007/s00521-016-2750-5 · DOI
- 10.1080/2326263X.2017.1304020 · DOI
- 10.3389/fnins.2014.00373 · DOI
- 10.1177/1071181312561367 · DOI
- 10.1109/TNSRE.2013.2292995 · DOI
- plechawska-wójcik, 2016, An analysis of EEG signal combined with pupillary response in the dynamics of human cognitive processing, Proc 9th Int Conf Human Syst Interact (HSI), 378
- 10.1145/2950112.2964585 · DOI
- 10.3389/fnhum.2017.00389 · DOI
- 10.1007/978-3-319-67642-5_8 · DOI
- 10.1007/s40708-016-0055-1 · DOI
- zhang, 2017, Feature weight driven interactive mutual information modeling for heterogeneous bio-signal fusion to estimate mental workload, SENSORS, 17
- brooks, 1998, Multi-Sensor Fusion Fundamentals and Applications with Software
- 10.2307/2334380 · DOI
- 10.1093/biomet/28.3-4.321 · DOI · Paper page
- 10.1016/j.chemolab.2010.04.012 · DOI
- 10.1109/JPROC.2010.2057231 · DOI
- 10.1007/978-3-642-21852-1_22 · DOI
- 10.1016/j.neuroimage.2011.07.084 · DOI · Paper page
- 10.1088/1741-2560/12/3/036004 · DOI
- 10.1109/TBME.2014.2300492 · DOI
- cain, 2007, A review of the mental workload literature
- 10.1109/NER.2015.7146546 · DOI
- 10.1109/PST.2016.7906958 · DOI
- 10.1109/TSMCB.2012.2212884 · DOI
- 10.1007/978-3-319-59259-6_4 · DOI
- 10.1007/978-3-319-40238-3_59 · DOI
- 10.1177/0018720814539505 · DOI · Paper page
- 10.1007/s12559-015-9365-5 · DOI
- 10.1007/978-3-319-39955-3_38 · DOI
- 10.1016/0301-0511(95)05165-1 · DOI · Paper page
- pomplun, 2003, Pupil dilation as an indicator of cognitive workload in human–computer interaction, Proc Int Conf HCI, 2003
- tsai, 2007, Task performance and eye activity: Predicting behavior relating to cognitive workload, Aviation Space Environ Med, 78, 176b
- 10.1016/j.ergon.2006.04.002 · DOI · Paper page
- mota, 2003, Automated posture analysis for detecting learner’s interest level, Proc Conf Comput Vis Pattern Recognit Workshop (CVPRW), 5, 49
- 10.1207/S15327108IJAP1201_2 · DOI · Paper page
- 10.1201/9781420053098 · DOI · Paper page
- 10.1109/ICPR.2010.913 · DOI
- 10.1109/TCYB.2016.2625419 · DOI · Paper page
- 10.1016/j.ijhcs.2009.03.005 · DOI · Paper page
- chanel, 2006, Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals, Advanced Multimedia Content Processing, 530
- hussain, 2011, Classification of cognitive load from task performance & multichannel physiology during affective changes, Proc Workshop MMCogEmS Inferring Cogn Emotional States Multimodal Meas (ICMI)
- 10.1518/001872007X249875 · DOI · Paper page
- 10.1016/j.neuroimage.2011.07.091 · DOI · Paper page
- 10.1109/CIBEC.2008.4786096 · DOI
- 10.1007/978-3-319-20684-4_68 · DOI
- 10.1145/2395123.2395127 · DOI
- 10.1007/s12541-013-0179-7 · DOI
- 10.1007/978-3-319-07515-0_11 · DOI
Cited by
Human Digital Twin in the context of Industry 5.0
Wang, B., Zhou, H., Li, X., Yang, G., Zheng, P., Song, C., Yuan, Y., Wuest, T., Yang, H., & Wang, L. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2023.102626 · Cited by 424
Human and artificial intelligence collaboration for socially shared regulation in learning
Järvelä, S., Nguyen, A., & Hadwin, A. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13325 · Cited by 342
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“Realizing new schemes or patterns that help solve a task may increase task engagement and foster learning.”
Multi-Sensor data fusion in intelligent fault diagnosis of rotating machines: A comprehensive review
Kibrete, F., Engida Woldemichael, D., & Shimels Gebremedhen, H. (2024) · Measurement · doi:10.1016/j.measurement.2024.114658 · Cited by 270
(2025) · doi:10.1109/jbhi.2023.3346205 · Cited by 47
Metadata
- Authors
- Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A.
- Venue
- IEEE Transactions on Cybernetics
- Published
- 2021-03
- Volume
- 51
- Issue
- 3
- Pages
- 1542-1555
- ISSN
- 2168-2267 (ISSN-L)
- 2168-2275
- OpenAlex
- W2975376327
- Cited by
- 184
- References
- 13
- FWCI
- 13.61
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- Commonwealth of Australia through the Australian Army
- Defence Science Partnerships agreement of the Defence Science and Technology Group
- Human Performance Research Network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review. https://app.cochat.ai/papers/10.1109/tcyb.2019.2939399
- The paper
- Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2021). Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review. IEEE Transactions on Cybernetics. https://doi.org/10.1109/tcyb.2019.2939399
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.