CoChat
Cited by 184

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review

Open access
Debie, Essam, Fernandez Rojas, Raul, Fidock, Justin, Barlow, Michael, Kasmarik, Kathryn, Anavatti, Sreenatha, Garratt, Matt, Abbass, Hussein A. · 2021 · IEEE Transactions on Cybernetics

Abstract

Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tcyb.2019.2939399

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 85

  1. bulling, 2008, Eyemote—Towards context-aware gaming using eye movements recorded from wearable electrooculography, Fun and Games, 33, 10.1007/978-3-540-88322-7_4 · doi:10.1007/978-3-540-88322-7_4
  2. 10.1109/TCYB.2014.2306916 · DOI
  3. 10.1016/0301-0511(92)90018-P · DOI · Paper page
  4. 10.1007/978-3-319-41947-3_13 · DOI
  5. 10.1016/j.bbr.2010.03.011 · DOI
  6. 10.1080/17470210600673818 · DOI · Paper page
  7. suoja, 2017, Application for pre-processing and visualization of electrodermal activity wearable data, Proc EMBEC NBC, 93
  8. 10.3389/fnhum.2013.00935 · DOI · Paper page
  9. 10.1016/0167-8760(89)90258-4 · DOI
  10. 10.1093/cercor/7.4.374 · DOI · Paper page
  11. 10.3389/fnhum.2017.00264 · DOI
  12. 10.1109/TCYB.2015.2479240 · DOI
  13. 10.1177/1541931214581048 · DOI
  14. rojas, 2017, Toward a functional near-infrared spectroscopy-based monitoring of pain assessment for nonverbal patients, J Biomed Opt, 22
  15. sciarini, 2009, Assessing Cognitive State With Multiple Physiological Measures A Modular Approach, 533
  16. 10.1177/0018720812437275 · DOI
  17. rojas, 2017, Physiological fluctuations show frequency-specific networks in FNIRS signals during resting state, Proc IEEE 39th Annu Int Conf Eng Med Biol Soc (EMBC), 2550
  18. 10.1177/027836498800700608 · DOI · Paper page
  19. 10.1109/CCECE.1996.548247 · DOI
  20. 10.1002/rob.4620070305 · DOI
  21. 10.1016/j.biopsycho.2013.11.004 · DOI
  22. 10.1177/001872089303500205 · DOI · Paper page
  23. rojas, 2017, FNIRS approach to pain assessment for non-verbal patients, Proc Int Conf Neural Inf Process, 778
  24. 10.1371/journal.pone.0164627 · DOI
  25. 10.1080/00140130802120267 · DOI · Paper page
  26. fernandez, 2015, Combining EEG with pupillometry to improve cognitive workload detection, Computer, 48, 18, 10.1109/MC.2015.314 · DOI
  27. 10.1097/00003643-200805001-00040 · doi:10.1097/00003643-200805001-00040
  28. abbass, 2011, Towards a code of best practice for evaluating air traffic control interfaces, Proc Aust Transp Res Forum, 469
  29. 10.1109/THMS.2017.2782483 · DOI
  30. frank, 2007, Monitoring Seated Postural Responses to Assess Cognitive State
  31. 10.1080/00140139.2014.956151 · DOI · Paper page
  32. hirshfield, 2009, Combining electroencephalograph and functional near infrared spectroscopy to explore users’ mental workload, Proc Int Conf Found Augmented Cogn, 239
  33. 10.1016/j.ijhcs.2009.07.005 · DOI · Paper page
  34. 10.1007/978-3-642-23768-3_43 · DOI
  35. 10.1145/971478.971495 · DOI
  36. 10.1126/science.628837 · DOI · Paper page
  37. 10.1016/j.neubiorev.2012.10.003 · DOI · Paper page
  38. 10.1007/s00521-016-2750-5 · DOI
  39. 10.1080/2326263X.2017.1304020 · DOI
  40. 10.3389/fnins.2014.00373 · DOI
  41. 10.1177/1071181312561367 · DOI
  42. 10.1109/TNSRE.2013.2292995 · DOI
  43. plechawska-wójcik, 2016, An analysis of EEG signal combined with pupillary response in the dynamics of human cognitive processing, Proc 9th Int Conf Human Syst Interact (HSI), 378
  44. 10.1145/2950112.2964585 · DOI
  45. 10.3389/fnhum.2017.00389 · DOI
  46. 10.1007/978-3-319-67642-5_8 · DOI
  47. 10.1007/s40708-016-0055-1 · DOI
  48. zhang, 2017, Feature weight driven interactive mutual information modeling for heterogeneous bio-signal fusion to estimate mental workload, SENSORS, 17
  49. brooks, 1998, Multi-Sensor Fusion Fundamentals and Applications with Software
  50. 10.2307/2334380 · DOI
  51. 10.1093/biomet/28.3-4.321 · DOI · Paper page
  52. 10.1016/j.chemolab.2010.04.012 · DOI
  53. 10.1109/JPROC.2010.2057231 · DOI
  54. 10.1007/978-3-642-21852-1_22 · DOI
  55. 10.1016/j.neuroimage.2011.07.084 · DOI · Paper page
  56. 10.1088/1741-2560/12/3/036004 · DOI
  57. 10.1109/TBME.2014.2300492 · DOI
  58. cain, 2007, A review of the mental workload literature
  59. 10.1109/NER.2015.7146546 · DOI
  60. 10.1109/PST.2016.7906958 · DOI
  61. 10.1109/TSMCB.2012.2212884 · DOI
  62. 10.1007/978-3-319-59259-6_4 · DOI
  63. 10.1007/978-3-319-40238-3_59 · DOI
  64. 10.1177/0018720814539505 · DOI · Paper page
  65. 10.1007/s12559-015-9365-5 · DOI
  66. 10.1007/978-3-319-39955-3_38 · DOI
  67. 10.1016/0301-0511(95)05165-1 · DOI · Paper page
  68. pomplun, 2003, Pupil dilation as an indicator of cognitive workload in human–computer interaction, Proc Int Conf HCI, 2003
  69. tsai, 2007, Task performance and eye activity: Predicting behavior relating to cognitive workload, Aviation Space Environ Med, 78, 176b
  70. 10.1016/j.ergon.2006.04.002 · DOI · Paper page
  71. mota, 2003, Automated posture analysis for detecting learner’s interest level, Proc Conf Comput Vis Pattern Recognit Workshop (CVPRW), 5, 49
  72. 10.1207/S15327108IJAP1201_2 · DOI · Paper page
  73. 10.1201/9781420053098 · DOI · Paper page
  74. 10.1109/ICPR.2010.913 · DOI
  75. 10.1109/TCYB.2016.2625419 · DOI · Paper page
  76. 10.1016/j.ijhcs.2009.03.005 · DOI · Paper page
  77. chanel, 2006, Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals, Advanced Multimedia Content Processing, 530
  78. hussain, 2011, Classification of cognitive load from task performance & multichannel physiology during affective changes, Proc Workshop MMCogEmS Inferring Cogn Emotional States Multimodal Meas (ICMI)
  79. 10.1518/001872007X249875 · DOI · Paper page
  80. 10.1016/j.neuroimage.2011.07.091 · DOI · Paper page
  81. 10.1109/CIBEC.2008.4786096 · DOI
  82. 10.1007/978-3-319-20684-4_68 · DOI
  83. 10.1145/2395123.2395127 · DOI
  84. 10.1007/s12541-013-0179-7 · DOI
  85. 10.1007/978-3-319-07515-0_11 · DOI

Cited by

See all 184

Metadata

Authors
Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A.
Venue
IEEE Transactions on Cybernetics
Published
2021-03
Volume
51
Issue
3
Pages
1542-1555
ISSN
  • 2168-2267 (ISSN-L)
  • 2168-2275
OpenAlex
W2975376327
Cited by
184
References
13
FWCI
13.61
Type
journal article
Language
en
Open access
green
Funders
  • Commonwealth of Australia through the Australian Army
  • Defence Science Partnerships agreement of the Defence Science and Technology Group
  • Human Performance Research Network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review. https://app.cochat.ai/papers/10.1109/tcyb.2019.2939399
The paper
Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2021). Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review. IEEE Transactions on Cybernetics. https://doi.org/10.1109/tcyb.2019.2939399

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.