Classification of cognitive load based on neurophysiological features from functional near-infrared spectroscopy and electrocardiography signals on n-back task
Abstract
Cognitive load can be estimated using individuals' task performance, their subjective measures, and neurophysiological measures. Neurophysiological measures, which among others include brain activation signals obtained with various brain imaging techniques, such as the functional near-infrared spectroscopy (fNIRS), and signals from the peripheral physiology, such as the electrocardiography (ECG) signal, allow an objective and continuous estimation of cognitive load. In this article, the fNIRS and ECG signals were simultaneously collected from 32 participants and used to classify three levels of cognitive load on n-back task. A set of 30 fNIRS and ECG features proposed in this article enables the classification of different levels of cognitive load on n-back task using the support vector machine (SVM), k-nearest neighbors (KNN), and linear discriminant analysis (LDA) classification models. When combining the fNIRS and ECG features, three difficulties of the n-back task were classified with the mean accuracies ranging from 61% to 67%, while two difficulties were classified with the mean accuracy ranging from 70% to 84%. The most important features in the classification are discussed. The results presented in this article extend the existing empirical evidence that combining brain imaging and peripheral physiology features increases the accuracy of multi-level cognitive load classification, thus further underscoring the importance of multimodal approach to cognitive load classification.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
The assessment and analysis of handedness: The Edinburgh inventory
Oldfield, R. C. (1971) · Neuropsychologia · doi:10.1016/0028-3932(71)90067-4 · Cited by 35,900 · see citing works
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · see citing works
An overview of heart rate variability metrics and norms
Shaffer, F., & Ginsberg, J. P. (2017) · Frontiers in Public Health · doi:10.3389/fpubh.2017.00258 · Cited by 7,185 · see citing works
Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_8 · Cited by 2,527 · see citing works
Age differences in short-term retention of rapidly changing information.
Kirchner, W. K. (1958) · Journal of Experimental Psychology · doi:10.1037/h0043688 · Cited by 1,798 · see citing works
Strangman, G. E., Culver, J. P., Thompson, J. H., & Boas, D. A. (2002) · NeuroImage · doi:10.1006/nimg.2002.1227 · Cited by 1,421 · see citing works
fNIRS-based brain-computer interfaces: a review
Naseer, N., & Hong, K.-S. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00003 · Cited by 988 · see citing works
Instructional control of cognitive load in the training of complex cognitive tasks
Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994) · Educational Psychology Review · doi:10.1007/bf02213420 · Cited by 968 · see citing works
A quantitative comparison of NIRS and fMRI across multiple cognitive tasks
Cui, X., Bray, S., Bryant, D. M., Glover, G. H., & Reiss, A. L. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.10.069 · Cited by 951 · see citing works
Non-invasive neuroimaging using near-infrared light
Strangman, G. E., Boas, D. A., & Sutton, J. P. (2002) · Biological Psychiatry · doi:10.1016/s0006-3223(02)01550-0 · Cited by 907 · see citing works
Optical brain monitoring for operator training and mental workload assessment
Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., & Onaral, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.023 · Cited by 650 · see citing works
Mental workload and neural efficiency quantified in the prefrontal cortex using fNIRS
Causse, M., Chua, Z. K., Peysakhovich, V., del Campo, N., & Matton, N. (2017) · Scientific Reports · doi:10.1038/s41598-017-05378-x · Cited by 402 · see citing works
Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS
Herff, C., Heger, D., Fortmann, O., Hennrich, J., Putze, F., & Schultz, T. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00935 · Cited by 368 · see citing works
Stereotype Threat Undermines Intellectual Performance by Triggering a Disruptive Mental Load
Croizet, J., Desprès, G., Gauzins, M.-E., Huguet, P., Leyens, J., & Méot, A. (2004) · Personality and Social Psychology Bulletin · doi:10.1177/0146167204263961 · Cited by 322 · see citing works
Sensitivity of fNIRS to cognitive state and load
Fishburn, F. A., Norr, M. E., Medvedev, A. V., & Vaidya, C. J. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00076 · Cited by 308 · see citing works
Measuring regularity by means of a corrected conditional entropy in sympathetic outflow
Porta, A., Baselli, G., Liberati, D., Montano, N., Cogliati, C. B., Gnecchi‐Ruscone, T., Malliani, A., & Cerutti, S. (1998) · Biological Cybernetics · doi:10.1007/s004220050414 · Cited by 289 · see citing works
Feature Extraction and Classification Methods for Hybrid fNIRS-EEG Brain-Computer Interfaces
Hong, K., Khan, M. J., & Hong, M. J. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00246 · Cited by 287 · see citing works
Modified Beer-Lambert law for blood flow
Baker, W. B., Parthasarathy, A. B., Busch, D. R., Mesquita, R. C., Greenberg, J. H., & Yodh, A. G. (2014) · Biomedical Optics Express · doi:10.1364/boe.5.004053 · Cited by 285 · see citing works
Mental stress assessment using simultaneous measurement of EEG and fNIRS
Al-Shargie, F., Kiguchi, M., Badruddin, N., Dass, S. C., Hani, A. F. M., & Tang, T. B. (2016) · Biomedical Optics Express · doi:10.1364/boe.7.003882 · Cited by 283 · see citing works
Hankins, T., & Wilson, G. F. (1998) · PubMed · Cited by 248 · see citing works
Heart rate variability and cognitive processing: The autonomic response to task demands
Luque‐Casado, A., Perales, J. C., Vélez, D. C., & Sanabria, D. (2015) · Biological Psychology · doi:10.1016/j.biopsycho.2015.11.013 · Cited by 247 · see citing works
Neural and psychophysiological correlates of human performance under stress and high mental workload
Mandrick, K., Peysakhovich, V., Rémy, F., Lepron, E., & Causse, M. (2016) · Biological Psychology · doi:10.1016/j.biopsycho.2016.10.002 · Cited by 225 · see citing works
Sliding-window motion artifact rejection for Functional Near-Infrared Spectroscopy
Ayaz, H., İzzetoğlu, M., Shewokis, P. A., & Onaral, B. (2010) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2010.5627113 · Cited by 173 · see citing works
Mental Workload Alters Heart Rate Variability, Lowering Non-linear Dynamics
Delliaux, S., Delaforge, A., Deharo, J., & Chaumet, G. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.00565 · Cited by 172 · see citing works
Real-Time State Estimation in a Flight Simulator Using fNIRS
Gateau, T., Durantin, G., Lancelot, F., Scannella, S., & Dehais, F. (2015) · PLoS ONE · doi:10.1371/journal.pone.0121279 · Cited by 167 · see citing works
EEG-based workload estimation across affective contexts
hl, C. M. ⁄., Jeunet, C., & Lotte, F. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00114 · Cited by 161 · see citing works
Naseer, N., Noori, F. M., Qureshi, N. K., & Hong, K. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00237 · Cited by 140 · see citing works
Reporting and Interpreting Working Memory Performance in n-back Tasks
Meule, A. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00352 · Cited by 133 · see citing works
Liu, Y., Ayaz, H., & Shewokis, P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00389 · Cited by 121 · see citing works
Li, T., Luo, Q., & Gong, H. (2010) · Behavioural Brain Research · doi:10.1016/j.bbr.2010.01.033 · Cited by 105 · see citing works
Hoshi, Y., Tsou, B. H.-P., Billock, V. A., Tanosaki, M., Iguchi, Y., Shimada, M., Shinba, T., Yamada, Y., & Oda, I. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00412-9 · Cited by 103 · see citing works
Hwang, H.-J., Lim, J., Kim, D.-W., & Im, C. (2014) · Journal of Biomedical Optics · doi:10.1117/1.jbo.19.7.077005 · Cited by 96 · see citing works
Mental workload classification using heart rate metrics
Henelius, A., Hirvonen, K., Holm, A., Korpela, J., & Müller, K. (2009) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2009.5332602 · Cited by 73 · see citing works
NIRS-measured prefrontal cortex activity in neuroergonomics: strengths and weaknesses
Derosière, G., Mandrick, K., Dray, G., et al. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00583 · Cited by 58 · see citing works
A convolutional neural network based approach to QRS detection
Šarlija, M., Jurišić, F., & Popović, S. (2017) · International symposium on image and signal processing and analysis/ISPA ... · doi:10.1109/ispa.2017.8073581 · Cited by 57 · see citing works
Liu, Y., Ayaz, H., & Shewokis, P. A. (2017) · Brain-Computer Interfaces · doi:10.1080/2326263x.2017.1304020 · Cited by 49 · see citing works
Posada-Quintero, H. F., & Bolkhovsky, J. (2019) · Behavioral Sciences · doi:10.3390/bs9040045 · Cited by 48 · see citing works
Molteni, E., Baselli, G., Cerutti, S., Bianchi, A. M., Contini, D., Caffini, M., Cubeddu, R., Torricelli, A. M., & Spinelli, L. (2012) · Journal of Biomedical Optics · doi:10.1117/1.jbo.17.5.056005 · Cited by 47 · see citing works
Gurel, N. Z., Jung, H., Hersek, S., & Inan, O. T. (2018) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2872651 · Cited by 47 · see citing works
Le, A. S., Aoki, H., Murase, F., & Ishida, K. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00431 · Cited by 45 · see citing works
Wearable brain imaging with multimodal physiological monitoring
Strangman, G. E., Ivković, V., & Zhang, Q. (2017) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00297.2017 · Cited by 39 · see citing works
Ogawa, Y., Kotani, K., & Jimbo, Y. (2014) · Brain and Behavior · doi:10.1002/brb3.238 · Cited by 34 · see citing works
Cosic, K., Popovic, S., Sarlija, M., Mijic, I., Kokot, M., Kesedzic, I., Strangman, G., Ivkovic, V., & Zhang, Q. (2019) · IEEE Access · doi:10.1109/access.2019.2957357 · Cited by 29 · see citing works
Effects of Performance and Task Duration on Mental Workload during Working Memory Task
Khaksari, K., Condy, E. E., Millerhagen, J., Anderson, A. A., Dashtestani, H., & Gandjbakhche, A. (2019) · Photonics · doi:10.3390/photonics6030094 · Cited by 28 · see citing works
McKendrick, R. D., & Harwood, A. E. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00405 · Cited by 28 · see citing works
Molteni, E., Butti, M., Bianchi, A. M., et al. (2008) · doi:10.1109/iembs.2008.4650092 · Cited by 26 · see citing works
MMOD-COG: A Database for Multimodal Cognitive Load Classification
Mijić, I., Šarlija, M., & Petrinović, D. (2019) · International symposium on image and signal processing and analysis/ISPA ... · doi:10.1109/ispa.2019.8868678 · Cited by 23 · see citing works
A Non-parametric Approach to the Overall Estimate of Cognitive Load Using NIRS Time Series
Keshmiri, S., Sumioka, H., Yamazaki, R., & Ishiguro, H. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00015 · Cited by 21 · see citing works
Anderson, A. A., Parsa, K., Geiger, S., Zaragoza, R., Kermanian, R., Miguel, H. O., Dashtestani, H., Chowdhry, F. A., Smith, E. G., Aram, S., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0198257 · Cited by 21 · see citing works
Ayaz, H., & Onaral, B. (2005) · doi:10.17918/00008780 · Cited by 20 · see citing works
Psycho Physiological and Subjective Responses to Mental Workload Levels during N-Back Task
Fallahi, M., Heidarimoghadam, R., Motamedzade, M., et al. (2016) · Journal of Ergonomics · doi:10.4172/2165-7556.1000181 · Cited by 17 · see citing works
Cognitive workload classification using cardiovascular measures and dynamic features
Magnúsdóttir, E. H., Jóhannsdóttir, K. R., Bean, C., Ólafsson, B., & Guðnason, J. (2017) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom.2017.8268269 · Cited by 17 · see citing works
Comparison of classification performance for fNIRS-BCI system
Qureshi, N. K., Noori, F. M., Abdullah, A., & Naseer, N. (2016) · doi:10.1109/icrai.2016.7791228 · Cited by 9 · see citing works
Identification of Cognitive Load in Naturalistic Driving
Angell, L., Pérez, M. A., & Soccolich, S. (2015) · VTechWorks (Virginia Tech) · Cited by 8 · see citing works
Rahman, M. A., Khanam, F., & Ahmad, M. (2019) · International Conference on Electrical, Computer and Communication Engineering · doi:10.1109/ecace.2019.8679317 · Cited by 7 · see citing works
Woo, H. S., Willems, B. F., Harrison, J., İzzetoğlu, K., Hah, S., & Ayaz, H. (2013) · Journal of Bioresource Management · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Cosic, K., Kuljanac, A., & Palermo, S. (2026) · Frontiers in psychology · doi:10.3389/fpsyg.2026.1821242 · Cited by 3 · see citing works
“Continuous and comprehensive real-time monitoring of complex neuro-psycho-physiological parameters using a wide range of wearable sensors and ML/DL models enable accurate identification of a human’s emotional, cognitive, and behavioral states (Kesedžić et al., 2021).”
Self-citation
- Authors
- Kesedzic, I., Sarlija, M., Bozek, J., Popovic, S., & Cosic, K.
- Venue
- IEEE Sensors Journal
- Published
- 2021-07-01
- Volume
- 21
- Issue
- 13
- Pages
- 14131-14140
- ISSN
- 1530-437X (ISSN-L)
- 1558-1748
- 2379-9153
- OpenAlex
- W3102376463
- Cited by
- 29
- References
- 56
- Citations by year
- 2026: 7 · 2025: 9 · 2024: 6 · 2023: 3 · 2022: 2 · 2021: 2
- FWCI
- 0.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Sensors Journal (publishedVersion)
- Funders
- Croatian Science Foundation through the Project “Young Researcher’s Career Development—Training of Doctoral Students.”
- European Regional Development Fund (KK.01.1.1.01.0009 (DATACROSS))
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Optical Imaging and Spectroscopy Techniques
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- cognitive load classification, n-back task, functional near-infrared spectroscopy, electrocardiogram, multimodal fusion, brain activation, peripheral physiological signals, support vector machine, k-nearest neighbors, linear discriminant analysis, task difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review