Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Human facial expressions are regarded as a vital indicator of one’s emotion and intention, and even reveal the state of health and wellbeing. Emotional states have been associated with information processing within and between subcortical and cortical areas of the brain, including the amygdala and prefrontal cortex. In this study, we evaluated the relationship between spontaneous human facial affective expressions and multi-modal brain activity measured via non-invasive and wearable sensors: functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) signals. The affective states of twelve male participants detected via fNIRS, EEG, and spontaneous facial expressions were investigated in response to both image-content stimuli and video-content stimuli. We propose a method to jointly evaluate fNIRS and EEG signals for affective state detection (emotional valence as positive or negative). Experimental results reveal a strong correlation between spontaneous facial affective expressions and the perceived emotional valence. Moreover, the affective states were estimated by the fNIRS, EEG, and fNIRS + EEG brain activity measurements. We show that the proposed EEG + fNIRS hybrid method outperforms fNIRS-only and EEG-only approaches. Our findings indicate that the dynamic (video-content based) stimuli triggers a larger affective response than the static (image-content based) stimuli. These findings also suggest joint utilization of facial expression and wearable neuroimaging, fNIRS, and EEG, for improved emotional analysis and affective brain–computer interface applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ekman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Subcortical and cortical brain activity during the feeling of self-generated emotions
Damasio, A. R., Grabowski, T. J., Bechara, A., Damasio, H., Ponto, L. L. B., Parvizi, J., & Hichwa, R. D. (2000) · Nature Neuroscience · doi:10.1038/79871 · Cited by 2,117
Emotion, plasticity, context, and regulation: Perspectives from affective neuroscience.
Davidson, R. J., Jackson, D. C., & Kalin, N. H. (2000) · Psychological Bulletin · doi:10.1037/0033-2909.126.6.890 · Cited by 1,322 · cited 3×
“both of them have received considerable attention on emotion inference and emotional mapping on brain activities [21,41,71].”
BackgroundPrefrontal Brain Asymmetry: A Biological Substrate of the Behavioral Approach and Inhibition Systems
Sutton, S. K., & Davidson, R. J. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00413.x · Cited by 1,263
Optical brain monitoring for operator training and mental workload assessment
Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., & Onaral, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.023 · Cited by 650 · cited 2×
“1 Hz cutoff frequency was used to achieve noise reduction in fNIRS signals [26].”
BackgroundMethodSelf-citationDolcos, F., LaBar, K. S., & Cabeza, R. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.05.015 · Cited by 439
Sex differences in the neural correlates of emotion: evidence from neuroimaging
Whittle, S., Yücel, M., Yap, M. B. H., & Allen, N. B. (2011) · Biological Psychology · doi:10.1016/j.biopsycho.2011.05.003 · Cited by 305 · cited 1×
“Finally, some prior work has shown that men and women differ in the neural mechanisms underlying their expression of specific emotions [73].”
ResultSundström Poromaa, I., & Gingnell, M. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00380 · Cited by 291
Cognitive, sensory, and emotional changes associated with the menstrual cycle: a review
Farage, M. A., Osborn, T. W., & MacLean, A. B. (2008) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-008-0708-2 · Cited by 208
Cited by
A multimodal emotion recognition method based on facial expressions and electroencephalography
Tan, Y., Sun, Z., Duan, F., Solé-Casals, J., & Caiafa, C. F. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103029 · Cited by 88
Saffaryazdi, N., Wasim, S. T., Dileep, K., Nia, A. F., Nanayakkara, S., Broadbent, E., & Billinghurst, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.864047 · Cited by 69
“Facial expressions are one of the most commonly used input modalities analyzed to identify emotional state (Sun et al., 2020).”
MethodState-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
Moussaoui, K., & Farah, M. (2024) · 2024 IEEE 7th International Conference on Advanced Technologies, Signal and Image Processing (ATSIP) · doi:10.1109/atsip62566.2024.10638977 · Cited by 2
Combination of facial expressions and EEG for multimodal emotion recognition
Thanh, L. N., & Nguyen, H. T. (2025) · VNU Journal of Science: Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.4566
“Sun et al. (2020) [8] investigated the relationship between spontaneous human facial expressions and multimodal brain activity, using wearable sensors to capture data from functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) signals.”
Metadata
- Authors
- Sun, Y., Ayaz, H., & Akansu, A. N.
- Venue
- Brain Sciences
- Publisher
- MDPI AG
- Published
- 2020-02-06
- Volume
- 10
- Issue
- 2
- Pages
- 85
- ISSN
- 2076-3425 (ISSN-L)
- OpenAlex
- W3005307761
- Cited by
- 57
- References
- 11
- FWCI
- 5.55
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression. https://app.cochat.ai/papers/10.3390/brainsci10020085
- The paper
- Sun, Y., Ayaz, H., & Akansu, A. N. (2020). Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression. Brain Sciences. https://doi.org/10.3390/brainsci10020085
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.