Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 200

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Open access
Cimtay, Yucel, Ekmekcioglu, Erhan, Caglar-Ozhan, Seyma · 2020 · IEEE Access

Abstract

Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2020.3023871

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 89

  1. 10.1109/MIS.2017.23 · DOI · Paper page
  2. 10.1109/MIS.2017.57 · DOI
  3. 10.1109/MIS.2019.2904691 · DOI · Paper page
  4. 10.1109/MIS.2019.2954966 · DOI
  5. 10.1109/MIS.2020.2979203 · DOI
  6. 10.1109/MIS.2016.31 · DOI · Paper page
  7. 10.1109/MIS.2017.3711649 · DOI
  8. 10.1109/MIS.2017.4531228 · DOI · Paper page
  9. 10.1109/MIS.2018.033001419 · DOI
  10. 10.1109/MIS.2019.2899142 · doi:10.1109/mis.2019.2899142
  11. 10.1109/MIS.2017.22 · DOI
  12. 10.1109/MIS.2018.2882362 · DOI · Paper page
  13. hazer-rau, 2020, The uulmMAC database—A multimodal affective corpus for affective computing in human-computer interaction, SENSORS, 20, 2308, 10.3390/s20082308 · DOI
  14. 10.1109/TENCON.2016.7847950 · DOI
  15. masui, 2020, Continuous estimation of emotional change using multimodal affective responses, Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR) Workshops, 290
  16. 10.1109/ICIMIA.2017.7975659 · doi:10.1109/icimia.2017.7975659
  17. 10.1109/BDCloud-SocialCom-SustainCom.2016.29 · DOI
  18. 10.1109/MIS.2019.2925204 · DOI
  19. 10.1109/TNN.2010.2091281 · DOI · Paper page
  20. 10.1109/ICBDA.2017.8078794 · DOI
  21. 10.1109/TCYB.2016.2633306 · DOI · Paper page
  22. 10.1109/JSTSP.2017.2764438 · doi:10.1109/jstsp.2017.2764438 · Paper page
  23. 2020, Seeds Dataset
  24. tipping, 2001, Sparse Bayesian learning and the relevance vector machine, J Mach Learn Res, 1, 211
  25. 10.1145/2911996.2912059 · DOI
  26. 10.1109/CVPR.1993.1626170 · DOI
  27. 10.1145/2070481.2070487 · DOI
  28. 10.1007/3-540-45014-9_5 · doi:10.1007/3-540-45014-9_5
  29. reuderink, 2008, Decision-level fusion for audio-visual laughter detection, Machine Learning for Multimodal Interaction (MLMI), 5237, 137, 10.1007/978-3-540-85853-9_13 · DOI
  30. 10.1109/CIHLI.2013.6613272 · DOI
  31. 10.1109/TCYB.2018.2797176 · DOI · Paper page
  32. 10.1109/ICCP.2018.8516437 · DOI
  33. 10.1080/10447318.2016.1159799 · DOI
  34. 10.1109/T-AFFC.2011.15 · DOI · Paper page
  35. 10.1109/BIBM.2015.7359713 · DOI
  36. wang, 2020, A review of emotion sensing: Categorization models and algorithms, Multimedia Tools Appl, 1
  37. 10.1109/URAI.2018.8441795 · DOI
  38. 10.1109/CSICC.2009.5349653 · DOI
  39. 10.1109/ACCESS.2019.2914872 · DOI
  40. 10.1109/TechSym.2016.7872646 · DOI
  41. 10.1109/TAFFC.2015.2411280 · DOI
  42. 2020, LUMED-2 Dataset
  43. 2020, FacesDB
  44. 10.1080/02699930903485076 · DOI · Paper page
  45. 10.1016/j.jneumeth.2010.04.028 · DOI · Paper page
  46. 2020, DEAP Dataset
  47. 2020, Pretrained Deep Neural Networks
  48. 2020, IMOTIONS
  49. 2020, Top 14 EEG Hardware companies [Ranked]
  50. 10.1109/ACIIAsia.2018.8470379 · DOI
  51. kim, 2005, Integrating information from speech and physiological signals to achieve emotional sensitivity, Proc INTERSPEECH, 809, 10.21437/Interspeech.2005-380 · DOI
  52. 10.1109/CVPR.2016.374 · DOI · Paper page
  53. syeda, 2017, Cross-subject continuous emotion recognition using speech and body motion in dyadic interactions, Proc INTERSPEECH, 1731
  54. 10.1007/s10579-015-9300-0 · DOI
  55. 10.1186/s12911-017-0562-x · DOI
  56. szegedy, 2017, Inception-v4, inception-resnet and the impact of residual connections on learning, Proc AAAI, 4, 1
  57. 10.1515/sh-2016-0011 · DOI
  58. 2020, Empatica e4
  59. cimtay, 2020, Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition, SENSORS, 20, 2034, 10.3390/s20072034 · DOI · Paper page
  60. 10.1109/ICASSP.2013.6637858 · DOI
  61. zhang, 2019, Cross-subject EEG-based emotion recognition with deep domain confusion, Intelligent Robotics and Applications (ICIRA), 11740, 558, 10.1007/978-3-030-27526-6_49 · DOI
  62. 10.1088/1742-6596/705/1/012051 · DOI
  63. 10.1162/089976698300017467 · DOI · Paper page
  64. 10.1109/FSKD.2016.7603437 · DOI
  65. gong, 2012, Geodesic flow kernel for unsupervised domain adaptation, Proc IEEE Conf Comput Vis Pattern Recognit, 2066
  66. shi, 2012, Information-theoretical learning of discriminative clusters for unsupervised domain adaptation, Proc Intl Conf on Machine Learning (ICML) 2012, 1275
  67. 10.1109/ICCV.2013.368 · DOI · Paper page
  68. 10.3389/fncom.2019.00053 · DOI
  69. 10.1016/j.jksuci.2019.11.003 · DOI · Paper page
  70. 10.1109/JSEN.2018.2883497 · DOI · Paper page
  71. 10.3389/fnbot.2017.00019 · DOI · Paper page
  72. 10.1037/0022-3514.53.4.712 · DOI · Paper page
  73. 10.1109/T-AFFC.2012.36 · DOI
  74. fabien, 2015, AVEC 2015: The first affect recognition challenge bridging across audio, video, and physiological data, Proc of International Audio/Visual Emotion Challenge and Workshop, 3
  75. 10.1109/ACCESS.2019.2949741 · DOI · Paper page
  76. 10.1109/ACCESS.2017.2672722 · DOI
  77. 10.1109/ACCESS.2019.2928691 · DOI · Paper page
  78. 10.3390/e18090272 · DOI · Paper page
  79. osman, 2016, Multimodal Affect Recognition Current Approaches and Challenges
  80. 10.1016/j.knosys.2013.02.014 · DOI · Paper page
  81. arns, 2013, A decade of EEG theta/beta ratio research in ADHD a meta-analysis, 17, 374
  82. 10.1145/3332169.3333569 · DOI
  83. 10.1093/brain/awy251 · DOI · Paper page
  84. 10.1016/j.neuroimage.2006.09.024 · DOI · Paper page
  85. thejaswini, 2019, Analysis of EEG based emotion detection of DEAP and SEED-IV databases using SVM, International Journal of Recent Technology and Engineering, 8, 207
  86. 10.1371/journal.pone.0174949 · DOI · Paper page
  87. 10.3390/s130810273 · DOI · Paper page
  88. 10.1371/journal.pone.0214507 · DOI
  89. 10.1109/ICSPIS.2017.8311585 · DOI

Cited by

See all 200

Metadata

Authors
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S.
Venue
IEEE Access
Published
2020
Volume
8
Pages
168865-168878
ISSN
2169-3536 (ISSN-L)
OpenAlex
W3084624816
Cited by
200
References
18
FWCI
18.51
Type
journal article
Language
en
Open access
gold
Funders
  • Institutional Links grant (352175665)
  • Newton–Katip Celebi partnership between the U.K. and Turkey
  • U.K. Department of Business, Energy and Industrial Strategy (BEIS) and The Scientific and Technological Research Council of Turkey (TUBITAK) and delivered by the British Council

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion. https://app.cochat.ai/papers/10.1109/access.2020.3023871
The paper
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020). Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion. IEEE Access. https://doi.org/10.1109/access.2020.3023871

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.