The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Deep Learning Human Mind for Automated Visual Classification
Open accessAbstract
xWhat if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories in a reading the mind effort. Afterward, we transfer the learned capabilities to machines by training a Convolutional Neural Network (CNN)-based regressor to project images onto the learned manifold, thus allowing machines to employ human brain-based features for automated visual classification. We use a 128-channel EEG with active electrodes to record brain activity of several subjects while looking at images of 40 ImageNet object classes. The proposed RNN-based approach for discriminating object classes using brain signals reaches an average accuracy of about 83%, which greatly outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain-driven approach obtains competitive performance, comparable to those achieved by powerful CNN models and it is also able to generalize over different visual datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Electroencephalography—Basic principles, clinical applications and related fields
(1982) · Journal of the Autonomic Nervous System · doi:10.1016/0165-1838(82)90067-4 · Cited by 3,427
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029
EEG signal classification using PCA, ICA, LDA and support vector machines
Subasi, A., & Ismail Gursoy, M. (2010) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.06.065 · Cited by 1,209
Convolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces
Cecotti, H., & Graser, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.125 · Cited by 778
The neural basis of visual body perception
Peelen, M. V., & Downing, P. E. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2195 · Cited by 697
Principles and Practice of Clinical Electrophysiology of Vision
(2006) · The MIT Press eBooks · doi:10.7551/mitpress/5557.001.0001 · Cited by 685
Cortical Regions Involved in Perceiving Object Shape
Kourtzi, Z., & Kanwisher, N. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-09-03310.2000 · Cited by 673
Representational dynamics of object vision: The first 1000 ms
Carlson, T., Tovar, D. A., Alink, A., & Kriegeskorte, N. (2013) · Journal of Vision · doi:10.1167/13.10.1 · Cited by 414
Classification of patterns of EEG synchronization for seizure prediction
Mirowski, P., Madhavan, D., LeCun, Y., & Kuzniecky, R. (2009) · Clinical Neurophysiology · doi:10.1016/j.clinph.2009.09.002 · Cited by 380
Op de Beeck, H. P., Torfs, K., & Wagemans, J. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.2511-08.2008 · Cited by 266
High temporal resolution decoding of object position and category
Carlson, T. A., Hogendoorn, H., Kanai, R., Mesik, J., & Turret, J. (2011) · Journal of Vision · doi:10.1167/11.10.9 · Cited by 185
Kaneshiro, B., Perreau Guimaraes, M., Kim, H.-S., Norcia, A. M., & Suppes, P. (2015) · PLOS ONE · doi:10.1371/journal.pone.0135697 · Cited by 180
Brain Activity-Based Image Classification From Rapid Serial Visual Presentation
Bigdely-Shamlo, N., Vankov, A., Ramirez, R. R., & Makeig, S. (2008) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2008.2003381 · Cited by 177
Identifying Object Categories from Event-Related EEG: Toward Decoding of Conceptual Representations
Simanova, I., van Gerven, M., Oostenveld, R., & Hagoort, P. (2010) · PLoS ONE · doi:10.1371/journal.pone.0014465 · Cited by 167
Single-trial classification of EEG in a visual object task using ICA and machine learning
Stewart, A. X., Nuthmann, A., & Sanguinetti, G. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2014.02.014 · Cited by 148
Deep Feature Learning for EEG RecordingsPreprint
Stober, S., Sternin, A., Owen, A. M., & Grahn, J. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04306 · Cited by 143
Das, K., Giesbrecht, B., & Eckstein, M. P. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.03.030 · Cited by 78
Wang, C., Xiong, S., Hu, X. P., Yao, L., & Zhang, J. (2012) · Journal of Neural Engineering · doi:10.1088/1741-2560/9/5/056013 · Cited by 75
Combining brain computer interfaces with vision for object categorization
Kapoor, A., Shenoy, P., & Tan, D. S. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587618 · Cited by 75
Human-aided computing: utilizing implicit human processing to classify images
Shenoy, P., & Tan, D. S. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357188 · Cited by 66
Research works printed but not linked: 1
- bashivan, 2016, Learning representations from EEG with deep recurrent-convolutional neural networks, ICLR 2016
Cited by
Deep learning-based electroencephalography analysis: a systematic review
Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab260c · Cited by 1,532
Deep learning for healthcare applications based on physiological signals: A review
Faust, O., Hagiwara, Y., Hong, T. J., Lih, O. S., & Acharya, U. R. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.04.005 · Cited by 1,026
Gu, X., Cao, Z., Jolfaei, A., Xu, P., Wu, D., Jung, T.-P., & Lin, C.-T. (2021) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3052811 · Cited by 395
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296
Brain-Computer Interface: Advancement and Challenges
Mridha, M. F., Das, S. C., Kabir, M. M., Lima, A. A., Islam, M. R., & Watanobe, Y. (2021) · Sensors · doi:10.3390/s21175746 · Cited by 263
Metadata
- Authors
- Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., & Shah, M.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 4503-4511
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2510506994
- Cited by
- 289
- References
- 25
- FWCI
- 35.72
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.