Single-trial EEG RSVP classification using convolutional neural networks
Abstract
Traditionally, Brain-Computer Interfaces (BCI) have been explored as a means to return function to paralyzed or otherwise debilitated individuals. An emerging use for BCIs is in human-autonomy sensor fusion where physiological data from healthy subjects is combined with machine-generated information to enhance the capabilities of artificial systems. While human-autonomy fusion of physiological data and computer vision have been shown to improve classification during visual search tasks, to date these approaches have relied on separately trained classification models for each modality. We aim to improve human-autonomy classification performance by developing a single framework that builds codependent models of human electroencephalograph (EEG) and image data to generate fused target estimates. As a first step, we developed a novel convolutional neural network (CNN) architecture and applied it to EEG recordings of subjects classifying target and non-target image presentations during a rapid serial visual presentation (RSVP) image triage task. The low signal-to-noise ratio (SNR) of EEG inherently limits the accuracy of single-trial classification and when combined with the high dimensionality of EEG recordings, extremely large training sets are needed to prevent overfitting and achieve accurate classification from raw EEG data. This paper explores a new deep CNN architecture for generalized multi-class, single-trial EEG classification across subjects. We compare classification performance from the generalized CNN architecture trained across all subjects to the individualized XDAWN, HDCA, and CSP neural classifiers which are trained and tested on single subjects. Preliminary results show that our CNN meets and slightly exceeds the performance of the other classifiers despite being trained across subjects.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Updating P300: An integrative theory of P3a and P3b
Polich, J. (2007) · Clinical Neurophysiology · doi:10.1016/j.clinph.2007.04.019 · Cited by 7,919 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Farwell, L. A., & Donchin, E. (1988) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(88)90149-6 · Cited by 3,228 · see citing works
Evoked-potential correlates of stimulus uncertainty
Sutton, S., Braren, M., Zubin, J., & John, E. R. (1965) · Science · doi:10.1126/science.150.3700.1187 · Cited by 3,061 · see citing works
Optimal spatial filtering of single trial EEG during imagined hand movement
Ramoser, H., Muller-Gerking, J., & Pfurtscheller, G. (2000) · IEEE Transactions on Rehabilitation Engineering · doi:10.1109/86.895946 · Cited by 2,457 · see citing works
Stochastic Gradient Descent Tricks
Bottou, L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_25 · Cited by 2,097 · see citing works
Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans
Wolpaw, J. R., & McFarland, D. J. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0403504101 · Cited by 1,512 · see citing works
The mental prosthesis: assessing the speed of a P300-based brain-computer interface
Donchin, E., Spencer, K., & Wijesinghe, R. S. (2000) · IEEE Transactions on Rehabilitation Engineering · doi:10.1109/86.847808 · Cited by 1,217 · see citing works
Designing optimal spatial filters for single-trial EEG classification in a movement task
Müller-Gerking, J., Pfurtscheller, G., & Flyvbjerg, H. (1999) · Clinical Neurophysiology · doi:10.1016/s1388-2457(98)00038-8 · Cited by 846 · see citing works
xDAWN Algorithm to Enhance Evoked Potentials: Application to Brain–Computer Interface
Rivet, B., Souloumiac, A., Attina, V., & Gibert, G. (2009) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2009.2012869 · Cited by 537 · see citing works
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334 · see citing works
Cortically coupled computer vision for rapid image search
Gerson, A. D., Parra, L. C., & Sajda, P. (2006) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2006.875550 · Cited by 258 · see citing works
Convolutional Neural Network for Multi-Category Rapid Serial Visual Presentation BCI
Manor, R., & Geva, A. B. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00146 · Cited by 144 · see citing works
A robust sensor-selection method for P300 brain–computer interfaces
Cecotti, H., Rivet, B., Congedo, M., Christian Jutten, B., Bertrand, O., Maby, E., & Mattout, J. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/1/016001 · Cited by 106 · see citing works
Foundations of Augmented Cognition
Schmorrow, D., & Fidopiastis, C. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39454-6 · Cited by 48 · see citing works
MCA-NMF: Multimodal Concept Acquisition with Non-Negative Matrix Factorization
Mangin, O., Filliat, D., ten Bosch, L., & Oudeyer, P. (2015) · PLoS ONE · doi:10.1371/journal.pone.0140732 · Cited by 21 · see citing works
Human-autonomy sensor fusion for rapid object detection
Robinson, R. M., Lee, H., McCourt, M. J., Marathe, A. R., Kwon, H., Ton, C. T., & Nothwang, W. D. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353390 · Cited by 18 · see citing works
Self-citationA New Single Trial P300 Classification Method
Li, K., Sankar, R. T., Cao, K., Arbel, Y., & Donchin, E. (2012) · International Journal of E-Health and Medical Communications · doi:10.4018/jehmc.2012100103 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 21 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces
Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. P., & Lance, B. J. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aace8c · Cited by 5,115 · see citing works
“We believe models that fall in (1), such as EEGNet and others [28, 30, 31], offer the best tradeoff between input dimensionality and the flexibility to discover relevant features by providing all available channels.”
MethodSelf-citationDeep learning with convolutional neural networks for EEG decoding and visualization
Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., & Ball, T. (2017) · Human Brain Mapping · doi:10.1002/hbm.23730 · Cited by 3,751 · see citing works
Deep learning for electroencephalogram (EEG) classification tasks: a review
Craik, A., He, Y., & Contreras-Vidal, J. L. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab0ab5 · Cited by 1,758 · see citing works
Deep learning-based electroencephalography analysis: a systematic review
Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab260c · Cited by 1,532 · see citing works
Data augmentation for deep-learning-based electroencephalography
Lashgari, E., Liang, D., & Maoz, U. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108885 · Cited by 398 · see citing works
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296 · see citing works
Deep learning-based EEG emotion recognition: Current trends and future perspectives
Wang, X., Ren, Y., Luo, Z., He, W., Hong, J. S., & Huang, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1126994 · Cited by 161 · see citing works
Borra, D., Fantozzi, S., & Magosso, E. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.05.032 · Cited by 142 · see citing works
Learning joint space–time–frequency features for EEG decoding on small labeled data.
Zhao, D., Tang, F., Si, B., & Feng, X. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.02.009 · Cited by 123 · see citing works
Deep learning for EEG data analytics: A survey
Li, G., Lee, C. H., Jung, J. J., Youn, Y. C., & Camacho, D. (2019) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5199 · Cited by 122 · see citing works
Zang, B., Lin, Y., Liu, Z., & Gao, X. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac1610 · Cited by 54 · see citing works
A systematic comparison of deep learning methods for EEG time series analysis
Walther, D., Viehweg, J., Haueisen, J., & Mäder, P. (2023) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2023.1067095 · Cited by 53 · see citing works
Opałka, S., Stasiak, B., Szajerman, D., & Wojciechowski, A. (2018) · Sensors · doi:10.3390/s18103451 · Cited by 40 · see citing works
Hartmann, K. G., Schirrmeister, R. T., & Ball, T. (2018) · International Winter Conference on Brain-Computer Interface (BCI) · doi:10.1109/iww-bci.2018.8311493 · Cited by 35 · see citing works
Cross-modal guiding and reweighting network for multi-modal RSVP-based target detection
Mao, J., Qiu, S., Wei, W., & He, H. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.009 · Cited by 29 · see citing works
Automated visual stimuli evoked multi-channel EEG signal classification using EEGCapsNet
Kumari, N., Anwar, S., & Bhattacharjee, V. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.11.019 · Cited by 28 · see citing works
Decoding P300 Variability Using Convolutional Neural Networks
Solon, A. J., Lawhern, V. J., Touryan, J., McDaniel, J. R., Ries, A. J., & Gordon, S. M. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00201 · Cited by 28 · see citing works
Self-citationDC-tCNN: A Deep Model for EEG-Based Detection of Dim Targets
Fan, L., Shen, H., Xie, F., Su, J., Yu, Y., & Hu, D. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3184725 · Cited by 27 · see citing works
SSVEP-assisted RSVP brain–computer interface paradigm for multi-target classification
Ko, L., Sankar, D. S. V., Huang, Y., Lu, Y.-C., Shaw, S., & Jung, T. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abd1c0 · Cited by 21 · see citing works
A review of feature extraction and classification algorithms for image RSVP-based BCI
Wang, Z., Healy, G. F., Smeaton, A. F., & Ward, T. E. (2018) · Institution of Engineering and Technology eBooks · doi:10.1049/pbce114e_ch12 · Cited by 21 · see citing works
DeePay: deep learning decodes EEG to predict consumer’s willingness to pay for neuromarketing
Hakim, A., Golan, I., Yefet, S., & Levy, D. J. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2023.1153413 · Cited by 19 · see citing works
Learning visual stimulus-evoked EEG manifold for neural image classification
Falciglia, S., Betello, F., Russo, S., & Napoli, C. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127654 · Cited by 19 · see citing works
Deep Learning Methods for EEG Neural Classification
Nakagome, S., Craik, A., Ravindran, A. S., He, Y., Cruz-Garza, J. G., & Contreras-Vidal, J. L. (2022) · Handbook of Neuroengineering · doi:10.1007/978-981-15-2848-4_78-1 · Cited by 12 · see citing works
Convolutional Networks Outperform Linear Decoders in Predicting EMG From Spinal Cord Signals
Guo, Y., Gok, S., & Şahin, M. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00689 · Cited by 11 · see citing works
Feature Selection of Deep Learning Models for EEG-Based RSVP Target Detection
Chen, J., Mao, Z., Zheng, R., Huang, Y., & He, L. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2018edp7095 · Cited by 11 · see citing works
Convolutional Neural Network-Based Visually Evoked EEG Classification Model on MindBigData
Kumari, N., Anwar, S., & Bhattacharjee, V. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-16-1543-6_22 · Cited by 7 · see citing works
Deep Learning Methods for EEG Neural Classification
Nakagome, S., Craik, A., Ravindran, A. S., He, Y., Cruz-Garza, J. G., & Contreras-Vidal, J. L. (2023) · Handbook of Neuroengineering · doi:10.1007/978-981-16-5540-1_78 · Cited by 6 · see citing works
Chiou, N., Günal, M., Koyejo, S., Perpetuini, D., Chiarelli, A. M., Low, K. A., Fabiani, M., & Gratton, G. (2024) · Bioengineering · doi:10.3390/bioengineering11080781 · Cited by 4 · see citing works
Semantic image sorting method for RSVP presentation
Wu, H., Li, F., Chu, W., Li, Y., Niu, Y., Shi, G., Zhang, L., & Chen, Y. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad4593 · Cited by 4 · see citing works
Convolutional Networks Outperform Linear Decoders in Predicting EMG from Spinal Cord Signals
Guo, Y. P., Gok, S., & Şahin, M. (2020) · Vide Leaf, Hyderabad eBooks · doi:10.37247/pans.1.2020.15 · Cited by 3 · see citing works
Exploiting Simplified Depth Estimation for Stereo-based 2D Object Detection
Lee, Y., Lee, H., Lee, E., Kwon, H., & Bhattacharyya, S. (2022) · International Conference on Artificial Intelligence and Pattern Recognition · doi:10.1109/aipr57179.2022.10092234 · Cited by 1 · see citing works
Self-citationZhang, X. (2019) · Summit (Simon Fraser University) · Cited by 1 · see citing works
Classifying single-trial event-related potentials using xDAWN and denoising auto-encoder
Cao, S., Wu, W., & Qi, F. (2017) · Chinese Control Conference (CCC) · doi:10.23919/chicc.2017.8028248
Xu, M., Gao, X., Li, F., Zhang, B., & Chen, Y. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/ae2d9a
- Authors
- Shamwell, J., Lee, H., Kwon, H., Marathe, A. R., Lawhern, V., & Nothwang, W.
- Affiliations
- DEVCOM Army Research Laboratory (US)
- Venue
- SPIE Proceedings
- Publisher
- SPIE
- Published
- 2016-05-25
- Volume
- 9836
- Pages
- 983622
- ISSN
- 0277-786X (ISSN-L)
- OpenAlex
- W2401944605
- Cited by
- 34
- References
- 21
- Citations by year
- 2025: 1 · 2024: 3 · 2023: 5 · 2022: 3 · 2021: 3 · 2020: 5 · 2019: 7 · 2018: 5 · 2017: 2
- FWCI
- 8.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- single-trial EEG classification, rapid serial visual presentation, convolutional neural networks, brain-computer interface, cross-subject generalization, multi-class classification, common spatial pattern
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review