Generative Adversarial Networks Conditioned by Brain Signals
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Recent advancements in generative adversarial networks (GANs), using deep convolutional models, have supported the development of image generation techniques able to reach satisfactory levels of realism. Further improvements have been proposed to condition GANs to generate images matching a specific object category or a short text description. In this work, we build on the latter class of approaches and investigate the possibility of driving and conditioning the image generation process by means of brain signals recorded, through an electroencephalograph (EEG), while users look at images from a set of 40 ImageNet object categories with the objective of generating the seen images. To accomplish this task, we first demonstrate that brain activity EEG signals encode visually-related information that allows us to accurately discriminate between visual object categories and, accordingly, we extract a more compact class-dependent representation of EEG data using recurrent neural networks. Afterwards, we use the learned EEG manifold to condition image generation employing GANs, which, during inference, will read EEG signals and convert them into images. We tested our generative approach using EEG signals recorded from six subjects while looking at images of the aforementioned 40 visual classes. The results show that for classes represented by well-defined visual patterns (e.g., pandas, airplane, etc.), the generated images are realistic and highly resemble those evoking the EEG signals used for conditioning GANs, resulting in an actual reading-the-mind process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011) · Current Biology · doi:10.1016/j.cub.2011.08.031 · Cited by 933 · cited 2×
“In a pioneering work [18], the authors try to generate impressions of what the subjects see based on fMRI images by imposing a prior built on a large image dataset extracted from YouTube.”
BackgroundBrain-Controlled Interfaces: Movement Restoration with Neural Prosthetics
Schwartz, A. B., Cui, X. T., Weber, D. J., & Moran, D. W. (2006) · Neuron · doi:10.1016/j.neuron.2006.09.019 · Cited by 792
Convolutional Neural Networks for P300 Detection with Application to Brain-Computer Interfaces
Cecotti, H., & Graser, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.125 · Cited by 778
Learning to generate chairs with convolutional neural networks
Dosovitskiy, A., Springenberg, J. T., & Brox, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298761 · Cited by 640 · cited 1×
“, a single bit in the vector indicates the class to generate), words [32, 23] or arbitrary real number vectors representing geometric transformations and coordinates in a 3D space [5].”
BackgroundReconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · cited 1×
“In [20], the authors describe an approach to recreate (partially) speech stimuli based on human auditory cortex data, acquired by cortical surface electrode arrays.”
BackgroundfMRI–EEG integrated cortical source imaging by use of time-variant spatial constraints
Liu, Z., & He, B. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.10.003 · Cited by 103 · cited 1×
“Additionally, to compensate for the low temporal resolution of the fMRI scanners, we are going to investigate methods to combine fMRI data with EEG data [12].”
Method
Cited by
Deep learning-based electroencephalography analysis: a systematic review
Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab260c · Cited by 1,532
“Music semantics [140] Personal trait/attribute Person identification [223, 221] Sex [188] Generation of data Data augmentation [192, 219, 152] Generating EEG [66] Spatial upsampling [33] Generating images conditioned on EEG [92] [128]”
BackgroundGu, X., Cao, Z., Jolfaei, A., Xu, P., Wu, D., Jung, T.-P., & Lin, C.-T. (2021) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3052811 · Cited by 395
Epilepsy Seizure Prediction on EEG Using Common Spatial Pattern and Convolutional Neural Network
Zhang, Y., Guo, Y., Yang, P., Chen, W., & Lo, B. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2933046 · Cited by 319
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296
Augmenting the size of EEG datasets using generative adversarial networks
Abdelfattah, S. M., Abdelrahman, G. M., & Wang, M. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489727 · Cited by 122
Metadata
- Authors
- Palazzo, S., Spampinato, C., Kavasidis, I., Giordano, D., & Shah, M.
- Venue
- 2017 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 3430-3438
- OpenAlex
- W2776446922
- Cited by
- 135
- References
- 9
- FWCI
- 21.25
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.