CoChat
Sign Up
PreprintCited by 14Neuroscience and Neural EngineeringEEG and Brain-Computer InterfacesTactile and Sensory Interactions

Hybrid Neural Autoencoders for Stimulus Encoding in Visual and Other Sensory Neuroprostheses

Open access
Granley, Jacob, Relic, Lucas, Beyeler, Michael·2022·arXiv
DOI 10.48550/arxiv.2205.13623arXiv 2205.13623
Open this paper in CoChat — free accountOpen PDF

Abstract

Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capabilities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus to produce a desired response? Here, we frame this as an end-to-end optimization problem, where a deep neural network stimulus encoder is trained to invert a known and fixed forward model that approximates the underlying biological system. As a proof of concept, we demonstrate the effectiveness of this Hybrid Neural Autoencoder (HNA) in visual neuroprostheses. We find that HNA produces high-fidelity patient-specific stimuli representing handwritten digits and segmented images of everyday objects, and significantly outperforms conventional encoding strategies across all simulated patients. Overall this is an important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove a promising solution for a variety of neuroprosthetic technologies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2205.13623

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • To find the stimulus that will elicit a desired response, one essentially needs to find the inverse of the forward model, which can be achieved in a number of ways.

    “To find the stimulus that will elicit a desired response, one essentially needs to find the inverse of the forward model, which can be achieved in a number of ways (Spencer et al., 2019; Fauvel and Chalk, 2022; Granley et al., 2022).”

    From Towards a <i>Smart Bionic Eye</i> : AI-powered artificial vision for the treatment of incurable blindness (2022) · cites this paper as Granley et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 14

Research works printed but not linked: 56

  1. [2][2] Eduardo Fernandez. Development of visual Neuroprostheses: trends and challenges. Bioelectronic Medicine , 4(1):12, August 2018. -
  2. [3][3] Blake S. Wilson, Charles C. Finley, Dewey T. Lawson, Robert D. Wolford, Donald K. Eddington, and William M. Rabinowitz. Better speech recognition with cochlear implants. Nature , 352(6332):236-238, July 1991. Number: 6332 Publisher: Nature Publishing Group. -
  3. [4][4] Eduardo Fernández, Arantxa Alfaro, Cristina Soto-Sánchez, Pablo Gonzalez-Lopez, Antonio M. Lozano, Sebastian Peña, Maria Dolores Grima, Alfonso Rodil, Bernardeta Gómez, Xing Chen, Pieter R. Roelfsema, John D. Rolston, Tyler S. Davis, and Richard A. Normann. Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex. Journal of Clinical Investigation , 131(23):e151331, December 2021. -
  4. [6][6] Daniel W. Tan, Matthew A. Schiefer, Michael W. Keith, James Robert Anderson, Joyce Tyler, and Dustin J. Tyler. A neural interface provides long-term stable natural touch perception. Science Translational Medicine , 6(257):257ra138-257ra138, October 2014. Publisher: American Association for the Advancement of Science. -
  5. [7][7] Cordelia Erickson-Davis and Helma Korzybska. What do blind people 'see' with retinal prostheses? Observations and qualitative reports of epiretinal implant users. PLOS ONE , 16(2):e0229189, February 2021. Publisher: Public Library of Science. -
  6. [8][8] Craig D. Murray. Embodiment and Prosthetics. In Pamela Gallagher, Deirdre Desmond, and Malcolm MacLachlan, editors, Psychoprosthetics , pages 119-129. Springer, London, 2008. -
  7. [10][10] Lucas Relic, Bowen Zhang, Yi-Lin Tuan, and Michael Beyeler. Deep Learning-Based Perceptual Stimulus Encoder for Bionic Vision. In Augmented Humans 2022 , AHs 2022, pages 323-325, New York, NY, USA, March 2022. Association for Computing Machinery. -
  8. [11][11] David Montes de Oca Zapiain, James A. Stewart, and Rémi Dingreville. Accelerating phase-field-based microstructure evolution predictions via surrogate models trained by machine learning methods. npj Computational Materials , 7(1):1-11, January 2021. Number: 1 Publisher: Nature Publishing Group. -
  9. [12][12] Mohammad Amin Nabian and Hadi Meidani. A Deep Neural Network Surrogate for High-Dimensional Random Partial Differential Equations. Probabilistic Engineering Mechanics , 57:14-25, July 2019. arXiv:1806.02957 [physics, stat]. - · arXiv
  10. [14][14] Gregg A. Tabot, John F. Dammann, Joshua A. Berg, Francesco V. Tenore, Jessica L. Boback, R. Jacob Vogelstein, and Sliman J. Bensmaia. Restoring the sense of touch with a prosthetic hand through a brain interface. Proceedings of the National Academy of Sciences , 110(45):18279-18284, November 2013. -
  11. [15][15] Marco Capogrosso, Tomislav Milekovic, David Borton, Fabien Wagner, Eduardo Martin Moraud, JeanBaptiste Mignardot, Nicolas Buse, Jerome Gandar, Quentin Barraud, David Xing, Elodie Rey, Simone Duis, Yang Jianzhong, Wai Kin D. Ko, Qin Li, Peter Detemple, Tim Denison, Silvestro Micera, Erwan Bezard, Jocelyne Bloch, and Grégoire Courtine. A brain-spine interface alleviating gait deficits after spinal cord injury in primates. Nature , 539(7628):284-288, November 2016. -
  12. [16][16] Christopher A. R. Chapman, Noah Goshi, and Erkin Seker. Multifunctional Neural Interfaces for ClosedLoop Control of Neural Activity. Advanced Functional Materials , 28(12):1703523, 2018. \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/adfm.201703523. - · DOI
  13. [17][17] Fabien B. Wagner, Jean-Baptiste Mignardot, Camille G. Le Goff-Mignardot, Robin Demesmaeker, Salif Komi, Marco Capogrosso, Andreas Rowald, Ismael Seáñez, Miroslav Caban, Elvira Pirondini, Molywan Vat, Laura A. McCracken, Roman Heimgartner, Isabelle Fodor, Anne Watrin, Perrine Seguin, Edoardo Paoles, Katrien Van Den Keybus, Grégoire Eberle, Brigitte Schurch, Etienne Pralong, Fabio Becce, John Prior, Nicholas Buse, Rik Buschman, Esra Neufeld, Niels Kuster, Stefano Carda, Joachim von Zitzewitz, Vincent Delattre, Tim Denison, Hendrik Lambert, Karen Minassian, Jocelyne Bloch, and Grégoire Courtine. Targeted neurotechnology restores walking in humans with spinal cord injury. Nature , 563(7729):65-71, November 2018. -
  14. [18][18] Lauren N. Ayton, Nick Barnes, Gislin Dagnelie, Takashi Fujikado, Georges Goetz, Ralf Hornig, Bryan W. Jones, Mahiul M. K. Muqit, Daniel L. Rathbun, Katarina Stingl, James D. Weiland, and Matthew A. Petoe. An update on retinal prostheses. Clinical Neurophysiology , 131(6):1383-1398, June 2020. -
  15. [20][20] Lewis Karapanos, Carla J. Abbott, Lauren N. Ayton, Maria Kolic, Myra B. McGuinness, Elizabeth K. Baglin, Samuel A. Titchener, Jessica Kvansakul, Dean Johnson, William G. Kentler, Nick Barnes, David A. X. Nayagam, Penelope J. Allen, and Matthew A. Petoe. Functional Vision in the Real-World Environment With a Second-Generation (44-Channel) Suprachoroidal Retinal Prosthesis. Translational Vision Science &amp; Technology , 10(10):7-7, August 2021. Publisher: The Association for Research in Vision and Ophthalmology. -
  16. [21][21] WmH. Dobelle. Artificial Vision for the Blind by Connecting a Television Camera to the Visual Cortex. ASAIO Journal , 46(1):3-9, February 2000. -
  17. [22][22] Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. A model of ganglion axon pathways accounts for percepts elicited by retinal implants. Scientific Reports , 9(1):1-16, June 2019. -
  18. [23][23] J. F. Rizzo, J. Wyatt, J. Loewenstein, S. Kelly, and D. Shire. Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Invest Ophthalmol Vis Sci , 44(12):5362-9, December 2003. -
  19. [24][24] Jacob Granley and Michael Beyeler. A Computational Model of Phosphene Appearance for Epiretinal Prostheses. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC) , pages 4477-4481, November 2021. ISSN: 2694-0604. -
  20. [25][25] Yvonne H-L. Luo, Joe Jiangjian Zhong, Monica Clemo, and Lyndon da Cruz. Long-term Repeatability and Reproducibility of Phosphene Characteristics in Chronically Implanted Argus II Retinal Prosthesis Subjects. American Journal of Ophthalmology , 170:100-109, October 2016. -
  21. [26][26] M. Beyeler, A. Rokem, G. M. Boynton, and I. Fine. Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies. J Neural Eng , 14(5):051003, June 2017. -
  22. [27][27] S. C. Chen, G. J. Suaning, J. W. Morley, and N. H. Lovell. Simulating prosthetic vision: I. Visual models of phosphenes. Vision Research , 49(12):1493-506, June 2009. -
  23. [29][29] Nishal P. Shah, Sasidhar Madugula, Lauren Grosberg, Gonzalo Mena, Pulkit Tandon, Pawel Hottowy, Alexander Sher, Alan Litke, Subhasish Mitra, and E.J. Chichilnisky. Optimization of Electrical Stimulation for a High-Fidelity Artificial Retina. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER) , pages 714-718, March 2019. ISSN: 1948-3554. -
  24. [30][30] Dorsa Haji Ghaffari, Yao-Chuan Chang, Ehsan Mirzakhalili, and James D. Weiland. Closed-loop Optimization of Retinal Ganglion Cell Responses to Epiretinal Stimulation: A Computational Study. In 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER) , pages 597-600, May 2021. ISSN: 1948-3554. -
  25. [31][31] Tristan Fauvel and Matthew Chalk. Human-in-the-loop optimization of visual prosthetic stimulation. preprint, Neuroscience, November 2021. -
  26. [32][32] Martin J. Spencer, Tatiana Kameneva, David B. Grayden, Hamish Meffin, and Anthony N. Burkitt. Global activity shaping strategies for a retinal implant. Journal of Neural Engineering , 16(2):026008, January 2019. Publisher: IOP Publishing. -
  27. [34][34] Robert E Marc, Bryan W Jones, Carl B Watt, and Enrica Strettoi. Neural remodeling in retinal degeneration. Progress in Retinal and Eye Research , 22(5):607-655, 2003. -
  28. [35][35] Zhou Wang and Alan C. Bovik. Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures. IEEE Signal Processing Magazine , 26(1):98-117, January 2009. Conference Name: IEEE Signal Processing Magazine. -
  29. [36][36] Guangtao Zhai and Xiongkuo Min. Perceptual image quality assessment: a survey. Science China Information Sciences , 63(11):211301, November 2020. -
  30. [37][37] Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Technical Report arXiv:1502.03167, arXiv, March 2015. arXiv:1502.03167 [cs] type: article. - · arXiv
  31. [38][38] Michael F Dorman, Anthony J Spahr, Philipos C Loizou, Cindy J Dana, and Jennifer S Schmidt. Acoustic simulations of combined electric and acoustic hearing (eas). Ear and Hearing , 26(4):371-380, 2005. -
  32. [39][39] Mario A Svirsky, Nai Ding, Elad Sagi, Chin-Tuan Tan, Matthew Fitzgerald, E Katelyn Glassman, Keena Seward, and Arlene C Neuman. Validation of acoustic models of auditory neural prostheses. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 8629-8633. IEEE, 2013. -
  33. [40][40] MF Dorman, PC Loizou, A Spahr, and CJ Dana. Simulations of combined acoustic/electric hearing. In Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No. 03CH37439) , volume 3, pages 1999-2001. IEEE, 2003. -
  34. [41][41] Michael F Dorman, Philipos C Loizou, and Dawne Rainey. Speech intelligibility as a function of the number of channels of stimulation for signal processors using sine-wave and noise-band outputs. The Journal of the Acoustical Society of America , 102(4):2403-2411, 1997. -
  35. [42][42] William B Cooper, Emily Tobey, and Philipos C Loizou. Music perception by cochlear implant and normal hearing listeners as measured by the montreal battery for evaluation of amusia. Ear and hearing , 29(4):618, 2008. -
  36. [43][43] Philipos C Loizou, Michael Dorman, Oguz Poroy, and Tony Spahr. Speech recognition by normal-hearing and cochlear implant listeners as a function of intensity resolution. The Journal of the Acoustical Society of America , 108(5):2377-2387, 2000. -
  37. [44][44] Hannes P Saal, Benoit P Delhaye, Brandon C Rayhaun, and Sliman J Bensmaia. Simulating tactile signals from the whole hand with millisecond precision. Proceedings of the National Academy of Sciences , 114(28):E5693-E5702, 2017. -
  38. [45][45] Elizaveta V Okorokova, Qinpu He, and Sliman J Bensmaia. Biomimetic encoding model for restoring touch in bionic hands through a nerve interface. Journal of neural engineering , 15(6):066033, 2018. -
  39. [46][46] Douglas J Weber, Rebecca Friesen, and Lee E Miller. Interfacing the somatosensory system to restore touch and proprioception: essential considerations. Journal of motor behavior , 44(6):403-418, 2012. -
  40. [47][47] Sung Soo Kim, Arun P Sripati, and Sliman J Bensmaia. Predicting the timing of spikes evoked by tactile stimulation of the hand. Journal of neurophysiology , 104(3):1484-1496, 2010. -
  41. [48][48] Milana P Mileusnic, Ian E Brown, Ning Lan, and Gerald E Loeb. Mathematical models of proprioceptors. i. control and transduction in the muscle spindle. Journal of neurophysiology , 96(4):1772-1788, 2006. -
  42. [49][49] N. M. Jansonius, J. Nevalainen, B. Selig, L. M. Zangwill, P. A. Sample, W. M. Budde, J. B. Jonas, W. A. Lagrèze, P. J. Airaksinen, R. Vonthein, L. A. Levin, J. Paetzold, and U. Schiefer. A mathematical description of nerve fiber bundle trajectories and their variability in the human retina. Vision Research , 49(17):2157-2163, August 2009. -
  43. [51][51] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. Technical Report arXiv:1409.1556, arXiv, April 2015. arXiv:1409.1556 [cs] type: article. - · arXiv
  44. [52][52] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal Style Transfer via Feature Transforms. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. -
  45. [53][53] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, June 2009. ISSN: 1063-6919. -
  46. [54][54] Sylvain Paris, Samuel W Hasinoff, and Jan Kautz. Local Laplacian Filters: Edge-aware Image Processing with a Laplacian Pyramid. Communications of the ACM , 58:11, 2015. -
  47. [55][55] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012. -
  48. [56][56] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. Technical Report arXiv:1405.0312, arXiv, February 2015. arXiv:1405.0312 [cs] type: article. - · arXiv
  49. [57][57] Nicole Han, Sudhanshu Srivastava, Aiwen Xu, Devi Klein, and Michael Beyeler. Deep Learning-Based Scene Simplification for Bionic Vision. In Augmented Humans Conference 2021 , AHs'21, pages 45-54, New York, NY, USA, February 2021. Association for Computing Machinery. -
  50. [58][58] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008. -
  51. [59][59] Eli Peli. Testing Vision Is Not Testing For Vision. Translational Vision Science &amp; Technology , 9(13):32-32, December 2020. Publisher: The Association for Research in Vision and Ophthalmology. -
  52. [60][60] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204 , 2017. - · arXiv
  53. [61][61] Alim Louis Benabid. Deep brain stimulation for Parkinson's disease. Current Opinion in Neurobiology , 13(6):696-706, December 2003. -
  54. [62][62] Bardia Bozorgzadeh, Douglas R. Schuweiler, Martin J. Bobak, Paul A. Garris, and Pedram Mohseni. Neurochemostat: A Neural Interface SoC With Integrated Chemometrics for Closed-Loop Regulation of Brain Dopamine. IEEE Transactions on Biomedical Circuits and Systems , 10(3):654-667, June 2016. Conference Name: IEEE Transactions on Biomedical Circuits and Systems. -
  55. [63][63] M. Beyeler, G. M. Boynton, I. Fine, and A. Rokem. pulse2percept: A Python-based simulation framework for bionic vision. In K. Huff, D. Lippa, D. Niederhut, and M. Pacer, editors, Proceedings of the 16th Science in Python Conference , pages 81-88, 2017. -
  56. [64][64] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. · arXiv

Authors
Granley, J., Relic, L., & Beyeler, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2205.13623
arXiv
2205.13623
OpenAlex
W4307001839
Cited by
14
References
0
Citations by year
2025: 3 · 2024: 7 · 2023: 3 · 2022: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neuroscience and Neural Engineering
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
visual prosthesis, stimulus encoding, inverse problems, forward model, blindness, end-to-end optimization, deep neural network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review