CoChat
Sign Up
Cited by 45Neuroscience and Neural EngineeringTactile and Sensory InteractionsVisual perception and processing mechanisms

Deep Learning–Based Scene Simplification for Bionic Vision

Open access
Han, Nicole, Srivastava, Sudhanshu, Xu, Aiwen, Klein, Devi, Beyeler, Michael·2021·Augmented Humans Conference 2021
DOI 10.1145/3458709.3458982
Open this paper in CoChat — free accountOpen PDF

Abstract

Retinal degenerative diseases cause profound visual impairment in more than 10 million people worldwide, and retinal prostheses are being developed to restore vision to these individuals. Analogous to cochlear implants, these devices electrically stimulate surviving retinal cells to evoke visual percepts (phosphenes). However, the quality of current prosthetic vision is still rudimentary. Rather than aiming to restore “natural” vision, there is potential merit in borrowing state-of-the-art computer vision algorithms as image processing techniques to maximize the usefulness of prosthetic vision. Here we combine deep learning–based scene simplification strategies with a psychophysically validated computational model of the retina to generate realistic predictions of simulated prosthetic vision, and measure their ability to support scene understanding of sighted subjects (virtual patients) in a variety of outdoor scenarios. We show that object segmentation may better support scene understanding than models based on visual saliency and monocular depth estimation. In addition, we highlight the importance of basing theoretical predictions on biologically realistic models of phosphene shape. Overall, this work has the potential to drastically improve the utility of prosthetic vision for people blinded from retinal degenerative diseases.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3458709.3458982

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A biophysically-grounded simulation model for the functional evaluation of cortical prosthetic vision remains to be developed.

    “Even though scientists increasingly realize that more realistic models of phosphene vision are required to narrow the gap between simulations and reality (Han et al., 2021; Dagnelie, 2008), a biophysically-grounded simulation model for the functional evaluation of cortical prosthetic vision remains to be developed.”

    From Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses (2024) · cites this paper as Han et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 42 · Cited by 45

Research works printed but not linked: 36

  1. W. I. Al-Atabany, T. Tong, and P. A. Degenaar. 2010. Improved content aware scene retargeting for retinitis pigmentosa patients. Biomed Eng Online 9 (Sept. 2010), 52. https://doi.org/10.1186/1475-925X-9-52 · DOI · Paper page
  2. Lauren N. Ayton, Nick Barnes, Gislin Dagnelie, Takashi Fujikado, Georges Goetz, Ralf Hornig, Bryan W. Jones, Mahiul M. K. Muqit, Daniel L. Rathbun, Katarina Stingl, James D. Weiland, and Matthew A. Petoe. 2020. An update on retinal prostheses. Clinical Neurophysiology 131, 6 (June 2020), 1383-1398. https: //doi.org/10.1016/j.clinph.2019.11.029 · DOI · Paper page
  3. Lauren N. Ayton, Peter J. Blamey, Robyn H. Guymer, Chi D. Luu, David A. X. Nayagam, Nicholas C. Sinclair, Mohit N. Shivdasani, Jonathan Yeoh, Mark F. McCombe, Robert J. Briggs, Nicholas L. Opie, Joel Villalobos, Peter N. Dimitrov, Mary Varsamidis, Matthew A. Petoe, Chris D. McCarthy, Janine G. Walker, Nick Barnes, Anthony N. Burkitt, Chris E. Williams, Robert K. Shepherd, Penelope J. Allen, and for the Bionic Vision Australia Research Consortium. 2014. First-in- Human Trial of a Novel Suprachoroidal Retinal Prosthesis. PLOS ONE 9, 12 (Dec. 2014), e115239. https://doi.org/10.1371/journal.pone.0115239 Publisher: Public Library of Science. · DOI · Paper page
  4. M. Beyeler, G. M. Boynton, I. Fine, and A. Rokem. 2017. pulse2percept: A Python- based simulation framework for bionic vision. In Proceedings of the 16th Science in Python Conference, K. Huff, D. Lippa, D. Niederhut, and M. Pacer (Eds.). 81-88. https://doi.org/10.25080/shinma-7f4c6e7-00c · DOI · Paper page
  5. Michael Beyeler, Geoffrey M. Boynton, Ione Fine, and Ariel Rokem. 2019. Model- Based Recommendations for Optimal Surgical Placement of Epiretinal Implants. In Medical Image Computing and Computer Assisted Intervention -MICCAI 2019 (Lecture Notes in Computer Science), Dinggang Shen, Tianming Liu, Terry M. Peters, Lawrence H. Staib, Caroline Essert, Sean Zhou, Pew-Thian Yap, and Ali Khan (Eds.). Springer International Publishing, 394-402. https://doi.org/10.1007/ 978-3-030-32254-0_44 · DOI
  6. Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. 2019. A model of ganglion axon pathways accounts for percepts elicited by retinal implants. Scientific Reports 9, 1 (June 2019), 1-16. https://doi.org/10.1038/s41598-019-45416-4 · DOI · Paper page
  7. Justin R. Boyle, Anthony J. Maeder, and Wageeh W. Boles. 2008. Region-of-interest processing for electronic visual prostheses. Journal of Electronic Imaging 17, 1 (Jan. 2008), 013002. https://doi.org/10.1117/1.2841708 Publisher: International Society for Optics and Photonics. · DOI · Paper page
  8. G. Dagnelie, P. Keane, V. Narla, L. Yang, J. Weiland, and M. Humayun. 2007. Real and virtual mobility performance in simulated prosthetic vision. J Neural Eng 4, 1 (March 2007), S92-101. https://doi.org/10.1088/1741-2560/4/1/S11 · DOI
  9. Cordelia Erickson-Davis and Helma Korzybska. 2020. What do blind people "see" with retinal prostheses? Observations and qualitative reports of epiretinal implant users. bioRxiv (Feb. 2020), 2020.02.03.932905. https://doi.org/10.1101/ 2020.02.03.932905 · DOI
  10. Eduardo Fernandez. 2018. Development of visual Neuroprostheses: trends and challenges. Bioelectronic Medicine 4, 1 (Aug. 2018), 12. https://doi.org/10.1186/ s42234-018-0013-8 · DOI · Paper page
  11. I. Fine and G. M. Boynton. 2015. Pulse trains to percepts: the challenge of creating a perceptually intelligible world with sight recovery technologies. Philos Trans R Soc Lond B Biol Sci 370, 1677 (Sept. 2015), 20140208. https://doi.org/10.1098/rstb. 2014.0208 · DOI · Paper page
  12. C. Godard, O. M. Aodha, M. Firman, and G. Brostow. 2019. Digging Into Self- Supervised Monocular Depth Estimation. In 2019 IEEE/CVF International Confer- ence on Computer Vision (ICCV). 3827-3837. https://doi.org/10.1109/ICCV.2019. 00393 ISSN: 2380-7504. · DOI · Paper page
  13. Matthias Kümmerer, Thomas S. A. Wallis, and Matthias Bethge. 2016. DeepGaze II: Reading fixations from deep features trained on object recognition. arXiv:1610.01563 [cs, q-bio, stat] (Oct. 2016). http://arxiv.org/abs/1610.01563 arXiv: 1610.01563. · arXiv
  14. Lago, Miguel. 2021. SimplePhy: An open-source tool for quick online perception experiments. Behavior Research Methods (2021). https://doi.org/10.3758/s13428- 020-01515-z · DOI
  15. Heng Li, Tingting Han, Jing Wang, Zhuofan Lu, Xiaofei Cao, Yao Chen, Liming Li, Chuanqing Zhou, and Xinyu Chai. 2017. A real-time image optimization strategy based on global saliency detection for artificial retinal prostheses. Information Sciences 415, Supplement C (Nov. 2017), 1-18. https://doi.org/10.1016/j.ins.2017. 06.014 · DOI · Paper page
  16. Heng Li, Xiaofan Su, Jing Wang, Han Kan, Tingting Han, Yajie Zeng, and Xinyu Chai. 2018. Image processing strategies based on saliency segmentation for object recognition under simulated prosthetic vision. Artificial Intelligence in Medicine 84 (Jan. 2018), 64-78. https://doi.org/10.1016/j.artmed.2017.11.001 · DOI · Paper page
  17. Junning Li, Yonggang Shi, and Arthur W. Toga. 2015. Controlling False Discovery Rate in Signal Space for Transformation-Invariant Thresholding of Statistical Maps. Information processing in medical imaging : proceedings of the ... conference 9123 (July 2015), 125-136. https://doi.org/10.1007/978-3-319-19992-4_10 · DOI
  18. P. Lieby, N. Barnes, C. McCarthy, Nianjun Liu, H. Dennett, J. G. Walker, V. Botea, and A. F. Scott. 2011. Substituting depth for intensity and real-time phosphene rendering: Visual navigation under low vision conditions. In 2011 Annual Inter- national Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, Boston, MA, 8017-8020. https://doi.org/10.1109/IEMBS.2011.6091977 · DOI
  19. H. Lorach, G. Goetz, R. Smith, X. Lei, Y. Mandel, T. Kamins, K. Mathieson, P. Huie, J. Harris, A. Sher, and D. Palanker. 2015. Photovoltaic restoration of sight with high visual acuity. Nat Med 21, 5 (May 2015), 476-82. https://doi.org/10.1038/nm.3851 · DOI
  20. Wen Lik Dennis Lui, Damien Browne, Lindsay Kleeman, Tom Drummond, and Wai Ho Li. 2011. Transformative reality: Augmented reality for visual prostheses. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 253-254. https://doi.org/10.1109/ISMAR.2011.6092402 ISSN: null. · DOI · Paper page
  21. Y. H. Luo and L. da Cruz. 2016. The Argus((R)) II Retinal Prosthesis System. Prog Retin Eye Res 50 (Jan. 2016), 89-107. https://doi.org/10.1016/j.preteyeres.2015.09. 003 · DOI · Paper page
  22. Yvonne H.-L. Luo, Joe Jiangjian Zhong, Monica Clemo, and Lyndon da Cruz. 2016. Long-term Repeatability and Reproducibility of Phosphene Characteristics in Chronically Implanted Argus II Retinal Prosthesis Subjects. American Journal of Ophthalmology 170 (Oct. 2016), 100-109. https://doi.org/10.1016/j.ajo.2016.07.021 Publisher: Elsevier. · DOI · Paper page
  23. Chris McCarthy, Janine G. Walker, Paulette Lieby, Adele Scott, and Nick Barnes. 2014. Mobility and low contrast trip hazard avoidance using augmented depth. Journal of Neural Engineering 12, 1 (Nov. 2014), 016003. https://doi.org/10.1088/ 1741-2560/12/1/016003 Publisher: IOP Publishing. · DOI · Paper page
  24. N. Parikh, L. Itti, and J. Weiland. 2010. Saliency-based image processing for retinal prostheses. Journal of Neural Engineering 7, 1 (Jan. 2010), 016006. https: //doi.org/10.1088/1741-2560/7/1/016006 Publisher: IOP Publishing. · DOI
  25. Alejandro Perez-Yus, Jesus Bermudez-Cameo, Gonzalo Lopez-Nicolas, and Jose J. Guerrero. 2017. Depth and Motion Cues With Phosphene Patterns for Pros- thetic Vision. 1516-1525. http://openaccess.thecvf.com/content_ICCV_2017_ workshops/w22/html/Perez-Yus_Depth_and_Motion_ICCV_2017_paper.html
  26. Roksana Sadeghi, Arathy Kartha, Michael P. Barry, Paul Gibson, Avi Caspi, Arup Roy, and Gislin Dagnelie. 2019. Thermal and Distance image filtering improve independent mobility in Argus II retinal implant. Journal of Vision 19, 15 (Dec. 2019), 23-23. https://doi.org/10.1167/19.15.23 Publisher: The Association for Research in Vision and Ophthalmology. · DOI
  27. Melani Sanchez-Garcia, Ruben Martinez-Cantin, and Josechu J. Guerrero. 2019. Indoor Scenes Understanding for Visual Prosthesis with Fully Convolutional Networks. In VISIGRAPP. https://doi.org/10.5220/0007257602180225 · DOI · Paper page
  28. A. J. Simpson and M. J. Fitter. 1973. What is the best index of detectability? Psychological Bulletin 80, 6 (1973), 481-488. https://doi.org/10.1037/h0035203 Place: US Publisher: American Psychological Association. · DOI
  29. A. Stacey, Y. Li, and N. Barnes. 2011. A salient information processing system for bionic eye with application to obstacle avoidance. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 5116-5119. https://doi.org/10.1109/IEMBS.2011.6091267 ISSN: 1558-4615. · DOI
  30. K. Stingl, K. U. Bartz-Schmidt, D. Besch, A. Braun, A. Bruckmann, F. Gekeler, U. Greppmaier, S. Hipp, G. Hortdorfer, C. Kernstock, A. Koitschev, A. Kusnyerik, H. Sachs, A. Schatz, K. T. Stingl, T. Peters, B. Wilhelm, and E. Zrenner. 2013. Artificial vision with wirelessly powered subretinal electronic implant alpha-IMS. Proc Biol Sci 280, 1757 (April 2013), 20130077. https://doi.org/10.1098/rspb.2013.0077 · DOI · Paper page
  31. Victor Vergnieux, Marc J.-M. Macé, and Christophe Jouffrais. 2017. Simplifica- tion of Visual Rendering in Simulated Prosthetic Vision Facilitates Navigation. Artificial Organs 41, 9 (Sept. 2017), 852-861. https://doi.org/10.1111/aor.12868 Publisher: John Wiley & Sons, Ltd. · DOI
  32. Jing Wang, Heng Li, Weizhen Fu, Yao Chen, Liming Li, Qing Lyu, Tingting Han, and Xinyu Chai. 2016. Image Processing Strategies Based on a Visual Saliency Model for Object Recognition Under Simulated Prosthetic Vision. Ar- tificial Organs 40, 1 (2016), 94-100. https://doi.org/10.1111/aor.12498 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/aor.12498. · DOI
  33. J. D. Weiland and M. S. Humayun. 2005. A biomimetic retinal stimulating array. IEEE Engineering in Medicine and Biology Magazine 24, 5 (Sept. 2005), 14-21. https://doi.org/10.1109/MEMB.2005.1511496 Conference Name: IEEE Engineering in Medicine and Biology Magazine. · DOI
  34. J. D. Weiland, N. Parikh, V. Pradeep, and G. Medioni. 2012. Smart image processing system for retinal prosthesis. In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 300-303. https://doi.org/10.1109/ EMBC.2012.6345928 ISSN: 1558-4615. · DOI
  35. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2016. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv:1608.05442 (2016). · arXiv
  36. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene Parsing through ADE20K Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

Authors
Han, N., Srivastava, S., Xu, A., Klein, D., & Beyeler, M.
Affiliations
University of California, Santa Barbara (US)
ORCID
Venue
Augmented Humans Conference 2021
Publisher
ACM
Published
2021-02-22
Pages
45-54
DOI
10.1145/3458709.3458982
OpenAlex
W3180443265
Cited by
45
References
42
Citations by year
2026: 5 · 2025: 11 · 2024: 9 · 2023: 6 · 2022: 11 · 2021: 3
FWCI
29.17
Type
conference paper
Language
en
Open access
Licence
Locations
Augmented Humans International Conference (AHs) (publishedVersion, cc-by)
Funders
National Institutes of Health (R00 EY-029329)
Topics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neuroscience and Neural Engineering
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
Keywords
retinal prosthesis, bionic vision, object segmentation, visual saliency, monocular depth estimation, retinal degeneration, outdoor scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review