CoChat
Sign Up
PreprintCited by 4Neuroscience and Neural EngineeringTactile and Sensory InteractionsVisual perception and processing mechanisms

Deep Learning--Based Scene Simplification for Bionic Vision

Open access
Han, Nicole, Srivastava, Sudhanshu, Xu, Aiwen, Klein, Devi, Beyeler, Michael·2021·arXiv
DOI 10.48550/arxiv.2102.00297arXiv 2102.00297
Open this paper in CoChat — free accountOpen PDF

Abstract

Retinal degenerative diseases cause profound visual impairment in more than 10 million people worldwide, and retinal prostheses are being developed to restore vision to these individuals. Analogous to cochlear implants, these devices electrically stimulate surviving retinal cells to evoke visual percepts (phosphenes). However, the quality of current prosthetic vision is still rudimentary. Rather than aiming to restore "natural" vision, there is potential merit in borrowing state-of-the-art computer vision algorithms as image processing techniques to maximize the usefulness of prosthetic vision. Here we combine deep learning--based scene simplification strategies with a psychophysically validated computational model of the retina to generate realistic predictions of simulated prosthetic vision, and measure their ability to support scene understanding of sighted subjects (virtual patients) in a variety of outdoor scenarios. We show that object segmentation may better support scene understanding than models based on visual saliency and monocular depth estimation. In addition, we highlight the importance of basing theoretical predictions on biologically realistic models of phosphene shape. Overall, this work has the potential to drastically improve the utility of prosthetic vision for people blinded from retinal degenerative diseases.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.00297

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The models had been pre-trained on natural image datasets.

    “The models had been pre-trained on natural image datasets.”

    From Visual Prostheses in the Era of Artificial Intelligence Technology (2025) · cites this paper as [83]

    Related Work (p. 3) says DeepGaze II uses a VGG-19 model pre-trained on the SALICON dataset, and Scene Simplification Strategies (p. 5) identifies the monodepth2 model used as pre-trained. The copy does not establish that all the models were pre-trained on natural-image datasets, or identify a pre-training dataset for monodepth2. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • These three studies proposed to combine outputs of two deep learning models to build their saliency maps.

    “These three studies have proposed to combine outputs of two deep learning models to build their saliency maps (saliency with depth,83 contours with surface normals,84 contours with semantic masks).82”

    From Visual Prostheses in the Era of Artificial Intelligence Technology (2025) · cites this paper as [83]

    Section 4.2.3 (p. 5) combines structural edges from scene parsing with labeled object masks from an object-segmentation algorithm, supporting the cited example of contours with semantic masks. But the paper describes this as an object-segmentation output; its saliency map is produced separately by DeepGaze II (Section 4.2.1, p. 5). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Direct mapping of visual scenes is not always the best approach to convey visual information, since transformed views of the scene may better highlight salient visual details.

    “Direct mapping of visual scenes is not always the best approach to convey visual information,83 since transformed views of the scene may better highlight salient visual details.”

    From Visual Prostheses in the Era of Artificial Intelligence Technology (2025) · cites this paper as [83]

  • The use of pre-trained convolutional neural networks was proposed in three studies as a pre-processing step for saliency extraction.

    “The use of pre-trained convolutional neural networks was proposed in three studies82–84 as a pre-processing step for saliency extraction.”

    From Visual Prostheses in the Era of Artificial Intelligence Technology (2025) · cites this paper as [83]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [26][26] Y. H. Luo and L. da Cruz. 2016. The Argus((R)) II Retinal Prosthesis System. Prog Retin Eye Res 50 (Jan. 2016), 89-107. https://doi.org/10.1016/j.preteyeres.2015.09 .003 - · DOI

Cited by

Sorted by citations

Related

Authors
Han, N., Srivastava, S., Xu, A., Klein, D., & Beyeler, M.
Affiliations
University of California, Santa Barbara (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.00297
arXiv
2102.00297
OpenAlex
W3129088126
Cited by
4
References
41
Citations by year
2025: 1 · 2024: 1 · 2022: 1 · 2021: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Institutes of Health (R00 EY-029329)
Topics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neuroscience and Neural Engineering
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
Keywords
retinal prosthesis, bionic vision, object segmentation, visual saliency, monocular depth estimation, retinal degeneration, simulated prosthetic vision, scene understanding, virtual patients

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review