End-to-end optimization of prosthetic vision

“The decoder is an image-to-image conversion model that is based on a residual network architecture (He, Zhang, Ren, & Sun, 2016), which is known for its useful training properties (Ebrahimi & Abadi, 2018; Huang, Wang, Tao, & Zhao, 2020).”

“…number of training examples, ϕ d ( x ( n ) ) and ϕ d ( ˆ x ( n ) ) are the d -th layer feature maps extracted from the input image and the reconstruction image using the VGG16 model pre-trained on the ImageNet dataset (Simonyan & Zisserman, 2015) and K is the number of feature maps of that layer.”

“Unless stated otherwise, we report the mean squared error (MSE), the structural similarity index (SSIM; Wang, Bovik, Sheikh, & Simoncelli, 2004), and either the peak signal to noise ratio (PSNR) or the feature similarity index (FSIM; Zhang, Zhang, Mou, & Zhang, 2011) between the reconstruction and…”

“The phosphene encodings that were found by the model in this condition are comparable to those found with traditional preprocessing approaches (see Figure 6e), and yield similar reconstruction quality (see Table 3) compared to edge detection (Canny, 1986) or holistic contour detection (Xie & Tu, 2017).”

“…lower values of d , the perceptual loss reveals more explicit differences in low-level features such as intensities and edges, whereas for higher values of d the perceptual loss focuses on more abstract textures or conceptual differences between the input and reconstruction (Zeiler & Fergus, 2014).”

“…paper were not chosen only because of their well-established application in optimization problems (Asgari Taghanaki, Abhishek, Cohen, Cohen-Adad, & Hamarneh, 2021; Zhang, Isola, Efros, Shechtman, & Wang, 2018) but also because they represent basic functions that are normally performed in the brain.”

“Our end-to-end architecture implements an auto-encoder architecture, where the SPV representations can be seen as a latent encoding of the original input (or some transformation thereof) (Bengio, Courville, & Vincent, 2013).”

“…two other types of reconstruction tasks: first, an unsupervised perceptual reconstruction task (see related work by, Johnson, Alahi, and Fei-Fei (2016), Ledig et al. (2017)) which, in contrast to the pixel-wise MSE-based reconstruction task, aims to only preserve high-level perceptual features.”

“[31, 32], which, in contrast to the pixel-wise MSE-based reconstruction task, Figure 3.”

“…index (SSIM; Wang, Bovik, Sheikh, & Simoncelli, 2004), and either the peak signal to noise ratio (PSNR) or the feature similarity index (FSIM; Zhang, Zhang, Mou, & Zhang, 2011) between the reconstruction and the input image, as evaluated using the Scikit-image library (version 0.16.2) for…”

“To this end, we made use of the ADE20k semantic segmentation dataset (Zhou et al., 2017; Zhou et al., 2019).”

“…representations found in deep neural networks illustrate a similar processing hierarchy to that of the visual cortex (Güçlü & van Gerven, 2015; Yamins, Hong, Cadieu, Solomon, Seibert, & DiCarlo, 2014) and boundary detection is one of these processing steps needed for segregation of objects…”

“For instance, the inflammatory response of brain tissue is a major concern that limits the long-term viability of cortical electrode implants [35, 36] and these adverse effects may partly be avoided by limiting the chronic electrical stimulation itself [5, 28].”

“Journal of Vision , 22 (2):20, 1–14, https://doi.org/10.1167/jov.22.2.20. electrical stimulation of small groups of neurons, they evoke a percept of spatially localized flashes of light, called phosphenes (Brindley & Lewin, 1968; Najarpour Foroushani, Pack, & Sawan, 2018).”

“The feature representations found in deep neural networks illustrate a similar processing hierarchy to that of the visual cortex (Güçlü & van Gerven, 2015; Yamins, Hong, Cadieu, Solomon, Seibert, & DiCarlo, 2014) and boundary detection is one of these processing steps needed for segregation of…”

“The VGG-feature loss and BCE-loss that were implemented in this paper, were not chosen only because of their well-established application in optimization problems [38, 39] but also because they represent basic functions that are normally performed in the brain.”

“These neural interfaces can functionally replace the eye with a camera that is connected to the retina (Weiland, Liu, & Humayun, 2005; Zrenner et al., 2011), thalamus (Pezaris & Reid, 2007), or visual cortex (Beauchamp & Yoshor, 2020; Chen et al.”

“Our proposed end-to-end method is compared against existing approaches: Canny edge detection (Canny, 1986) and holistically nested edge detection (Xie & Tu, 2017).”

“Globally, over 30 million people suffer from blindness [1].”

“Furthermore, residual networks demonstrate computational similarities with recurrent networks that are found in the biological visual system (Kubilius et al., 2019; Liao & Poggio, 2016; Schrimpf et al., 2018).”

“…can functionally replace the eye with a camera that is connected to the retina (Weiland, Liu, & Humayun, 2005; Zrenner et al., 2011), thalamus (Pezaris & Reid, 2007), or visual cortex (Beauchamp & Yoshor, 2020; Chen et al., 2020; Dobelle, Mladejovsky, & Girvin, 1974; Tehovnik & Slocum, 2013).”

“In this paper, we focus on visual implants that reside in the primary visual cortex (V1), which are reported to have an enormous potential in future treatment of visual impairment (Beauchamp et al., 2020; Beauchamp & Yoshor, 2020; Chen et al., 2020; Lewis, Ackland, Lowery, & Rosenfeld, 2015).”

“…a similar processing hierarchy to that of the visual cortex (Güçlü & van Gerven, 2015; Yamins, Hong, Cadieu, Solomon, Seibert, & DiCarlo, 2014) and boundary detection is one of these processing steps needed for segregation of objects from background (Roelfsema, Lamme, Spekreijse, & Bosch, 2002).”

“By selective stimulation and by making use of the roughly retinotopical organization of V1, it is possible to generate a controlled arrangement of phosphenes, in such a way that they may provide a meaningful representation of the visual environment (figure 1) [9].”

“Considering the potentially adverse effects of electrical stimulation (Lewis et al., 2015; McCreery, Agnew, Yuen, & Bullara, 1988), one might want to introduce such a sparsity requirement that constrains the stimulation protocol, limiting the neural degradation by enforcing minimal energy transfer…”

“Furthermore, residual networks demonstrate computational similarities with recurrent networks that are found in the biological visual system (Kubilius et al., 2019; Liao & Poggio, 2016; Schrimpf et al., 2018).”

“…can functionally replace the eye with a camera that is connected to the retina (Weiland, Liu, & Humayun, 2005; Zrenner et al., 2011), thalamus (Pezaris & Reid, 2007), or visual cortex (Beauchamp & Yoshor, 2020; Chen et al., 2020; Dobelle, Mladejovsky, & Girvin, 1974; Tehovnik & Slocum, 2013).”

“These neural interfaces can functionally replace the eye with a camera that is connected to the retina (Weiland, Liu, & Humayun, 2005; Zrenner et al., 2011), thalamus (Pezaris & Reid, 2007), or visual cortex (Beauchamp & Yoshor, 2020; Chen et al., 2020; Dobelle, Mladejovsky, & Girvin, 1974;…”

“…can restore a rudimentary form of vision (Chen, Wang Fernandez, & Roelfsema, 2020; Fernandez, 2018; Lewis et al., 2016; Lozano et al., 2020; Riazi-Esfahani, Maghami, Sodagar, Lashay, & riazi-Esfahani, 2014; Roelfsema, Denys, & Klink, 2018; Shepherd, Shivdasani, Nayagam, Williams, & Blamey, 2013).”

“Furthermore, residual networks demonstrate computational similarities with recurrent networks that are found in the biological visual system (Kubilius et al., 2019; Liao & Poggio, 2016; Schrimpf et al., 2018).”

“At this time, cortical visual prostheses do not allow for systematic control over phosphene brightness (Najarpour Foroushani et al., 2018; Troyk et al., 2003).”

“Considering the potentially adverse effects of electrical stimulation (Lewis et al., 2015; McCreery, Agnew, Yuen, & Bullara, 1988), one might want to introduce such a sparsity requirement that constrains the stimulation protocol, limiting the neural degradation by enforcing minimal energy transfer…”

“For instance, due to inadvertent activation of underlying axon pathways, phosphenes generated with retinal prostheses may demonstrate distorted shapes that vary across subjects and even individual electrodes (Beyeler et al., 2019).”

“Similar to earlier work in the field of simulated prosthetic vision (Bollen, Guclu, et al., 2019; Bollen, van Wezel, et al., 2019; Dagnelie et al., 2007; Parikh et al., 2013; Sanchez-Garcia et al., 2020; Vergnieux et al., 2017; Vergnieux et al., 2014), the previous experiments in this study are…”

“…visual prosthetics may provide a promising solution that can restore a rudimentary form of vision (Chen, Wang Fernandez, & Roelfsema, 2020; Fernandez, 2018; Lewis et al., 2016; Lozano et al., 2020; Riazi-Esfahani, Maghami, Sodagar, Lashay, & riazi-Esfahani, 2014; Roelfsema, Denys, &…”

“…can restore a rudimentary form of vision (Chen, Wang Fernandez, & Roelfsema, 2020; Fernandez, 2018; Lewis et al., 2016; Lozano et al., 2020; Riazi-Esfahani, Maghami, Sodagar, Lashay, & riazi-Esfahani, 2014; Roelfsema, Denys, & Klink, 2018; Shepherd, Shivdasani, Nayagam, Williams, & Blamey, 2013).”

“…may provide a promising solution that can restore a rudimentary form of vision (Chen, Wang Fernandez, & Roelfsema, 2020; Fernandez, 2018; Lewis et al., 2016; Lozano et al., 2020; Riazi-Esfahani, Maghami, Sodagar, Lashay, & riazi-Esfahani, 2014; Roelfsema, Denys, & Klink, 2018; Shepherd,…”

“in terms of their entry point in the visual system), they share the same basic mechanism of action: through electrical stimulation of neurons in the visual system, they evoke a percept of spatially localized flashes of light, called phosphenes [10, 11].”

“Note that a variation on these approaches is investigated in Sanchez-Garcia et al. (2020), who demonstrated that preprocessing with semantic segmentation may successfully improve object recognition performance in simulated prosthetic vision (compared to preprocessing with conventional edge…”

“The decoder is an image-to-image conversion model that is based on a residual network architecture (He, Zhang, Ren, & Sun, 2016), which is known for its useful training properties (Ebrahimi & Abadi, 2018; Huang, Wang, Tao, & Zhao, 2020).”

“…to earlier work in the field of simulated prosthetic vision (Bollen, Guclu, et al., 2019; Bollen, van Wezel, et al., 2019; Dagnelie et al., 2007; Parikh et al., 2013; Sanchez-Garcia et al., 2020; Vergnieux et al., 2017; Vergnieux et al., 2014), the previous experiments in this study are…”

“Both early human trials (Brindley & Lewin, 1968), as well as recent animal studies (Schiller, Slocum, Kwak, Kendall, & Tehovnik, 2011) have shown that individual phosphenes elicited by stimulation in V1 have different sizes, which increase with foveal eccentricity.”

“…same-sized phosphenes, placed on a distorted rectangular grid, this time, we use a phosphene map that is inspired by the aforementioned studies of Srivastava et al. (2007, 2009), who simulate phosphenes in the lower left quadrant of the visual field, with phosphene densities and phosphene sizes…”

“The decoder is an image-to-image conversion model that is based on a residual network architecture (He, Zhang, Ren, & Sun, 2016), which is known for its useful training properties (Ebrahimi & Abadi, 2018; Huang, Wang, Tao, & Zhao, 2020).”

“Knowledge about the perceived phosphene coverage, which is unique for every patient, can be informative for finding a suitable encoding strategy (Buffoni, Coulombe, & Sawan, 2005; Kiral-Kornek, Savage, O’Sullivan-Greene, Burkitt, & Grayden, 2013).”

“However, rather than same-sized phosphenes, placed on a distorted rectangular grid, this time, we use a phosphene map that is inspired by the aforementioned studies of Srivastava et al. (2007, 2009), who simulate phosphenes in the lower left quadrant of the visual field, with phosphene densities and phosphene sizes ad…”

“…can functionally replace the eye with a camera that is connected to the retina (Weiland, Liu, & Humayun, 2005; Zrenner et al., 2011), thalamus (Pezaris & Reid, 2007), or visual cortex (Beauchamp & Yoshor, 2020; Chen et al., 2020; Dobelle, Mladejovsky, & Girvin, 1974; Tehovnik & Slocum, 2013).”

“Where MSE and PSNR are image quality assessment metrics that operate on pixel intensity, SSIM and FSIM are popular alternatives that better reflect perceptual quality (Preedanan, Kondo, Bunnun, & Kumazawa, 2018).”

“…et al., 2019; Bollen, van Wezel, et al., 2019; Dagnelie et al., 2007; Parikh et al., 2013; Sanchez-Garcia et al., 2020; Vergnieux et al., 2017; Vergnieux et al., 2014), the previous experiments in this study are performed using a basic simulation of cortical prosthetic vision, with…”

“For instance, with reinforcement learning strategies (see White et al., 2019), the model could be extended to perform tasks that more closely related to the everyday actions that need to be performed by the end-user, such as object manipulation (Levine, Finn, Darrell, & Abbeel, 2015) or object…”

“…algorithms, such as segmentation models for object recognition (Sanchez-Garcia, Martinez-Cantin, & Guerrero, 2020), or facial landmark detection algorithms for emotion recognition (Bollen, Guclu, van Wezel, van Gerven, & Gucluturk, 2019; Bollen, van Wezel, van Gerven, & Güçlütürk, 2019).”

“…can functionally replace the eye with a camera that is connected to the retina (Weiland, Liu, & Humayun, 2005; Zrenner et al., 2011), thalamus (Pezaris & Reid, 2007), or visual cortex (Beauchamp & Yoshor, 2020; Chen et al., 2020; Dobelle, Mladejovsky, & Girvin, 1974; Tehovnik & Slocum, 2013).”

“…algorithms, such as segmentation models for object recognition (Sanchez-Garcia, Martinez-Cantin, & Guerrero, 2020), or facial landmark detection algorithms for emotion recognition (Bollen, Guclu, van Wezel, van Gerven, & Gucluturk, 2019; Bollen, van Wezel, van Gerven, & Güçlütürk, 2019).”

“Amongthevarious strategies to generate aproper stimulation strategy, thereareapproachesbasedondeepneuralnetworksusedtobuildanend-to-endneuralprosthesis 192 thatcanalsobetrainedto approximatethe underlying biological system 193 .”

“…increasing interest of prosthetic engineers (e.g., Sanchez-Garcia, Martinez-Cantin, & Guerrero, 2020; Han et al., 2021; Bollen et al., 2019; Bollen, van Wezel, van Gerven, & Güçlütürk, 2019; De Ruyter Van Steveninck, Güçlü, van Wezel, & Van Gerven, 2020; Lozano et al., 2020; Lozano et al., 2018).”

“Machine learning approaches, such as convolutional neural networks [189] and deep learning [190, 191], could recognise multiple objects in complex scenes, helping navigation, obstacle avoidance and target object finding.”

“The VGG term aims to capture higher-level differences between images [33, 52].”

“These results are in line with previous results on constrained end- to-end optimization, that task-relevant information can be maximized under sparsity constraints ( de Ruyter van Steveninck et al., 2022a ).”

“…et al., 2009; Kiral-Kornek et al., 2013; Beyeler et al., 2019; Bruce and Beyeler, 2022) or finding ways to avoid them (Vilkhu et al., 2021; de Ruyter van Steveninck et al., 2022; Granley et al., 2022), these often theoretical insights have yet to be incorporated into a new generation of…”

“This is in contrast to previous DSEs, which either require retraining for every new patient [12, 13, 27], or can only vary two patient-specific parameters [14].”

“[10] used a similar approach to ours by training an encoder-decoder deep neural network with a scoreboard model in the loop.”

“These results may thus inform recent AI-based stimulus optimization algorithms (Spencer et al 2019, Granley et al 2022, Relic et al 2022, de Ruyter van Steveninck et al 2022), which aim to select the optimal stimulation parameters on each electrode based on their predicted effect on phosphene…”

“For recording and stimulation, a platinum ground wire circling the recording chamber served as a distant ground.”

“During this conversion process, several algorithms are designed to manipulate the current between electrodes in the desired way for better vision restoration [26, 27].”

“While previous research has focused on direct mechanistic image translation from camera footage to stimulation patterns (Wang et al 2021), often using a single kind of feature, such as edge, depth, saliency, segmentation maps, or their combinations (Boyle et al 2001, Dagnelie et al 2007, Parikh et al 2010, 2013, McCar…”

“However, in these studies, the goal of the encoders was to define stimulation patterns that accounted for phosphene distortions, characteristic of epiretinal implants, using multi-electrode unipolar stimulation.”

“The VGG term aims to capture higher-level differences between images [33, 52].”

“Van Steveninck et al. proposed an end-to-end optimization strategy in conjunction with a fi xed phosphene model 56 .”

“The bottleneck at the predicted percept is kept as 28×28, which is narrower than 49×49 as in [19] or 256×256 as in [16].”

“…uniformly spaced, full field covering phosphene configurations (Chen et al 2009, Bollen et al 2019, Sanchez-Garcia et al 2020, van Steveninck et al 2022, Avraham et al 2021, Wang et al 2021), which is a highly unlikely assumption given the anatomy and functional organization of the human brain.”

“infrared image [39], end-to-end optimization [40], semantic”

“…to the basic edge location information, semantic edges can be directly used or easily extended to solve many tasks such as refined object detection [12], image-based geopositioning [26], panoptic segmentation [17] and vision restoration applications in brain-machine interface research [9, 27].”

“Therefore, relatively high resolution patterns through stimulation of multiple electrodes can be freely tested, based on optimizations of even complex end-to-end machine learning algorithms (Kü ç üko ˘ glu et al 2022, 2025, Granley et al 2022a, de Ruyter van Steveninck et al 2022a).”

“With the help of powerful machine learning and deep learning techniques [5, 9], the optimization tasks can be solved, e.g ., using an evolutionary heuristic algorithm [46], a reconstruction decoder [55] or reinforcement learning [31].”

“1A), leveraging deep learning to approximate the inverse mapping from percepts to stimuli [17], [19]–[22].”

“Of the five end-to-end studies that we identified, two relied on auto-encoding strategies, 88,91 where the task to be learnt is reconstructing an original image, and three entailed reinforcement learning approaches, 87,89,90 for which the scope of learned tasks is wider.”

“In visual prostheses (Fernandez 2018, Ayton et al 2020), deep neural networks have been proposed to translate camera images into electrical stimulation patterns delivered to the retina or cortex (de Ruyter van Steveninck et al 2022, Granley et al 2022a, 2023, Moure et al 2025).”

“Moreover, end-to-end optimization approaches using deep auto-encoder architectures can automatically generate task-specific stimulation protocols from visual inputs 71,186 .”

“The scoreboard model, which approximates subretinal and cortical stimulation [72, 64], produced localized Gaussian “dots”, whereas the axon map model, designed for epiretinal implants, incorporated current spread along retinal ganglion cell axons to produce elongated, comet-shaped percepts [18, 63]…”

“…stimulation parameters for a given electrode array, including optimization of current amplitudes and electrode combinations [22, 6], hybrid calibration of electrode sensitivity profiles [13], and encoding frameworks that map images to stimulation patterns through learned transformations [14, 23].”

“A deep autoencoder-based architecture that includes a highly adjustable prosthetic vision simulation module attempts to automatically find a task-specific stimulation protocol (van Steveninck et al. 2022).”

“Answering this could pave the way for accurate modeling of visual processing in the blind human cortex, enabling the design of prostheses which deliver more perceptually intelligible visual experiences (de Ruyter van Steveninck et al., 2022; Granley et al., 2022a;b; 2023; Castro et al., 2023).”

“Existing phosphene rendering strategies in the simulated prosthetic vision literature, although effective, often preserve excessive visual detail, resulting in stimulation patterns that would require hundreds of simultaneously activated electrodes (de Ruyter van Steveninck et al 2022a).”

“…direction adopts neural networks to predict stimulation patterns and employs biophysical phosphene models to simulate phosphene perception and generate decoded images [36, 37, 7, 38, 39], but primarily relies on images as supervised signals without leveraging real neural responses for validation.”

“Taken from [11] By stimulating the neural tissue, the aim of this device is to invoke a light perception called a phosphene.”

“In this context, integrating retinal models with deep learning algorithms allows not only enhancing scene understanding but also compensating for both spatial and temporal distortions by optimizing stimulation strategies [19], leveraging the irregular shapes of phosphenes [20].”

“Moreover, recent computational models of prosthetic vision assume linear summation of phosphenes (Spencer et al., 2019; de Ruyter van Steveninck et al., 2022), but this has yet to be demonstrated empirically.”

“Some recent studies demonstrated that the simulation of prosthetic vision can even be incorporated directly into the deep learning-based optimization pipeline for end-to-end optimization (7, 12, 17).”

“This inverse network was trained alongside the forward model (with frozen weights), similar to an autoencoder [30, 33, 34].”