CoChat

The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index; 23 of the 43 sources its claims cite were checked for support.

What if eye...? Computationally recreating vision evolution

Open access

Abstract

Natural selection has produced diverse vision systems, from simple patches of photoreceptors to complex camera eyes, representing just one set of evolutionary outcomes. Computational evolution offers a way to systematically test hypotheses, isolate individual factors, and ask the “why” questions behind vision. We recreate vision evolution by coevolving eyes and behaviors in embodied agents and use this to illuminate principles shaping vision across different levels of the Marr’s hierarchy. This leads to three key findings: First, we provide computational evidence that task-specific selection drives bifurcation in eye evolution. Second, we reveal how optical innovations naturally emerge to resolve fundamental trade-offs between light collection and spatial precision. Third, we uncover scaling laws between visual acuity and neural processing that provide insights into long-standing hypothesis behind eye and brain size. Our work introduces a paradigm that uses embodied artificial intelligence (AI) as hypothesis-testing machines that can help accelerate discoveries in vision science.

Overview

CoChat Summary

TL;DR

The paper introduces an embodied-AI framework that coevolves eyes and behavior via CMA-ES and PPO to test “what-if” scenarios in vision evolution. It reports task-specific eye bifurcation, lens emergence resolving light-vs-acuity trade-offs, and power-law scaling between visual acuity and neural capacity.

Key findings

  • Task-specific selection drives divergent eye morphologies from the same initial single-eye, one-photoreceptor agent: Navigation evolves distributed eyelet-type vision with seven eyes of four photoreceptors each (1×4) and a 135° total FOV, while Detection evolves two forward-facing camera-type eyes with 225 photoreceptors each (15×15).
  • Optical innovation resolves the light-collection vs spatial-precision trade-off: without optics, agents progress from open pupil to cup to near-pinhole eyes and plateau; introducing optics at generation 30 leads to lens-like structures by generation 50 and well-defined lenses beyond, with larger pupils and reliable detection by generation 130. Image Quality (MTF spatial precision × maximum light collection) separates high performers (F > 24.4) from poor performers (F < 3.4).
  • Task error scales as a power law with neural parameters: L = (9.50×10⁻³)·N⁰·⁶⁹ for navigation, L = (7.31×10⁻³)·N⁰·⁷⁴ for detection, and L = (2.43×10⁻²)·N⁰·⁵⁹ for tracking; limited visual acuity imposes a performance ceiling that neural scaling alone cannot overcome, and temporal memory can partially compensate for reduced neural capacity.
  • The genetic encoding can represent ~10²⁰ unique configurations, and the framework independently evolved distributed eyelet-type architectures without being explicitly designed to do so.

Methods

  • Coevolution framework: outer loop uses CMA-ES with population size 16 over 50–100 generations; inner loop trains each agent with PPO up to 1 million steps, scoring fitness as average reward over six evaluation episodes. Experiments use five seeds (0, 42, 420, 1337, 4002) and evaluate upward of ~3200 independent agent morphologies.
  • Embodied agents are point-mass spheres of radius 0.2 units in MuJoCo/gymnasium environments, solving three tasks: Navigation (maze orientation/obstacle avoidance), Detection (food vs poison object discrimination), and Tracking (moving objects).
  • Genetic encoding has morphological genes (eye number, placement, FOV), optical genes (4×4 programmable phase mask/DOE upsampled to 51×51, refractive index in [1.0, 2.0], pupil radius in [0,1]), and neural genes (hidden-layer neurons 1–512, temporal memory). Imaging uses a PSF convolution model with Gaussian sensor noise; analyses use CPD, MTF, PSNR/SSIM, and Image Quality.

Limitations

  • Stated: the work recreates the system-level process of vision evolution rather than its historical timeline; complex biological phenomena such as eye genomics, development, biophysical models, and mechanistic neural circuits are poorly understood; evolutionary dynamics are often chaotic; experiments use isolated single-task settings; the phenotype-to-performance mapping is simplified; and the imaging model assumes depth-independent PSFs, a circular amplitude mask, and a 4×4 phase mask for tractability, while abstracting away fluid-dynamics details of lens emergence.
  • Stated: the framework does not yet include multitask or multiagent settings, richer light properties such as spectral, polarization, or temporal sampling, or mechanistic circuits; neural network weights are not encoded in the genome.
  • Evident from the text: the tasks and environments are simplified simulations, so the causal claims about natural vision evolution remain computational hypotheses rather than direct biological tests; the paper itself notes that isolated scenarios help understand extreme cases but do not capture the joint demands animals face.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1126/sciadv.ady2888

Cites

86 of 86 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. Computational evolution of embodied AI agents reveals how environmental pressures shaped natural vision evolution. We evolve artificial embodied agents to show how three evolutionary branch points shaped vision evolution. We use our framework to understand (A) how environmental specificity led to distinct eye morphologies, (B) how optical elements emerge when embodied agents evolve to discriminate between objects while accounting for physical trade-offs of light collection versus spatial precision in an environment, and (C) how visual task error scales as power law with visual acuity and number of parameters revealing that poor visual acuity creates a fundamental bottleneck that cannot be overcome by simply scaling neural capacity. (D) Our framework mirrors natural selection: An outer loop governs genetic inheritance and selection over evolutionary timescales, while an inner loop enables agents to learn through sensory feedback (lifetime adaptation). This nested structure reflects the Baldwin effect (46), where agents better able to learn throughout their lifetime are more likely to be selected, accelerating evolutionary adaptation of their specific morphologies. (E) The agent’s digital anatomy parallels biological visual systems: from eye morphology and placement, through optical elements and photoreceptors (mimicking retinal organization), to neural processing (analogous to visual cortex). (F) Agents evolve to solve three distinct visual tasks to probe how environmental pressures shape vision: (i) Navigation: orientation and obstacle avoidance through a maze-like environment; (ii) Detection: object discrimination between a goal object (food) and adversarial object(s) (poison); and (iii) Tracking: identical to Detection, but the objects move. Our results highlight how embodied agents can serve as scientific instruments to understand biological visual intelligence.
Fig. 1. Computational evolution of embodied AI agents reveals how environmental pressures shaped natural vision evolution. We evolve artificial embodied agents to show how three evolutionary branch points shaped vision evolution. We use our framework to understand (A) how environmental specificity led to distinct eye morphologies, (B) how optical elements emerge when embodied agents evolve to discriminate between objects while accounting for physical trade-offs of light collection versus spatial precision in an environment, and (C) how visual task error scales as power law with visual acuity and number of parameters revealing that poor visual acuity creates a fundamental bottleneck that cannot be overcome by simply scaling neural capacity. (D) Our framework mirrors natural selection: An outer loop governs genetic inheritance and selection over evolutionary timescales, while an inner loop enables agents to learn through sensory feedback (lifetime adaptation). This nested structure reflects the Baldwin effect (46), where agents better able to learn throughout their lifetime are more likely to be selected, accelerating evolutionary adaptation of their specific morphologies. (E) The agent’s digital anatomy parallels biological visual systems: from eye morphology and placement, through optical elements and photoreceptors (mimicking retinal organization), to neural processing (analogous to visual cortex). (F) Agents evolve to solve three distinct visual tasks to probe how environmental pressures shape vision: (i) Navigation: orientation and obstacle avoidance through a maze-like environment; (ii) Detection: object discrimination between a goal object (food) and adversarial object(s) (poison); and (iii) Tracking: identical to Detection, but the objects move. Our results highlight how embodied agents can serve as scientific instruments to understand biological visual intelligence.CC BY · PubMed Central
Fig. 2. Our genetic encoding enables vision to evolve computationally. Our encoding mirrors the natural separation between sensory and neural development through three gene clusters. Morphological genes determine agent properties relating to spatially sampling the environment such as eye placement and FOV. Optical genes determine agent properties relating to how each eye interacts with incoming light in a physically plausible way, including the number of photoreceptors, optical elements, and pupil size. Neural genes describe the behavior learning capacity of the agent. These independently mutable genes enable the computational exploration of evolutionary pathways that mirror those in natural vision evolution.
Fig. 2. Our genetic encoding enables vision to evolve computationally. Our encoding mirrors the natural separation between sensory and neural development through three gene clusters. Morphological genes determine agent properties relating to spatially sampling the environment such as eye placement and FOV. Optical genes determine agent properties relating to how each eye interacts with incoming light in a physically plausible way, including the number of photoreceptors, optical elements, and pupil size. Neural genes describe the behavior learning capacity of the agent. These independently mutable genes enable the computational exploration of evolutionary pathways that mirror those in natural vision evolution.CC BY · PubMed Central
Fig. 3. Low- and high-acuity spatial tasks lead to distributed and camera eyes, respectively. (A) We initialize a population of agents for two visual tasks (Detection and Navigation) with a single eye with one photoreceptor. We then evolve a population of agents subject to morphological mutations: add photoreceptor, add eye, and adjust placement. (B) In the Navigation task, we first observe an emergence of dispersed vision, where many eyes are used. Some agents in early generations develop compound-eye configurations with multiple single-photoreceptor eyes distributed around the body. By generation 50, most agents evolve into an eyelet-type vision system consisting of seven individual eyes, each with four photoreceptors (4 by 1 resolution), distributed over the entire diameter of the agent. This resembles the visual architecture found in strepsipteran insects (55) rather than typical compound eyes. (C) In the Detection task, we initially observe the emergence of low-resolution vision. After 50 generations, the population has converged on a morphology consisting of two forward-facing, high-resolution camera-type eyes each with 225 photoreceptors (15 by 15 resolution). (D) Configuration versus generation plots are shown, depicting the evolutionary progression of the mean agent in the population and the task dependence on evolutionary adaptation. For each task, we run five trials varying the random seed for CMA-ES. The plots show the mean across all trials as a solid line, the mean of each run as dotted lines, and the 95% bootstrapped confidence interval across all trials as shaded regions.
Fig. 3. Low- and high-acuity spatial tasks lead to distributed and camera eyes, respectively. (A) We initialize a population of agents for two visual tasks (Detection and Navigation) with a single eye with one photoreceptor. We then evolve a population of agents subject to morphological mutations: add photoreceptor, add eye, and adjust placement. (B) In the Navigation task, we first observe an emergence of dispersed vision, where many eyes are used. Some agents in early generations develop compound-eye configurations with multiple single-photoreceptor eyes distributed around the body. By generation 50, most agents evolve into an eyelet-type vision system consisting of seven individual eyes, each with four photoreceptors (4 by 1 resolution), distributed over the entire diameter of the agent. This resembles the visual architecture found in strepsipteran insects (55) rather than typical compound eyes. (C) In the Detection task, we initially observe the emergence of low-resolution vision. After 50 generations, the population has converged on a morphology consisting of two forward-facing, high-resolution camera-type eyes each with 225 photoreceptors (15 by 15 resolution). (D) Configuration versus generation plots are shown, depicting the evolutionary progression of the mean agent in the population and the task dependence on evolutionary adaptation. For each task, we run five trials varying the random seed for CMA-ES. The plots show the mean across all trials as a solid line, the mean of each run as dotted lines, and the 95% bootstrapped confidence interval across all trials as shaded regions.CC BY · PubMed Central
Fig. 4. Computational evolution reveals how lensing resolves a fundamental trade-off in vision. We demonstrate that to achieve maximum fitness in the Detection task, evolution learns to evolve optical structures that balance spatial precision (optical resolution of the eye quantified using MTF) against maximizing light collection. (A) The evolutionary sequence shows five key stages of eye development: (1) open pupil with maximum light collection but poor spatial precision, (2) cup eye and (3) pinhole eye that achieve better spatial precision by reducing pupil size at the cost of light collection, followed by the emergence of (4) unfocused lens-based eyes, where a diffuse lens with low or uniform refractive index does not yet concentrate light effectively, and (5) focused lens-based eyes that maintain spatial precision by evolving optical structures while allowing larger pupils for more light collection. Agent images show the scene as perceived at each stage of evolution using the evolved optical genes (pupil, lens, and aperture). The ground truth scene is shown in fig. S8. (B) Without optics (dark blue), pupil size decreases to improve precision, sacrificing the SNR. When lensing is enabled (orange line, generation 30), larger pupils emerge as lenses evolve to maintain precision while increasing light collection. (C) The Image Quality metric (image sharpness × light collection) quantifies this trade-off resolution: Pinhole eyes (3) plateau at low values due to limited light collection, whereas lens-based eyes [(4) and (5)] achieve higher quality by combining good spatial precision with larger pupils. This mirrors the evolutionary pressure that drove the emergence of biological lenses, which enabled enhanced vision across lighting conditions. The dark line and shaded region show the mean across five independent evolution runs and the 95% bootstrapped confidence interval, respectively. The dotted lines represent the mean across each run.
Fig. 4. Computational evolution reveals how lensing resolves a fundamental trade-off in vision. We demonstrate that to achieve maximum fitness in the Detection task, evolution learns to evolve optical structures that balance spatial precision (optical resolution of the eye quantified using MTF) against maximizing light collection. (A) The evolutionary sequence shows five key stages of eye development: (1) open pupil with maximum light collection but poor spatial precision, (2) cup eye and (3) pinhole eye that achieve better spatial precision by reducing pupil size at the cost of light collection, followed by the emergence of (4) unfocused lens-based eyes, where a diffuse lens with low or uniform refractive index does not yet concentrate light effectively, and (5) focused lens-based eyes that maintain spatial precision by evolving optical structures while allowing larger pupils for more light collection. Agent images show the scene as perceived at each stage of evolution using the evolved optical genes (pupil, lens, and aperture). The ground truth scene is shown in fig. S8. (B) Without optics (dark blue), pupil size decreases to improve precision, sacrificing the SNR. When lensing is enabled (orange line, generation 30), larger pupils emerge as lenses evolve to maintain precision while increasing light collection. (C) The Image Quality metric (image sharpness × light collection) quantifies this trade-off resolution: Pinhole eyes (3) plateau at low values due to limited light collection, whereas lens-based eyes [(4) and (5)] achieve higher quality by combining good spatial precision with larger pupils. This mirrors the evolutionary pressure that drove the emergence of biological lenses, which enabled enhanced vision across lighting conditions. The dark line and shaded region show the mean across five independent evolution runs and the 95% bootstrapped confidence interval, respectively. The dotted lines represent the mean across each run.CC BY · PubMed Central
Fig. 5. Task-dependent scaling laws reveal how visual acuity bounds performance and how temporal memory compensates for neural capacity. (A to C) Our experiments reveal visual task-dependent power-law scaling between number of parameters and visual acuity (CPD). This demonstrates that scaling in sensory input is required for embodied tasks to avoid a bottleneck that cannot be overcome by neural scaling alone. (D) A minimum required visual acuity compared to number of parameters for different embodied tasks suggests a hierarchy in the emergent behaviors depending on the task. (E) Temporal processing shows complementary scaling with neural capacity, where increased temporal memory (# Frames) can compensate for reduced neural processing, which is particularly evident in tasks with larger networks. Together, these results quantify how visual intelligence emerges from the interplay between sensory, neural, and temporal capabilities.
Fig. 5. Task-dependent scaling laws reveal how visual acuity bounds performance and how temporal memory compensates for neural capacity. (A to C) Our experiments reveal visual task-dependent power-law scaling between number of parameters and visual acuity (CPD). This demonstrates that scaling in sensory input is required for embodied tasks to avoid a bottleneck that cannot be overcome by neural scaling alone. (D) A minimum required visual acuity compared to number of parameters for different embodied tasks suggests a hierarchy in the emergent behaviors depending on the task. (E) Temporal processing shows complementary scaling with neural capacity, where increased temporal memory (# Frames) can compensate for reduced neural processing, which is particularly evident in tasks with larger networks. Together, these results quantify how visual intelligence emerges from the interplay between sensory, neural, and temporal capabilities.CC BY · PubMed Central
Fig. 6. Imaging model. Our simulation implements both wave and geometric optics using OpenGL (90). (A) Our scene imaging model shows how a depth-dependent blur kernel is derived: Light from a point source in the 3D scene propagates through free space, passing through a pupil plane, which is composed of (i) an aperture with a variable aperture radius (r) and (ii) a programmable phase mask with height map H(x,y) and refractive index n, before forming an image on the sensor plane. (B) The approximated model uses a 2D convolution between the scene, a depth map, and a single blur kernel using the farfield approximation (i.e., the PSF) for computational efficiency.
Fig. 6. Imaging model. Our simulation implements both wave and geometric optics using OpenGL (90). (A) Our scene imaging model shows how a depth-dependent blur kernel is derived: Light from a point source in the 3D scene propagates through free space, passing through a pupil plane, which is composed of (i) an aperture with a variable aperture radius (r) and (ii) a programmable phase mask with height map H(x,y) and refractive index n, before forming an image on the sensor plane. (B) The approximated model uses a 2D convolution between the scene, a depth map, and a single blur kernel using the farfield approximation (i.e., the PSF) for computational efficiency.CC BY · PubMed Central

Verification

Claims the paper makes

13 Valid · 2 Not valid · 4 Not checked · 89 Own results and interpretations

60 more claims · 4 with their cited sources checked

Create a free account to see all 108 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 20
  • It may state two separate claims as one. 30
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 86 references

Research works printed but not linked: 9

  1. M. Land D.-E. Nilsson Animal Eyes (Oxford Univ. Press 2002).
  2. D. Floreano S. Nolfi “Adaptive behavior in competing co-evolving species” in 4th European Conference on Artificial Life (1997) pp. 378–387.
  3. E. Warrant D.-E. Nilsson Invertebrate Vision (Cambridge Univ. Press 2006).
  4. E. Conti V. Madhavan F. P. Such J. Lehman K. O. Stanley J. Clune Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. arXiv:171206560 (2018).
  5. N. Hansen A. Ostermeier “Adapting arbitrary normal mutation” in Proceedings of the 1996 IEEE International Conference on Evolutionary Computation (1996) pp. 312–317.
  6. A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, N. Dormann, Stable-baselines3: Reliable reinforcement learning implementations. J. Mach. Learn. Res. 22, 1–8 (2021).
  7. M. Woo J. Neider T. Davis D. Shreiner OpenGL Programming Guide: The Official Guide to Learning OpenGL Version 1.2 (Addison-Wesley Longman Publishing Co. Inc. 1999).
  8. E. Adelson J. Bergen “The plenoptic function and the elements of early vision” (1997); https://www.researchgate.net/publication/2684713_The_Plenoptic_Function_and_the_Elements_of_Early_Vision.
  9. “Allometry: The study of biological scaling” (2018); https://api.semanticscholar.org/CorpusID:199531454.

Cited by

Metadata

Authors
Tiwary, K., Young, A., Tasneem, Z., Klinghoffer, T., Dave, A., Poggio, T., Nilsson, D.-E., Cheung, B., & Raskar, R.
Affiliations
  • MIT Media Lab (US)
  • Rice University (US)
  • Draper Laboratory (US)
  • Massachusetts Institute of Technology (US)
  • Lund University (SE)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
Corresponding author
  • Tiwary, Kushagra
  • Young, Aaron
  • Cheung, Brian
  • Raskar, Ramesh
Venue
Science Advances
Published
2025-12-19
Volume
11
Issue
51
Pages
eady2888-eady2888
ISSN
2375-2548 (ISSN-L)
OpenAlex
W4417439479
Cited by
10
References
74
Citations by year
2026: 6 · 2025: 4
FWCI
2.57
Type
journal article
Language
en
Open access
Locations
  • Science Advances (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • Lund University Publications (Lund University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Defense Science and Engineering Graduate
  • Qualcomm Innovation Fellowship
  • National Science Foundation GRPF (2022339767)
  • Charles Stark Draper Laboratory Scholarship
  • MIT GenAI Impact Grant
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Evolution and Genetic Dynamics
Keywords
evolution of vision, eye evolution, embodied agents, brain size, photoreceptors, camera eye, computational evolution
MeSH terms
Animals, Artificial Intelligence, Biological Evolution, Eye, Humans, Vision, Ocular

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.