Principles of Neural Design

“(Sterling and Laughlin 2015, xv) 1.”

“33,34 Attention is a key mechanism that likely plays a major role in efficient neural computation.”

“…the brain is not uniform; its functional specialization is underpinned by significant regional heterogeneity in brain architecture and physiology, where variations in cyto-, myelo-, chemoarchitecture, neural receptors and gene expression give rise to distinct local E-I ratios 38, 39, 40, 41, 42 .”

“First, long, small-diameter axons are widely found in mammalian central nervous systems 9,12,13 , including humans 14 .”

“Unless otherwise stated, forward speed = 12 mm/s, and perturbation strength = 1.875 rad/s. while preserving behavioral realism, thus laying the groundwork for future studies on the effect of delay on other aspects of locomotion and sensorimotor control.”

“[6] By predicting future deficiency (e.g.,upcomingwinterfamine),animalspreemptivelyprepareexcessive amounts of resources (e.g., squirrel hoarding acorns) to cope with and survive the future deficiency.”

“Even before birth, brains are specialized to solve tasks that are likely to be encountered in their lifetimes and environment [1].”

“The broad idea of efficient coding is that neural circuits are costly, and were selected in a manner that produces the greatest benefit for the investment [69].”

“Although such an eccentric increase in TDE receptive fields nicely fits the rotation disk (where velocity increases as one moves out of the center), this solution is rather general and universally encountered in animal visual systems [55].”

“The reason is that every experimental intervention must be performed on a specific realization or implementation of the structure.”

“To reduce the costs of signal processing and downstream transmission, the receptor currents have the slowest kinetics and thus the narrowest bandwidth that still allows the receptor to carry out its function [43].”

“From a mechanistic point of view, one explanation may be that the energetic and space constraints on brain wiring imply that long-range, between-region signals can be transmitted only at low-bandwidth and with some latency (Sterling & Laughlin, 2015).”

“Two general lessons have emerged simultaneously from both engineering pursuits and naturalistic inquiry: (1) a neural system’s performance typically improves with increasing number and connectivity of processing primitives, both in biology [1, 2, 3] and artificial intelligence [4, 5]; and (2) analog processing with sp…”

“It has been increasingly recognized that evolution has optimized neurons and their circuits for best simultaneous performance in terms of functional effectiveness and economy (Laughlin and Sejnowski, 2003; Sterling and Laughlin, 2015; Pallasdies et al., 2021).”

“The multiple feedback loops governing that behavior are illustrated in Figure 1 (from [3]).”

“The first is what we could call the problem of ‘junk mail’: (Unless some additional control apparatus is postulated) at any moment each unit active at the lower level is broadcasting largely unwanted action potentials to every comparator unit with which it is connected; and each action potential has a significant cost…”

“Allostasis: All the brain anticipate needs and provide organism’s physiological infrastructure to that need in to regulate the organism’s internal milieu[229].”

“) or treat it as an information-theoretical quantity (Friston, 2013). If it is referenced, it is usually in relation to analysis of the metabolic activity that supports neural information processing (Sterling and Laughlin, 2017); however, it is seldom regarded as playing a direct functional role in the generation of c…”

“In brains, each additional neuron adds metabolic costs, maintenance, and wiring— making a smaller brain preferable [8, 21, 38].”

“Each variable is associated with a preferred operating range [𝑣 $%& , 𝑣 $() ] , analogous to the viable physiological bounds observed in biological systems [29,48].”

“In sharp contrast, energy-efficiency is a core principle of biological computation (Olshausen & Field, 1997; Quiroga et al., 2008; Sterling & Laughlin, 2015).”

“Its primary objective is to realize high-parallelism, energy-efficient computational paradigms by mimicking the structural organization and information transmission mechanisms inherent to biological neural systems [10,11].”

“Following the “Biological Minimal Cost Principle” [62], it is more pragmatically significant for a robotic system to achieve reliable survival in limited environments at minimal cost than to blindly pursue Artificial General Intelligence (AGI).”

“In parallel, compression via Barlow-Attneave redundancy reduction may support energetic efficiency (42, 45, 101, 129) and be most beneficial when the input is high-dimensional and contains redundant statistical regularities.”

“Given synaptic density across species, coupled with the fact that the dendritic radius in larger brains is slightly thicker to mitigate conduction delays, what other factors could compensate for lower input resistance [39]?”

“The visual system optimizes contrast sensitivity by dedicating the limited dynamic range of ON and OFF RGCs to signaling increments and decrements, respectively [26,27].”

“For example, the retina communicates via the optic nerve to the cortex, which is argued to be a several-fold compression of information capture from raw visual input [31].”

“…behaviourism, embody a prime example of predictive regulation (allostasis): seeing or smelling food, or hearing the tic-toc of a metronome that is predictively associated with the appearance of food, activates the body’s digestive systems before the demand arises (Sterling & Laughlin, 2015).”

“Thus, we kept the two sensory oscillators independent and incorporated the contralateral sensori-motor connections present in the Braitenberg vehicles (52) and in many of the biological neural organizations (83).”

“The organism turns a random amount in the medium before the cilia are reengaged in generating forward movement in a new direction [4,8,10].”

“9 and 17, we use tensor terminology and refer to { ⃗ (cid:101) ε (cid:101) µ } as contravariant basis vectors and { ⃗ (cid:101) ε (cid:101) µ } as covariant basis vectors (see Appendix A).”

“This primitive analogue device is economically favoured and widely utilized in retinal neurocircuitry to establish various meaningful functions (Sterling and Laughlin, 2015).”

“These noise additions could arise here, as it does in many biological systems (Hornung & Barkai, 2008; Sterling & Laughlin, 2015), through variations in component number within the transmission machinery such as neurotrans-mitters or synaptic vesicles or vesicle release probability (Franks et al.,…”

“…and post STDP trace contributions) to three or even four factor and continual learning rules which may incorporate other local factors such as the stochastic synaptic neurotransmitter release probability or even meta/ global influences such as homeostatic [9] control and dopaminergic reward [11].”

“Nevertheless, shared information and correlations are surprisingly high given that the statistics of the environment is not the only in driving factor in vision [90], [91].”

“Reducing the learning rate may reduce the metabolic cost of making the updates, since less synaptic plasticity is required and the brain has been heavily optimized by evolution to minimize energy expenditure (Sterling & Laughlin, 2015).”

“Instead, the vertebrate retina lies directly on the optical path, such that the light passes through the ganglion cells, then the inner synaptic layer, then the amacrine cells, then the bipolar cells, then the horizontal cells, then the outer synaptic layer, then the photoreceptor cell bodies, and then finally, the ac…”

“…that calcium oscillations in astrocytes might contribute to lowering energy consumption during long-distance information propagation through an ensemble of astrocytes and neurons, in comparison to the route involving only neuronal synapses (Attwell and Laughlin 2001, Sterling and Laughlin 2015).”

“One of the largest contributions to the metabolic costs of neuronal signalling is the length of an axon (Niven and 247 Laughlin, 2008; Cherniak, C, 1994; Sterling and Laughlin, 2015).”

“2 Introduction 39 40 Evolution has matched neural resources to the sensory information that is available to the 41 animal (1, 2).”

“Such synaptic learning also plays a crucial role in biological learning, but in ensemble with for example structural adaptations of the network and influence from different neuromodulators providing reward signals or adapting neuron behaviour (Sterling and Laughlin, 2015; Price et al., 2017).”

“Thus, neuro-archaeological accounts must involve an understanding of how brains mediate multiple states of consciousness (Sterling and Laughlin 2015).”

“Using bits per Joule as a proxy for the evolutionary fitness of nervous systems can explain many features of nervous system structure and function (Sterling & Laughlin, 2015).”

“[15] Peter Sterling and Simon Laughlin.”

“Hardware neural networks (HNNs) are computing systems in which memory and computing units are not distinct entities exchanging data through a communication bus but rather they are distributed in a way that resembles the organization of synapses and neurons in the human brain [24].”

“Although the actual mechanism of a human brain is very complex [3], it can still be simplified as a connecting model of neu-1 rons and synapses [2].”

“…pumps are all parts of neurons and theoretical predictions from information theory and biophysics suggest that is not possible to make these components more efficient: neural information processing already happens close to the minimum cost allowed by thermodynamics (Sterling & Laughlin, 2015).”

“Excitatory (glutamatergic) transmission, encompassing vesicle cycling and postsynaptic depolarisation, accounts for the vast majority of signalling energy, while inhibitory (GABAergic) transmission is substantially cheaper (Sengupta, Stemmler, Laughlin and Niven, 2010; Sterling and Laughlin, 2015).”

“The human brain has been shaped by evolution [1] to handle cognitive tasks, such as pattern recognition, with very low power consumption [2, 3].”

“There, channel capacity is preferentially increased through the differential allocation of neuronal and metabolic resources across cortical systems, constraining the resources available to alternative processing pathways (51, 52, 73, 74).”

“This energy constraint on neural signaling is one of the principles of neural design (Niven et al. 2007; Sterling and Laughlin 2015; Sengupta et al. 2013).”

“Axon volume and energy rise with the square of transmission rate, so the design rule is to send only what is needed, as slowly as possible [230].”

“In their book “Principles of Neural Design,” Sterling and Laughlin ( 2015 ) suggested that “Visual sensors are fast and energetically costly.”

“…substantial metabolic cost that scales non-linearly: doubling temporal resolution may demand a quadrupling or greater increase in energy consumption (Attwell & Laughlin, 2001; Laughlin et al., 1998), with synaptic transmission being the largest consumer of neural energy (Sterling & Laughlin, 2015).”

“The computational performance (flops per kg per watt) of the human brain is several orders of magnitude greater than the best digital computers [138].”

“Complex cognition is valuable when it increases the expected return from action, but it is costly to build, costly to operate, and costly to coordinate [89].”

“[10, 12, 24] The fact that Cl − is negatively charged explains why the sign of the impulse peak is negative for the LMC (and why LMCs are sign inverting).”

“This model involves representing the predicted state of the body, the predicted state of the world, and predictively adjusting physiological states and actions to meet the body’s needs [8,11–14 , 16].”

“The coding capacity of a single neuron in neuron space depends on its mean firing rate and spike timing precision [63].”

“Energy budgets are not small corrections; they can shape coding strategies, circuit architectures, and even evolutionary trajectories 3,5 .”

“Mechanisms which developed to support this proactive homeostatic regulation, such as the aforementioned forward models (cf. above), meet this need to be an anticipatory model of the environment (Pezzulo, 2017; Sterling & Laughlin, 2015).”

“This number appears disproportionate to the relatively low number of ATP-consuming molecules such as ion pumps and actin-remodeling proteins [31].”

“Within neuroscience, many of our current best models for cognitive capacities treat them, metaphorically, as the product of intelligent design (Chirimuuta, 2022; Sterling and Laughlin, 2015).”

“Extensions to tensor fields and explicit network topology are natural next steps [26, 27].”

“This problem is simple enough to illustrate a standard workflow in Spark, yet challenging enough for present SNNs. Additionally, a start-to-end example could provide a rough sketch for how to deal with other different problems.”

“By enhancing responses to these frequencies, the I Cl[Ca] -mediated feedback loop restores their Signal-to-Noise Ratio (SNR), retaining information about slowly changing features of the visual scene, that is relevant for, for example, detection of slow motion.”

“This reflects the metabolic demands of execuIve funcIons, yet adapIve mechanisms balance energy use with sustained performance (Li et al., 2020; Padamsey et al., 2023; Raichle et al., 2002; Sterling et al., 2015).”

“In arthropod and vertebrate visual systems, graded voltage signals are processed over one or two synaptic stages before the information is recoded into trains of action potentials (Sterling and Laughlin 2015).”

“This is particularly necessary in the case of disorders such as autism spectrum disorders (ASD) [18] which have high inter subject variability.”

“For another, human brains are substantially constrained by powerful energetic constraints (Sterling and Laughlin, 2015).”

“First, synaptic growth in-493 curs supralinear energy costs, making unrestricted expansion metabolically inefficient ( Sterling and Laughlin, 2015 ).”

“A foundational principle enabling such high-performance information processing is ‘matched filtering’, whereby sensorimotor systems are specialized to process only task-relevant information 10–13 .”

“Recent work has suggested that metabolic efficiency, or bits/spike, is a more potent constraint on neural design than coding efficiency, or bits/second (55, 56).”

“Some infants did not fit any of Ainsworth’s three type A, B, or C classifications, but instead showed a confused mix of behaviors which involved a mix of type A and C behaviors, disorientation, and threat responses (Landa & Duschinsky, 2013b).”

“• Thepotential 𝑉 ∈ C ( R , R ) ischosensothatAssumptions 1 • The variable 𝑥 𝑠 ( 𝑡 ) ∈ R of (1b) serves as modulating • The time constants ( 𝜏, 𝜏 𝑠 ) adjust the timescale separation of 𝑠 ≫ 𝜏 > 0, so that 𝑥 𝑠 evolves more slowly than x , and 𝛼 is the input gain.”

“Spiking neurons generally have refractory periods that enforce longer average interspike intervals (51–53).”

“Such constraints been suggested to give rise to a variety of neural phenomena [Sterling and Laughlin, 2015], for example disentangled representations [Pehlevan et al., 2017, Olshausen and Field, 1996, Plumbley, 2003, Whittington et al., 2023].”

“In this case, the goal is to identify mechanisms from neuroscience [5], [6] and apply them to design an intelligent, autonomous mobile sensor platform that senses, perceives, acts, adapts, and learns in real-time and within a limited energy budget.”

“The human brain is far “smarter” than a current generation supercomputer yet consumes 100,000-fold less space and energy (Sterling & Laughlin, 2015).”