Learning with sparse reward in a gap junction network inspired by the insect mushroom body
Open accessAbstract
Animals can learn in real-life scenarios where rewards are often only available when a goal is achieved. This ‘distal’ or ‘sparse’ reward problem remains a challenge for conventional reinforcement learning algorithms. Here we investigate an algorithm for learning in such scenarios, inspired by the possibility that axo-axonal gap junction connections, observed in neural circuits with parallel fibres such as the insect mushroom body, could form a resistive network. In such a network, an active node represents the task state, connections between nodes represent state transitions and their connection to actions, and current flow to a target state can guide decision making. Building on evidence that gap junction weights are adaptive, we propose that experience of a task can modulate the connections to form a graph encoding the task structure. We demonstrate that the approach can be used for efficient reinforcement learning under sparse rewards, and discuss whether it is plausible as an account of the insect mushroom body.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The bee faces a classic problem of sparse reward: successful foraging only pays off upon return to the hive, so proper behaviour must be shaped in the absence of immediate reward.
“The bee also faces a classic problem of sparse reward [23]: successful foraging only pays off upon return to the hive, which means that proper behaviour must be shaped in the absence of immediate reward.”
From Phenomenal interface theory: a model for basal consciousness (2025) · cites this paper as [23]
The Introduction (page not provided) describes sparse reward as learning appropriate actions when only the final goal state provides reinforcement, and says insects appear capable of doing so. It does not say that a bee’s foraging reward arrives only upon return to the hive; that specific account is unsupported by our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredExperience-dependent modulation of gap junctions connecting the neurons of the mushroom bodies could support the mapping of states to outcomes such that a sequence of actions can be learned when the reward is obtained only on completion of the sequence.
“Experience-dependent modulation of gap junctions connecting the neurons of the mushroom bodies could support the mapping of states to outcomes such that a sequence of actions can be learned when the reward is obtained only on completion of the sequence [23].”
From Phenomenal interface theory: a model for basal consciousness (2025) · cites this paper as [23]
The mushroom bodies could have the capacity to solve the sparse reward problem.
“More recent work [23] exploring the theoretical capacities of gap junctions in the mushroom body network suggests the mushroom bodies could have the capacity to solve the so-called ‘sparse reward’ problem”
From Phenomenal interface theory: a model for basal consciousness (2025) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot the international journal of robotics research and application · doi:10.1108/ir.1999.04926fae.001 · Cited by 39,228 · see citing works
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641 · see citing works
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,464 · cited 1× · see citing works
“This performance is comparable with the original results for this task presented in the work by Dietterich [37].”
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355 · see citing works
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
Mushroom body memoir: from maps to models
Heisenberg, M. (2003) · Nature Reviews Neuroscience · doi:10.1038/nrn1074 · Cited by 1,286 · cited 1× · see citing works
“The mushroom body (MB) is an increasingly well-studied circuit in the insect brain [8–10] which plays an important role in learning and cognition [11–13].”
A connectome and analysis of the adult Drosophila central brain
Scheffer, L. K., Xu, C. S., Januszewski, M., Lu, Z., Takemura, S.-Y., Hayworth, K. J., Huang, G. B., Shinomiya, K., Maitlin-Shepard, J., Berg, S., et al. (2020) · eLife · doi:10.7554/elife.57443 · Cited by 1,128 · cited 2× · see citing works
“Recent research on the mushroom body has identified abundant connections between KCs [10, 14, 15, 18].”
The neuronal architecture of the mushroom body provides a logic for associative learning
Aso, Y., Hattori, D., Yu, Y., Johnston, R. M., Iyer, N. A., Ngo, T.-T., Dionne, H., Abbott, L., Axel, R., Tanimoto, H., et al. (2014) · eLife · doi:10.7554/elife.04577 · Cited by 1,051 · see citing works
Mushroom body output neurons encode valence and guide memory-based action selection in Drosophila
Aso, Y., Sitaraman, D., Ichinose, T., Kaun, K. R., Vogt, K., Belliart-Guérin, G., Plaçais, P.-Y., Robie, A. A., Yamagata, N., Schnaitmann, C., et al. (2014) · eLife · doi:10.7554/elife.04580 · Cited by 707 · cited 1× · see citing works
“Each MBON reads out from a large proportion of the KCs and connects to other regions in the insect brain, with different MBON activities linked to the production of different motivated actions such as approach or avoidance [17].”
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · see citing works
Learning to Navigate in Complex EnvironmentsPreprint
Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A. J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03673 · Cited by 679 · see citing works
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 668 · see citing works
Determining lightness from an image
Horn, B. K. P. (1974) · Computer Graphics and Image Processing · doi:10.1016/0146-664x(74)90022-7 · Cited by 645 · see citing works
Chittka, L., & Niven, J. (2009) · Current Biology · doi:10.1016/j.cub.2009.08.023 · Cited by 598 · see citing works
Fast Poisson disk sampling in arbitrary dimensions
Bridson, R. (2007) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1278780.1278807 · Cited by 542 · cited 1× · see citing works
“For our simulation, they are generated with Poisson disk sampling [38] which ensures that the generated locations are not too close to each other.”
The complete connectome of a learning and memory centre in an insect brain
Eichler, K., Li, F., Litwin-Kumar, A., Park, Y., Andrade, I. V., Schneider-Mizell, C. M., Saumweber, T., Huser, A., Eschbach, C., Gerber, B., et al. (2017) · Nature · doi:10.1038/nature23455 · Cited by 494 · see citing works
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 486 · cited 1× · see citing works
“One broad category is curiosity-driven learning [25, 26, 42, 43], which assumes there exist internal rewards related to seeking information in addition to the external reward.”
Schmitz, D., Schuchmann, S., Fisahn, A., Draguhn, A., Buhl, E. H., Petrasch‐Parwez, E., Dermietzel, R., Heinemann, U., & Traub, R. D. (2001) · Neuron · doi:10.1016/s0896-6273(01)00410-x · Cited by 456 · cited 1× · see citing works
“To date, suggested functional roles of such interactions include synchronization, regulation of oscillations [2, 5], linear interpolation [6] and faster communication [7] between neurons.”
Handbook Of Behavioral Neuroscience
(2008) · Handbook of behavioral neuroscience · doi:10.1016/s1569-7339(08)00237-3 · Cited by 425 · see citing works
The connectome of the adult Drosophila mushroom body provides insights into function
Li, F., Lindsey, J. W., Marin, E. C., Otto, N., Dreher, M., Dempsey, G., Stark, I., Bates, A. S., Pleijzier, M. W., Schlegel, P., et al. (2020) · eLife · doi:10.7554/elife.62576 · Cited by 403 · cited 4× · see citing works
“The mushroom body (MB) is an increasingly well-studied circuit in the insect brain [8–10] which plays an important role in learning and cognition [11–13].”
InfluentialLarge-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401 · cited 1× · see citing works
“One broad category is curiosity-driven learning [25, 26, 42, 43], which assumes there exist internal rewards related to seeking information in addition to the external reward.”
The hippocampus as a cognitive graph.
Muller, R. U., Stead, M., & Pach, J. (1996) · The Journal of general physiology · doi:10.1085/jgp.107.6.663 · Cited by 391 · see citing works
A connectome of a learning and memory center in the adult Drosophila brain
Takemura, S.-Y., Aso, Y., Hige, T., Wong, A., Lu, Z., Xu, C. S., Rivlin, P. K., Hess, H., Zhao, T., Parag, T., et al. (2017) · eLife · doi:10.7554/elife.26975 · Cited by 367 · cited 3× · see citing works
“For the most part, these KC-KC connections occur between KC axons where they converge onto a single MBON post-synaptic density, sometimes forming ‘rosettes’ with a set of KC synapses surrounding the MBON [14, 19].”
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357 · cited 1× · see citing works
“We note this means the model could potentially account for devaluation phenomena [41].”
Traub, R. D., Schmitz, D., Jefferys, J. G. R., & Draguhn, A. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00755-6 · Cited by 286 · see citing works
Cellular-Resolution Population Imaging Reveals Robust Sparse Coding in the Drosophila Mushroom Body
Honegger, K. S., Campbell, R. A. A., & Turner, G. C. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.1099-11.2011 · Cited by 250 · cited 1× · see citing works
“Each KC only attaches to and reads from about one to seven PNs, such that the sensory inputs are mapped to a sparse code in a high-dimensional space [16].”
The Drosophila Mushroom Body: From Architecture to Algorithm in a Learning Circuit
Modi, M. N., Shuai, Y., & Turner, G. C. (2020) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-0621333 · Cited by 216 · see citing works
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · cited 1× · see citing works
“Separating learning of the task dynamics and the reward structure is also a key concept for the successor representation approach to RL [46, 47].”
Gap junction-mediated electrical transmission: Regulatory mechanisms and plasticity
Pereda, A. E., Curti, S., Hoge, G., Cachope, R., Flores, C. E., & Rash, J. E. (2012) · Biochimica et Biophysica Acta (BBA) - Biomembranes · doi:10.1016/j.bbamem.2012.05.026 · Cited by 177 · cited 1× · see citing works
“Our model brings these ideas together, additionally inspired by the demonstration that gap junctions can be heterotypic (i.e., with asymmetric ion flow) and that there can be significant plasticity in these connections [30].”
InfluentialElectrical coupling between pyramidal cells in adult cortical regions
Mercer, A., Bannister, A. P., & Thomson, A. M. (2007) · Journal of Neurocytology · doi:10.1007/s11068-006-9005-9 · Cited by 126 · cited 2× · see citing works
“…forming the output from pyramidal cells create an ‘axonal plexus’ [2] through gap junction connections, such that action potentials in one neuron create spikelets (or even trigger action potentials) in neighbouring axons [3] and thus affect downstream chemical signalling by the neighbours [4].”
Increasing the Action Gap: New Operators for Reinforcement Learning
G. Bellemare, M., Ostrovski, G., Guez, A., Thomas, P. S., & Munos, R. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.10303 · Cited by 123 · see citing works
Heterotypic Gap Junctions between Two Neurons in the Drosophila Brain Are Critical for Memory
Wu, C., Shih, M.-F. M., Lai, J. S.-Y., Yang, H.-T., Turner, G. C., Chen, L., & Chiang, A. (2011) · Current Biology · doi:10.1016/j.cub.2011.02.041 · Cited by 112 · cited 1× · see citing works
“Knock-out of gap junctions in the dorsal paired medial (DPM) and anterior paired lateral (APL) neurons, which both receive multiple KC inputs, impairs anes-thesia-sensitive memory in odour-shock learning [21, 22].”
Outcome expectations drive learned behaviour in larval Drosophila
Gerber, B., & Hendel, T. (2006) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2006.3673 · Cited by 93 · cited 2× · see citing works
“As show in Fig 12 this qualitatively matches the data in [39].”
Analogue computation of collision-free paths
Tarassenko, L., & Blake, A. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1991.131636 · Cited by 90 · see citing works
Connectomics and function of a memory network: the mushroom body of larval Drosophila
Thum, A. S., & Gerber, B. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2018.10.007 · Cited by 89 · see citing works
A biologically inspired neural net for trajectory formation and obstacle avoidance
Glasius, R., Komoda, A., & Gielen, S. C. A. M. (1996) · Biological Cybernetics · doi:10.1007/bf00209422 · Cited by 83 · see citing works
Spatial Learning and Action Planning in a Prefrontal Cortical Network Model
Martinet, L., Sheynikhovich, D., Benchenane, K., & Arleo, A. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002045 · Cited by 79 · see citing works
Localized inhibition in the Drosophila mushroom body
Amin, H., Apostolopoulou, A. A., Suárez-Grimalt, R., Vrontou, E., & Lin, A. C. (2020) · eLife · doi:10.7554/elife.56954 · Cited by 66 · see citing works
Visualization of a Distributed Synaptic Memory Code in the Drosophila Brain
Bilz, F., Geurten, B. R. H., Hancock, C. E., Widmann, A., & Fiala, A. (2019) · Neuron · doi:10.1016/j.neuron.2020.03.010 · Cited by 65 · see citing works
Robust coding of flow-field parameters by axo-axonal gap junctions between fly visual interneurons
Cuntz, H., Haag, J., Förstner, F., Segev, I., & Borst, A. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0703697104 · Cited by 63 · cited 1× · see citing works
“To date, suggested functional roles of such interactions include synchronization, regulation of oscillations [2, 5], linear interpolation [6] and faster communication [7] between neurons.”
Learning with reinforcement prediction errors in a model of the Drosophila mushroom body
Bennett, J., Philippides, A. O., & Nowotny, T. (2021) · Nature Communications · doi:10.1038/s41467-021-22592-4 · Cited by 61 · cited 1× · see citing works
“We note these two tasks are much more complex than the typical reinforcement learning tasks used in evaluating models of the MB [34–36].”
Gap junction networks in mushroom bodies participate in visual learning and memory in Drosophila
Liu, Q., Yang, X., Tian, J., Gao, Z., Wang, M., Li, Y., & Guo, A. (2016) · eLife · doi:10.7554/elife.13238 · Cited by 60 · see citing works
Gap junction plasticity as a mechanism to regulate network-wide oscillations
Pernelle, G., Nicola, W., & Clopath, C. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006025 · Cited by 58 · see citing works
Shih, M.-F. M., Davis, F. P., Henry, G. L., & Dubnau, J. (2018) · G3 Genes Genomes Genetics · doi:10.1534/g3.118.200726 · Cited by 55 · cited 1× · see citing works
“Indeed, the functional relevance and properties of KC-KC connections is an open question, as the extent of these connections has only recently been recognised, and there is still some debate as to the extent of gap junction vs. chemical synapse connectivity [68].”
Lateral Axonal Modulation Is Required for Stimulus-Specific Olfactory Conditioning in Drosophila
Manoim, J. E., Davidson, A. M., Weiss, S., Hige, T., & Parnas, M. (2022) · Current Biology · doi:10.1016/j.cub.2022.09.007 · Cited by 54 · see citing works
Dopamine-mediated interactions between short- and long-term memory dynamics
Huang, C., Luo, J., Woo, S. J., Roitman, L. A., Li, J., Pieribone, V. A., Kannan, M., Vasan, G., & Schnitzer, M. J. (2024) · Nature · doi:10.1038/s41586-024-07819-w · Cited by 53 · see citing works
NMDAR-mediated modulation of gap junction circuit regulates olfactory learning in C. elegans
Choi, M.-K., Liu, H., Wu, T., Yang, W., & Zhang, Y. (2020) · Nature Communications · doi:10.1038/s41467-020-17218-0 · Cited by 47 · see citing works
Predictive olfactory learning in Drosophila
Zhao, C., Widmer, Y. F., Diegelmann, S., Petrovici, M. A., Sprecher, S. G., & Senn, W. (2021) · Scientific Reports · doi:10.1038/s41598-021-85841-y · Cited by 45 · see citing works
A Mechanistic Model for Reward Prediction and Extinction Learning in the Fruit Fly
Springer, M., & Nawrot, M. P. (2021) · eNeuro · doi:10.1523/eneuro.0549-20.2021 · Cited by 33 · see citing works
Robot path planning using VLSI resistive grids
Marshall, G. F. (1994) · IEE Proceedings - Vision Image and Signal Processing · doi:10.1049/ip-vis:19941325 · Cited by 31 · cited 1× · see citing works
“The platforms for the computation include Field Programmable Analog Arrays [52], Very Large-scale Integration [24, 51, 59], and circuit simulators such as PSPICE [55].”
Shyu, W.-H., Lee, W.-P., Chiang, M.-H., Chang, C. C., Fu, T., Chiang, H., Wu, T., & Wu, C. (2019) · PLoS Genetics · doi:10.1371/journal.pgen.1008153 · Cited by 29 · cited 2× · see citing works
“On the other hand, there is no direct evidence to date that KC-KC gap junction connections can be modified in the way we propose, although several authors have speculated that gap junction modulation could play a role in associative learning in the MB [23, 67].”
Rapid path planning for robotic manipulators usinganemulated resistive grid
Althöfer, K., Fraser, D. A., & Bugmann, G. (1995) · Electronics Letters · doi:10.1049/el:19951313 · Cited by 23 · see citing works
Robot path planning using Field Programmable Analog Arrays
Koziol, S., Hasler, P. E., & Stilman, M. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225303 · Cited by 22 · see citing works
Long-term potentiation in an innexin-based electrical synapse
Welzel, G., & Schuster, S. (2018) · Scientific Reports · doi:10.1038/s41598-018-30966-w · Cited by 18 · see citing works
Endotaxis: A neuromorphic algorithm for mapping, goal-learning, navigation, and patrolling
Zhang, T., Rosenberg, M., Jing, Z., Perona, P., & Meister, M. (2023) · eLife · doi:10.7554/elife.84141 · Cited by 17 · see citing works
Reconfigurable Analog VLSI circuits for robot path planning
Koziol, S., & Hasler, P. E. (2011) · NASA/ESA Conference on Adaptive Hardware and Systems (AHS) · doi:10.1109/ahs.2011.5963964 · Cited by 15 · see citing works
Physical Maze Solvers. All Twelve Prototypes Implement 1961 Lee Algorithm
Adamatzky, A. (2016) · Emergence, complexity and computation · doi:10.1007/978-3-319-46376-6_23 · Cited by 12 · see citing works
Gap Junctions in the Brain: Hardwired but Functionally Versatile
Gutiérrez, R. (2022) · The Neuroscientist · doi:10.1177/10738584221120804 · Cited by 7 · cited 1× · see citing works
“It is widely accepted that gap junction connections between neurons could play a role in the communication and computations performed in biological nervous systems [1].”
A path planner of mobile robot based on multi-grid circuit map
Yun, L., Liu, Z., Sun, H., & Yuan, J. (2005) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc.2005.1527140 · Cited by 6 · see citing works
Exploring a Novel Electrical-Modeling-Based Route Planning for Vehicle Guidance
Callejas-Molina, R. A., Díaz-Carmona, J., Vázquez-Leal, H., Mayorga-Cruz, D., & Leal, R. L. (2020) · Mathematical Problems in Engineering · doi:10.1155/2020/4348964 · Cited by 6 · see citing works
Endotaxis: A neuromorphic algorithm for mapping, goal-learning, navigation, and patrollingPreprint
Zhang, T., Rosenberg, M., Jing, Z., Perona, P., & Meister, M. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.24.461751 · Cited by 6 · cited 1× · see citing works
“A similar model that uses ‘virtual odours’ in a learned graph [66] to move towards goals draws a comparison to the mushroom body architecture, and similarly suggests that recurrent KC connectivity could form the substrate for the connection between graph nodes.”
Endotaxis: A neuromorphic algorithm for mapping, goal-learning, navigation, and patrolling
Zhang, T., Rosenberg, M., Jing, Z., Perona, P., & Meister, M. (2024) · eLife · doi:10.7554/elife.84141.3 · Cited by 6 · see citing works
Parallel Analogue Computation for Real-Time Path Planning
Tarassenko, L., Marshall, G. F., Gomez-Castaneda, F., & Murray, A. (1991) · Springer eBooks · doi:10.1007/978-1-4615-3752-6_9 · Cited by 5 · see citing works
Munro, E., & Kopell, N. (2012) · Journal of Neurophysiology · doi:10.1152/jn.00709.2011 · Cited by 5 · cited 1× · see citing works
“…forming the output from pyramidal cells create an ‘axonal plexus’ [2] through gap junction connections, such that action potentials in one neuron create spikelets (or even trigger action potentials) in neighbouring axons [3] and thus affect downstream chemical signalling by the neighbours [4].”
Matsumoto, J., Makino, Y., Miura, H., & Yano, M. (2011) · Biological Cybernetics · doi:10.1007/s00422-011-0454-6 · Cited by 4 · cited 1× · see citing works
“Learning of state, or more explicitly, place, connections to form a topological graph has been investigated in a range of hippocampus-inspired models, often linked with activity propagation methods for planning, e.g. [62–65].”
Robot's path planning based on emulated finite resistive grids
Petavratzis, E. K., VOLOS, C. K., Stouboulos, I. N., Kyprianidis, I. M., Nistazakis, H. E., & Tombras, G. S. (2018) · International Conference on Modern Circuits and Systems Technologies · doi:10.1109/mocast.2018.8376608 · Cited by 4 · cited 2× · see citing works
“Later, it was developed for hardware-accelerated path planning, particularly as an improvement over potential field methods, and applied to scenarios such as mazes [24, 50–53], grid worlds with obstacles [54, 55], city roads [56], and robot arms [57, 58].”
Huerta-Chua, J., Diaz-Arango, G. U., Vázquez-Leal, H., Méndez, J. F., Moreno, M., Ambrosio, R., & Hernández-Mejía, C. (2021) · Sensors · doi:10.3390/s21093274 · Cited by 4 · cited 1× · see citing works
“Later, it was developed for hardware-accelerated path planning, particularly as an improvement over potential field methods, and applied to scenarios such as mazes [24, 50–53], grid worlds with obstacles [54, 55], city roads [56], and robot arms [57, 58].”
Gap Junctions Underlying Labile Memory
Shih, M.-F. M., & Wu, C. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-803471-2.00003-5 · Cited by 2 · see citing works
Ayrinhac, S. (2018) · Emergence, complexity and computation · doi:10.1007/978-3-319-77510-4_16 · Cited by 2 · cited 1× · see citing works
“A range of equivalent physical implementations of the principle have been suggested [60], such as using a thermal camera to image the current flow in a graph instantiated as a printed circuit board [61].”
Kick, D. R., & Schulz, D. J. (2022) · Journal of Neuroscience · doi:10.1523/jneurosci.2402-20.2022 · Cited by 1 · cited 1× · see citing works
“Indeed, a number of factors can dynamically regulate the number of gap junctions that are open or closed, or that are present on the membrane, and activity-dependent depression and potentiation have been demonstrated in invertebrate gap junctions [31–33].”
Flagged first, then by citations
Research works printed but not linked: 7
- [25]Pathak D, Agrawal P, Efros AA, Darrell T. Curiosity-driven exploration by self-supervised prediction. In: International Conference on Machine Learning; 2017. p. 2778– 2787.
- [27]Sekar R, Rybkin O, Daniilidis K, Abbeel P, Hafner D, Pathak D. Planning to explore via self-supervisedworld models. 37th International Conference on Machine Learning. 2020; p. 8541–8550.
- [42]Bellemare MG, Srinivasan S, Ostrovski G, Schaul T, Saxton D, Munos R. Unifying count-based exploration and intrinsic motivation. In: Proceedings of the 30th International Conference on Neural Information Processing Systems; 2016. p. 1479–1487.
- [53]Marshall GF, Tarassenko L. Robot path planning using resistive grids. In: Second International Conference on Artificial Neural Networks; 1991. p. 149–152.
- [71]Jafari O, Maurya P, Nagarkar P, Islam KM, Crushev C. A survey on locality sensitive hashing algorithms and their applications. arXiv preprint arXiv:210208942. 2021;.
- [72]Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal Policy Optimization Algorithms. arXiv. 2017;.
- [73]Schnitzer M, Huang C, Luo J, Woo SJ, Roitman L, Li J, et al. Dopamine signals integrate innate and learnt valences to regulate memory dynamics. Research Square. 2022;.
Cited by
Sorted by citations
Phenomenal interface theory: a model for basal consciousness
Klein, C., & Barron, A. B. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2024.0301 · Cited by 8 · see citing works
Lochner, S., Honerkamp, D., Valada, A., & Straw, A. D. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1460006 · Cited by 7 · see citing works
Reinforcement learning in densely recurrent biological networks
Churchland, M. W., & García‐Ojalvo, J. (2025) · iScience · doi:10.1016/j.isci.2025.114436
“8 These examples highlight the promise—and challenge—of training highly recurrent natural networks.”
AGAM: A randomly initialized policy network for action mask fusion in reinforcement learning
Xue, B., Chen, Q., Wang, Z., Du, L., Zhao, J., & Hu, T. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116563
- Authors
- Wei, T., Guo, Q., & Webb, B.
- Affiliations
- Sun Yat-sen University (CN)
- Huawei Technologies (China) (CN)
- University of Edinburgh (GB)
- Corresponding author
- Wei, Tianqi
- Guo, Qinghai
- Webb, Barbara
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-05-23
- Volume
- 20
- Issue
- 5
- Pages
- e1012086
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC11152299
- OpenAlex
- W4398258546
- Cited by
- 4
- References
- 70
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1
- FWCI
- 0.59
- Type
- journal article
- Language
- en
- Update notices
- new_version (2024-06-05)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- Huawei Technologies (YBN2020045132)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurobiology and Insect Physiology Research
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Keywords
- reinforcement learning, gap junctions, parallel fibers, electric current, decision-making
- MeSH terms
- Algorithms, Animals, Insecta, Learning, Models, Neurological, Nerve Net, Reward, Gap Junctions, Computational Biology, Mushroom Bodies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
