Multimodal representation learning for place recognition using deep Hebbian predictive coding
Open accessAbstract
Recognising familiar places is a competence required in many engineering applications that interact with the real world such as robot navigation. Combining information from different sensory sources promotes robustness and accuracy of place recognition. However, mismatch in data registration, dimensionality, and timing between modalities remain challenging problems in multisensory place recognition. Spurious data generated by sensor drop-out in multisensory environments is particularly problematic and often resolved through adhoc and brittle solutions. An effective approach to these problems is demonstrated by animals as they gracefully move through the world. Therefore, we take a neuro-ethological approach by adopting self-supervised representation learning based on a neuroscientific model of visual cortex known as predictive coding. We demonstrate how this parsimonious network algorithm which is trained using a local learning rule can be extended to combine visual and tactile sensory cues from a biomimetic robot as it naturally explores a visually aliased environment. The place recognition performance obtained using joint latent representations generated by the network is significantly better than contemporary representation learning techniques. Further, we see evidence of improved robustness at place recognition in face of unimodal sensor drop-out. The proposed multimodal deep predictive coding algorithm presented is also linearly extensible to accommodate more than two sensory modalities, thereby providing an intriguing example of the value of neuro-biologically plausible representation learning for multimodal navigation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The effectiveness of multisensory integration in predictive coding driving place recognition in mobile rodent-like robots exemplifies how this approach may help to define so-called 'indicators of consciousness' in artificial agents.
“For instance, the effectiveness of multisensory integration in predictive coding driving place recognition in mobile rodent-like robots (Pearson et al., 2021) exemplifies how this approach may help to define socalled 'indicators of consciousness' in artificial agents (Lee & Pennartz, 2025; Pennartz et al., 2019).”
From Integrated information and predictive processing theories of consciousness: An adversarial collaborative review (2026) · cites this paper as Pearson et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 4× · see citing works
“This is interesting for two reasons, fi rstly, PCN is a mechanistic implementation of the parsimonious theory that perception arises from generative, inferential representations of what causes sensory inputs to arise Helmholtz (1867); Gregory (1980); Mumford (1992); Pennartz (2015); Friston (2010).”
BackgroundInfluentialDistributed hierarchical processing in the primate cerebral cortex
Felleman, D. J., & Van Essen, D. C. (1991) · Cerebral Cortex · doi:10.1093/cercor/1.1.1 · Cited by 7,783 · see citing works
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 2× · see citing works
“The base compute unit in a PCN is the same throughout the network, referred to by Rao and Ballard as a Predictive Estimator Rao and Ballard (1999), wherein only local computation and updates are performed during training and inference.”
BackgroundMethodRepresentational similarity analysis-connecting the branches of systems neuroscience
Kriegeskorte, N. (2008) · Frontiers in Systems Neuroscience · doi:10.3389/neuro.06.004.2008 · Cited by 3,355 · see citing works
Canonical Microcircuits for Predictive Coding
Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.038 · Cited by 2,421 · cited 1× · see citing works
“Moreover, the basic feedforward-feedback structure of PCNs resemble the core architecture of the sensory neocortex Felleman and Van Essen (1991); Bastos et al. (2012); Pennartz et al. (2019).”
MethodTreatise on Physiological Optics
von Helmholtz, H., & Southall, P. C. (1962) · Medical Entomology and Zoology · Cited by 2,196 · see citing works
Multisensor data fusion: A review of the state-of-the-art
Khaleghi, B., Khamis, A. M., Karray, F. O., & Razavi, S. (2011) · Information Fusion · doi:10.1016/j.inffus.2011.08.001 · Cited by 2,097 · see citing works
Visual Place Recognition: A Survey
Lowry, S., Sünderhauf, N., Newman, P., Leonard, J. J., Cox, D., Corke, P., & Milford, M. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2496823 · Cited by 1,079 · see citing works
On the computational architecture of the neocortex: II The role of cortico-cortical loops
Mumford, D. (1992) · Biological Cybernetics · doi:10.1007/bf00198477 · Cited by 1,074 · see citing works
Gregory, R. L. (1980) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1980.0090 · Cited by 789 · see citing works
A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots
Giusti, A., Guzzi, J., Ciresan, D. C., He, F.-L., Rodriguez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Caro, G. D., et al. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2015.2509024 · Cited by 665 · see citing works
Multimodal deep learning for robust RGB-D object recognition
Eitel, A., Springenberg, J. T., Spinello, L., Riedmiller, M., & Burgard, W. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353446 · Cited by 647 · see citing works
The limits and potentials of deep learning for robotics
Sünderhauf, N., Brock, O., Scheirer, W. J., Hadsell, R. T., Fox, D., Leitner, J., Upcroft, B., Abbeel, P., Burgard, W., Milford, M. J., et al. (2018) · QUT ePrints (Queensland University of Technology) · Cited by 519 · see citing works
Neural scene representation and rendering
Eslami, S. M. A., Rezende, D. J., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al. (2018) · Science · doi:10.1126/science.aar6170 · Cited by 511 · see citing works
A review of predictive coding algorithms
Spratling, M. W. (2017) · Brain and Cognition · doi:10.1016/j.bandc.2015.11.003 · Cited by 437 · cited 1× · see citing works
“…Coding Network (PCN) is both from sensory input to output and the opposite, i.e., each layer in the network predicts representations of the previous layer in parallel, ultimately predicting the actual input being passed into the lowest layer of the network Rao and Ballard (1999); Spratling (2017).”
BackgroundRatSLAM: a hippocampal model for simultaneous localization and mapping
Milford, M. J., Wyeth, G. F., & Prasser, D. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2004.1307183 · Cited by 404 · see citing works
Control of synaptic plasticity in deep cortical networks
Roelfsema, P. R., & Holtmaat, A. J. D. G. (2018) · Nature reviews. Neuroscience · doi:10.1038/nrn.2018.6 · Cited by 245 · see citing works
Dempster-Shafer theory for sensor fusion in autonomous mobile robots
Murphy, R. R. (1998) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.681240 · Cited by 238 · see citing works
Gao, P., Bermejo, R., & Zeigler, H. P. (2001) · Journal of Neuroscience · doi:10.1523/jneurosci.21-14-05374.2001 · Cited by 181 · see citing works
Gu, Y., Wang, Y., & Li, Y. (2019) · Applied Sciences · doi:10.3390/app9102110 · Cited by 149 · see citing works
Joint Multimodal Learning with Deep Generative Models
Suzuki, M., Nakayama, K., & Matsuo, Y. (2016) · arXiv · doi:10.48550/arxiv.1611.01891 · Cited by 126 · see citing works
Prescott, T., Pearson, M., Mitchinson, B., Sullivan, J. C., & Pipe, A. (2009) · IEEE robotics & automation magazine · doi:10.1109/mra.2009.933624 · Cited by 119 · see citing works
Self-citationFalotico, E., Vannucci, L., Ambrosano, A., Albanese, U., Ulbrich, S., Tieck, J. C. V., Hinkel, G., Kaiser, J., Peric, I., Denninger, O., et al. (2017) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2017.00002 · Cited by 111 · see citing works
Adaptive Robot Body Learning and Estimation Through Predictive Coding
Lanillos, P., & Cheng, G. (2018) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros.2018.8593684 · Cited by 81 · see citing works
Towards a Unified View on Pathways and Functions of Neural Recurrent Processing
Pennartz, C. M. A., Dora, S., Muckli, L., & Lorteije, J. A. M. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.07.005 · Cited by 73 · cited 1× · see citing works
“Moreover, the basic feedforward-feedback structure of PCNs resemble the core architecture of the sensory neocortex Felleman and Van Essen (1991); Bastos et al. (2012); Pennartz et al. (2019).”
MethodSelf-citationMitchinson, B., & Prescott, T. (2013) · PLoS Comput. Biol. · doi:10.1371/journal.pcbi.1003236 · Cited by 66 · see citing works
Self-citationPredictive learning: its key role in early cognitive development
Nagai, Y. (2019) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2018.0030 · Cited by 61 · see citing works
Extended Kalman filter for frequent local and infrequent global sensor data fusion
Roumeliotis, S. I., & Bekey, G. A. (1997) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.287638 · Cited by 54 · see citing works
The Brain's Representational Power: On Consciousness and the Integration of Modalities
Pennartz, C. M. A. (2015) · doi:10.7551/mitpress/9780262029315.001.0001 · Cited by 43 · see citing works
Self-citationDefining the Pose of Any 3D Rigid Object and an Associated Distance
Brégier, R., Devernay, F., Leyrit, L., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1052-4 · Cited by 42 · see citing works
The limits and potentials of deep learning for robotics
Sünderhauf, N., Brock, O., Scheirer, W., Hadsell, R., Fox, D., Leitner, J., Upcroft, B., Abbeel, P., Burgard, W., Milford, M., et al. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364918770733 · Cited by 38 · see citing works
Simultaneous localisation and mapping on a multi-degree of freedom biomimetic whiskered robot
Pearson, M. J., Fox, C., Sullivan, J. C., et al. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630633 · Cited by 30 · see citing works
Self-citationReal-time scene understanding for UAV imagery based on deep convolutional neural networks
Sheppard, C., & Rahnemoonfar, M. (2017) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2017.8127435 · Cited by 29 · see citing works
Making BREAD: Biomimetic Strategies for Artificial Intelligence Now and in the Future
Krichmar, J. L., Severa, W., Khan, S. M., & Olds, J. L. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00666 · Cited by 26 · see citing works
Struckmeier, O., Tiwari, K., Salman, M., et al. (2019) · IEEE International Conference on Cyborg and Bionic Systems (CBS) · doi:10.1109/cbs46900.2019.9114526 · Cited by 22 · see citing works
Self-citationPredictive coding-based deep dynamic neural network for visuomotor learning
Hwang, J., Kim, J., Ahmadi, A., et al. (2017) · Joint IEEE International Conference on Development and Learning and Epigenetic Robotics · doi:10.1109/devlrn.2017.8329798 · Cited by 18 · see citing works
Jointly Trained Variational Autoencoder for Multi-Modal Sensor Fusion
Korthals, T., Hesse, M., Leitner, J., Melnik, A., & Rückert, U. (2019) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion43075.2019.9011314 · Cited by 18 · see citing works
A Deep Predictive Coding Network for Inferring Hierarchical Causes Underlying Sensory Inputs
Dora, S., Pennartz, C. M. A., & Bohté, S. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01424-7_45 · Cited by 16 · see citing works
Self-citationChoi, M., & Tani, J. (2017) · Neural Computation · doi:10.1162/neco_a_01026 · Cited by 14 · see citing works
Gist+RatSLAM: An Incremental Bio-inspired Place Recognition Front-End for RatSLAM
Kazmi, S. M. A. M., & Mertsching, B. (2016) · doi:10.4108/eai.3-12-2015.2262532 · Cited by 13 · see citing works
Predictive Coding Strategies for Developmental Neurorobotics
Park, J.-C., Lim, J. H., Choi, H., & Kim, D. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00134 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Flach P. Kull M. (2015). “ Precision-recall-gain Curves: Pr Analysis Done Right,”. Advances in Neural Information Processing Systems. Editors Cortes C. Lawrence N. Lee D. Sugiyama M. Garnett R. (Redhook: Curran Associates, Inc.), 28.
Cited by
Sorted by citations
Pennartz, C. M. A. (2022) · Behavioural Brain Research · doi:10.1016/j.bbr.2022.113969 · Cited by 40 · see citing works
Self-citationPennartz, C. M. A., Oude Lohuis, M. N., & Olcese, U. (2023) · Philosophical Transactions of the Royal Society B · doi:10.1098/rstb.2022.0336 · Cited by 38 · see citing works
Self-citationPredictive coding with spiking neurons and feedforward gist signaling
Lee, K., Dora, S., Mejias, J. F., Bohte, S. M., & Pennartz, C. M. A. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1338280 · Cited by 14 · see citing works
Self-citationCorcoran, A. W., Haun, A. M., Dorman, R., Tononi, G., Friston, K. J., & Pennartz, C. M. A. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106742 · Cited by 2 · see citing works
Self-citation
- Authors
- Pearson, M. J., Dora, S., Struckmeier, O., Knowles, T. C., Mitchinson, B., Tiwari, K., Kyrki, V., Bohte, S., & Pennartz, C. M. A.
- Affiliations
- University of the West of England (GB)
- Centrum Wiskunde & Informatica (NL)
- Aalto University (FI)
- University of Sheffield (GB)
- University of Amsterdam (NL)
- Corresponding author
- Pearson, Martin J.
- ORCID
- Venue
- Frontiers in Robotics and AI
- Publisher
- Frontiers Media SA
- Published
- 2021-12-13
- Volume
- 8
- Pages
- 732023-732023
- ISSN
- 2296-9144 (ISSN-L)
- PMC
- PMC8710724
- OpenAlex
- W4200396728
- Cited by
- 19
- References
- 44
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 5 · 2023: 6 · 2022: 5
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Robotics and AI (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands (submittedVersion, cc-by)
- UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Loughborough University Research Repository (Loughborough University) (submittedVersion, cc-by)
- Funders
- Horizon 2020 Framework Programme (945539)
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- place recognition, predictive coding, Hebbian learning, multimodal representation learning, multisensory integration, visual-tactile fusion, self-supervised learning, visual cortex model, latent representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
