Abstraction for Deep Reinforcement Learning
Open accessAbstract
We characterise the problem of abstraction in the context of deep reinforcement learning. Various well established approaches to analogical reasoning and associative memory might be brought to bear on this issue, but they present difficulties because of the need for end-to-end differentiability. We review developments in AI and machine learning that could facilitate their adoption.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
An LLM encodes knowledge in the way an encyclopedia does, and one emergent property is that it encodes everyday-world knowledge that no encyclopedia captures.
“it can reasonably be claimed that one emergent property of an LLM is that it encodes kinds of knowledge of the everyday world and the way it works that no encyclopedia captures.18”
From Talking about Large Language Models (2024) · cites this paper as [18]
Section 3.5, “Abstraction Through Language” (page unavailable), discusses large language models as a possible route to abstraction and notes GPT-3’s few-shot generalisation. It does not say that LLMs encode everyday-world knowledge unavailable in encyclopedias. I searched our copy for the encyclopedia comparison and for LLMs’ commonsense knowledge. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
(2017) · International Conference on Learning Representations · Cited by 3,248 · see citing works
Kronenfeld, D. B. (1978) · Language · doi:10.2307/412850 · Cited by 3,094 · see citing works
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550 · see citing works
Recurrent World Models Facilitate Policy EvolutionPreprint
Ha, D. R., & Schmidhuber, J. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1809.01999 · Cited by 1,592 · see citing works
The structure-mapping engine: Algorithm and examples
Falkenhainer, B., Forbus, K. D., & Gentner, D. (1989) · Artificial Intelligence · doi:10.1016/0004-3702(89)90077-5 · Cited by 1,454 · see citing works
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310 · see citing works
Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought
Hofstadter, D. R. (1995) · Cited by 527 · see citing works
Object-Centric Learning with Slot AttentionPreprint
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., & Kipf, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15055 · Cited by 399 · see citing works
A theory of the discovery and predication of relational concepts.
Doumas, L. A. A., Hummel, J. E., & Sandhofer, C. M. (2008) · Psychological Review · doi:10.1037/0033-295x.115.1.1 · Cited by 354 · see citing works
The copycat project: A model of mental fluidity and analogy-making
Hofstadter, D. R., & Mitchell, M. (1995) · Cited by 271 · see citing works
Self-citationMONet: Unsupervised Scene Decomposition and Representation
Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., & Lerchner, A. (2019) · arXiv · doi:10.48550/arxiv.1901.11390 · Cited by 270 · see citing works
Imagination-Augmented Agents for Deep Reinforcement LearningPreprint
Weber, T., Racanière, S., Reichert, D., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06203 · Cited by 267 · see citing works
Multi-Object Representation Learning with Iterative Variational InferencePreprint
Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., & Lerchner, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.00450 · Cited by 258 · see citing works
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations
Garnelo, M., & Shanahan, M. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.12.010 · Cited by 255 · see citing works
Self-citationOrganization of the Hearsay II speech understanding system
Lesser, V., Fennell, R. D., Erman, L. D., & Reddy, D. (1975) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/tassp.1975.1162648 · Cited by 247 · see citing works
Hopfield Networks is All You Need
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., et al. (2020) · arXiv · doi:10.48550/arxiv.2008.02217 · Cited by 236 · see citing works
Measuring abstract reasoning in neural networksPreprint
Barrett, D. G. T., Hill, F., Santoro, A., Morcos, A. S., & Lillicrap, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.04225 · Cited by 233 · see citing works
Darla Improving zero-shot transfer in reinforcement learning
Higgins, I., Pal, A., Rusu, A. A., Matthey, L., Burgess, C. P., Pritzel, A., Botvinick, M., Blundell, C., & Lerchner, A. (2017) · arXiv · doi:10.48550/arxiv.1707.08475 · Cited by 219 · see citing works
A Survey of Reinforcement Learning Informed by Natural Language
Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas, J., Grefenstette, E., Whiteson, S., & Rocktäschel, T. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/880 · Cited by 208 · see citing works
A cognitive architecture that combines internal simulation with a global workspace
Shanahan, M. (2005) · Consciousness and Cognition · doi:10.1016/j.concog.2005.11.005 · Cited by 195 · see citing works
Self-citationRAVEN: A Dataset for Relational and Analogical Visual REasoNing
Zhang, C., Gao, F., Jia, B., Zhu, Y., & Zhu, S. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00546 · Cited by 189 · see citing works
Dense Associative Memory for Pattern RecognitionPreprint
Krotov, D., & Hopfield, J. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01164 · Cited by 162 · see citing works
Analogical mapping by constraint satisfaction
Holyoak, K. J. (1989) · Cognitive Science · doi:10.1016/0364-0213(89)90016-5 · Cited by 151 · see citing works
Towards Deep Symbolic Reinforcement Learning
Garnelo, M., Arulkumaran, K., & Shanahan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05518 · Cited by 146 · see citing works
Self-citationAndreas, J., Klein, D., & Levine, S. (2017) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n18-1197 · Cited by 144 · see citing works
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent RepresentationsPreprint
Engelcke, M., Kosiorek, A. R., Jones, Ō. P., & Posner, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.13052 · Cited by 141 · see citing works
On a Model of Associative Memory with Huge Storage Capacity
Demircigil, M., Heusel, J., Löwe, M., Upgang, S., & Vermet, F. (2017) · Journal of Statistical Physics · doi:10.1007/s10955-017-1806-y · Cited by 136 · see citing works
(2019) · Cited by 135 · see citing works
DreamCoder: growing generalizable, interpretable knowledge with wake–sleep Bayesian program learning
Ellis, K., Wong, L., Nye, M., Sablé-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A., & Tenenbaum, J. B. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0050 · Cited by 125 · see citing works
Abstraction and analogy‐making in artificial intelligence
Mitchell, M. (2021) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.14619 · Cited by 125 · see citing works
Self-citationRepresentation Learning via Invariant Causal MechanismsPreprint
Mitrovic, J., McWilliams, B., Walker, J., Buesing, L., & Blundell, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07922 · Cited by 106 · see citing works
On the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102 · see citing works
Investigating Human Priors for Playing Video GamesPreprint
Dubey, R., Agrawal, P., Pathak, D., Griffiths, T. L., & Efros, A. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10217 · Cited by 101 · see citing works
Smolensky, P. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/2102.003.0006 · Cited by 86 · see citing works
SCAN: Learning Hierarchical Compositional Visual ConceptsPreprint
Higgins, I., Sonnerat, N., Matthey, L., Pal, A., Burgess, C., Bošnjak, M., Shanahan, M., Botvinick, M., Hassabis, D., & Lerchner, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03389 · Cited by 79 · see citing works
Self-citationBootstrap Latent-Predictive Representations for Multitask Reinforcement LearningPreprint
Guo, Z. D., Pires, B. Á., Piot, B., Grill, J.-B., Altché, F., Munos, R., & Azar, M. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14646 · Cited by 78 · see citing works
A Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63 · see citing works
Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs
Lázaro-Gredilla, M., Lin, D., Guntupalli, J. S., & Dileep George, A. (2019) · Science Robotics · doi:10.1126/scirobotics.aav3150 · Cited by 61 · see citing works
Artificial Intelligence and the Common Sense of Animals
Shanahan, M., Crosby, M., Beyret, B., & Cheke, L. G. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.002 · Cited by 59 · see citing works
Self-citationCoordination among neural modules through a shared global workspace
Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., & Bengio, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.01197 · Cited by 59 · see citing works
Evans, R., Hernández-Orallo, J., Welbl, J., Kohli, P., & Sergot, M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103438 · Cited by 58 · see citing works
LISA: A computational model of analogical inference and schema induction
Hummel, J. E., & Holyoak, K. J. (1996) · eScholarship (California Digital Library) · Cited by 52 · see citing works
Temporal Difference Variational Auto-EncoderPreprint
Gregor, K., Papamakarios, G., Besse, F., Buesing, L., & Weber, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03107 · Cited by 46 · see citing works
Structure-Mapping: A Theoretical Framework for Analogy
Gentner, D. (1988) · Elsevier eBooks · doi:10.1016/b978-1-4832-1446-7.50026-1 · Cited by 40 · see citing works
Predicate learning in neural systems: using oscillations to discover latent structure
Martin, A. E., & Doumas, L. A. A. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.04.008 · Cited by 33 · see citing works
The Kanerva Machine: A Generative Distributed MemoryPreprint
Wu, Y. X., Wayne, G., Graves, A., & Lillicrap, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01756 · Cited by 29 · see citing works
The Animal-AI Testbed and Competition.
Crosby, M., Beyret, B., Shanahan, M., Hernández‐Orallo, J., Cheke, L. G., & Halina, M. (2019) · RiuNet (Universitat Politècnica de València) · Cited by 29 · see citing works
Self-citationDirect Human-AI Comparison in the Animal-AI Environment
Voudouris, K. N., Crosby, M., Beyret, B., Hernández‐Orallo, J., Shanahan, M., Halina, M., & Cheke, L. G. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.711821 · Cited by 23 · see citing works
Self-citationCoBERL: Contrastive BERT for Reinforcement LearningPreprint
Banino, A., Badia, A. P., Walker, J., Scholtes, T., Mitrovic, J., & Blundell, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05431 · Cited by 20 · see citing works
A Global Workspace Theory of Conscious Experience
Baars, B. J. (2013) · Psychology Press eBooks · doi:10.4324/9780203773659-16 · Cited by 18 · see citing works
Tell me why! Explanations support learning relational and causal structure
Lampinen, A. K., Roy, N. A., Dasgupta, I., Chan, S. C. Y., Tam, A. C., McClelland, J. L., Yan, C., Santoro, A., Rabinowitz, N. C., Wang, J. X., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.03753 · Cited by 14 · see citing works
Unsupervised Object-Based Transition Models for 3D Partially Observable EnvironmentsPreprint
Creswell, A., Kabra, R., Burgess, C., & Shanahan, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04693 · Cited by 8 · see citing works
Self-citationNeural Analogical MatchingPreprint
Crouse, M., Nakos, C., Abdelaziz, I., & Forbus, K. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03573 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- John J. Hopfield. Hopfield network. Schol- arpedia, 2(5):1977, 2007.
- George Lakoff and Mark John- son. Metaphors We Live By. University of Chicago Press, 1980.
- Gary F Marcus. The Algebraic Mind. MIT Press, 2001.
- Roger C. Schank and Robert P. Abelson. Scripts, Plans, Goals, and Understanding: An Inquiry Into Human Knowledge Structures. Psychology Press, 1977.
- Pedro A Tsividis, Thomas Pouncy, Jacqueline L Xu, Joshua B Tenenbaum, and Samuel J Ger- shman. Human learning in Atari. In AAAI Spring Sympo- sium on Science of Intelligence: Computational Principles of Natural and Artificial Intelligence, 2017.
- Mark Turner. The Literary Mind: The Origins of Thought and Language. Oxford University Press, 1998.
Cited by
Sorted by citations
Talking about Large Language Models
Shanahan, M. (2024) · Communications of the ACM · doi:10.1145/3624724 · Cited by 381 · see citing works
“Plausibly, an LLM with enough parameters trained on a sufficiently large dataset with the right statistical properties can acquire a pattern completion mechanism with a degree of generality (Shanahan and Mitchell, 2022).”
BackgroundSelf-citationA rubric for human-like agents and NeuroAI
Momennejad, I. (2022) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0446 · Cited by 41 · see citing works
Cooperative Multi-Agent Learning for Navigation via Structured State Abstraction
Abdel-Aziz, M. K., Elbamby, M. S., Samarakoon, S., & Bennis, M. (2024) · IEEE Transactions on Communications · doi:10.1109/tcomm.2024.3365520 · Cited by 12 · see citing works
: A Semantics-Guided Safety Enhancement Framework for AI-Enabled Cyber-Physical Systems
Song, J., Xie, X., & Ma, L. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3282981 · Cited by 11 · see citing works
Deane, G. (2022) · Artificial Life · doi:10.1162/artl_a_00368 · Cited by 9 · see citing works
Neurodynamical Computing at the Information Boundaries of Intelligent Systems
Monaco, J. D., & Hwang, G. M. (2022) · Cognitive Computation · doi:10.1007/s12559-022-10081-9 · Cited by 6 · see citing works
DGTRL: Deep graph transfer reinforcement learning method based on fusion of knowledge and data
Chen, G., Qi, J., Gao, Y. Q., Zhu, X., Dong, Z., & Sun, Y. (2023) · Information Sciences · doi:10.1016/j.ins.2023.120019 · Cited by 5 · see citing works
The origin and function of external representations
Fernando, C., Osindero, S., & Banarse, D. (2024) · Adaptive Behavior · doi:10.1177/10597123241262534 · Cited by 7 · see citing works
A Definition of Open-Ended Learning Problems for Goal-Conditioned AgentsPreprint
Sigaud, O., Baldassarre, G., Colas, C., Doncieux, S., Duro, R. J., Oudeyer, P.-Y., Perrin, N., & Santucci, V. G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.00344 · Cited by 4 · see citing works
Overcoming classic challenges for artificial neural networks by providing incentives and practice
Irie, K., & Lake, B. M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01121-8 · Cited by 3 · see citing works
Analogical reasoning as a core AGI capability
Firt, E. (2025) · AI and Ethics · doi:10.1007/s43681-025-00785-7 · Cited by 2 · see citing works
AIGenC: AI Generalisation via Creativity
Cătărău-Cotuţiu, C., Mondragón, E., & Alonso, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49011-8_4 · Cited by 2 · see citing works
Cognitive Escape Reinforcement Learning for Complex Decision MakingPreprint
Wu, Q., Zhao, S., Zhou, F., Zhang, H., Huang, Y., & Ma, K. (2023) · Research Square · doi:10.21203/rs.3.rs-2661516/v1 · Cited by 2 · see citing works
Cooperative Navigation via Relational Graphs and State Abstraction
Mostafa, S., Abdel-Aziz, M. K., & Bennis, M. (2023) · IEEE Networking Letters · doi:10.1109/lnet.2023.3285295 · Cited by 1 · see citing works
From Birth to Loss of Representations in Artificial Neural Networks
Stecher, P. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94748-3_20 · Cited by 1 · see citing works
Language Models Understand Us, Poorly
Moore, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.16 · Cited by 2 · see citing works
Ma, Z., Cui, Y., & Huang, Y. (2024) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10919663
- Authors
- Shanahan, M., & Mitchell, M.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Santa Fe Institute (US)
- Corresponding author
- Shanahan, Murray
- Venue
- Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
- Published
- 2022-07
- Pages
- 5588-5596
- OpenAlex
- W4285606041
- Cited by
- 17
- References
- 61
- Citations by year
- 2025: 3 · 2024: 4 · 2023: 6 · 2022: 4
- Type
- conference abstract
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- abstraction, deep reinforcement learning, analogical reasoning, associative memory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
