Observational Learning by Reinforcement Learning
Open accessAbstract
Observational learning is a type of learning that occurs as a function of observing, retaining and possibly replicating or imitating the behaviour of another agent. It is a core mechanism appearing in various instances of social learning and has been found to be employed in several intelligent species, including humans. In this paper, we investigate to what extent the explicit modelling of other agents is necessary to achieve observational learning through machine learning. Especially, we argue that observational learning can emerge from pure Reinforcement Learning (RL), potentially coupled with memory. Through simple scenarios, we demonstrate that an RL agent can leverage the information provided by the observations of an other agent performing a task in a shared environment. The other agent is only observed through the effect of its actions on the environment and never explicitly modeled. Two key aspects are borrowed from observational learning: i) the observer behaviour needs to change as a result of viewing a 'teacher' (another agent) and ii) the observer needs to be motivated somehow to engage in making use of the other agent's behaviour. The later is naturally modeled by RL, by correlating the learning agent's reward with the teacher agent's behaviour.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work on emergent social learning in RL has only considered one iteration of information transmission from an expert to a student.
“However, these works have only considered one iteration of information transmission from an expert to a student.”
From Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning (2024) · cites this paper as Borsa et al., 2017
The Introduction (p. 2) describes this paper’s scenarios as involving a learner observing an expert, and the Conclusion (p. 8) describes an agent that learned from a teacher and could then act without one. This supports a single expert-to-student transmission in this study, but the paper does not establish that prior work generally considered only one iteration. (LLM, conf=0.83)
Experimental — accuracy not yet measuredPrior work has successfully demonstrated emergent social learning in RL, showing that it helps agents solve hard exploration problems and adapt online to new tasks.
“Prior work has successfully demonstrated emergent social learning in RL [Borsa et al., 2017, Ndousse et al., 2021, Bhoopchand et al., 2023], showing that it helps agents solve hard exploration problems and adapt online to new tasks.”
From Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning (2024) · cites this paper as Borsa et al., 2017
Machines can learn to seek and transmit information and can act as conversational and pedagogical agents, similar to teachers.
“Besides serving as a persistent medium of cultural storage analogous to a book, machines can learn to seek and transmit information 79,81 and can act as conversational and pedagogical agents, similar to teachers 82 .”
From Machine culture (2023) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Kytle, J., & Bandura, A. (1978) · Contemporary Sociology · doi:10.2307/2065952 · Cited by 25,422 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · see citing works
(1998) · Kybernetes · doi:10.1108/k.1998.27.9.1093.3 · Cited by 1,576 · see citing works
ALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · doi:10.1184/r1/6603146 · Cited by 1,403 · see citing works
Is imitation learning the route to humanoid robots?
Schaal, S. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01327-3 · Cited by 1,325 · see citing works
Learning to Navigate in Complex EnvironmentsPreprint
Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A. J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03673 · Cited by 679 · see citing works
Robot Learning from Demonstration
Atkeson, C. G., & Stefan Schaal, A. (2012) · SpringerReference · doi:10.1007/springerreference_302487 · Cited by 532 · see citing works
Imitation, culture and cognition
Heyes, C. M. (1993) · Animal Behaviour · doi:10.1006/anbe.1993.1281 · Cited by 369 · see citing works
Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion
Kolter, J. Z., Abbeel, P., & Ng, A. Y. (2007) · Cited by 122 · see citing works
Imitation learning for locomotion and manipulation
Ratliff, N., Bagnell, J. A., & Srinivasa, S. S. (2007) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2007.4813899 · Cited by 104 · see citing works
Showing versus Doing: Teaching by DemonstrationPreprint
Ho, M. K., Littman, M. L., MacGlashan, J., Cushman, F. A., & Austerweil, J. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14004 · Cited by 71 · see citing works
Training parsers by inverse reinforcement learning
Neu, G., & Szepesvári, C. (2009) · Machine Learning · doi:10.1007/s10994-009-5110-1 · Cited by 69 · see citing works
Learning from Demonstrations: Is It Worth Estimating a Reward Function?
Piot, B., Geist, M., & Pietquin, O. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40988-2_2 · Cited by 26 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 8
- [4][4] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. -
- [5][5] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In International Conference on Machine Learning (ICML) , 2015. -
- [6][6] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML) , 2016. -
- [16][16] S. Russell. Learning agents for uncertain environments. In Proceedings of the Conference on Learning Theory (COLT) , 1998. -
- [21][21] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on AI and Statistics (AISTATS) , 2011. -
- [24][24] Bilal Piot, Olivier Pietquin, and Matthieu Geist. Predicting when to laugh with structured classification. In Proceedings of INTERSPEECH , 2014. -
- [26][26] Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017. -
- [28][28] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems (AAMAS) , 2014. -
Cited by
Sorted by citations
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Cook, J., Lu, C., Hughes, E., Leibo, J. Z., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2406.00392 · Cited by 6 · see citing works
- Authors
- Borsa, D., Piot, B., Munos, R., & Pietquin, O.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Pages
- 1117-1124
- ISSN
- 1548-8403 (ISSN-L)
- arXiv
- 1706.06617
- OpenAlex
- W2964014957
- Cited by
- 47
- References
- 20
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 2 · 2023: 6 · 2022: 7 · 2021: 10 · 2020: 10 · 2019: 5 · 2018: 3
- FWCI
- 3.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Adaptive Agents and Multi-Agents Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- observational learning, reinforcement learning, social learning, imitation, emergent learning, memory, reward function design, multi-agent learning, shared environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
