CoChat
Sign Up
PreprintCited by 23Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

Emergent Social Learning via Multi-agent Reinforcement Learning

Open access
Ndousse, Kamal, Eck, Douglas, Levine, Sergey, Jaques, Natasha·2020·arXiv
DOI 10.48550/arxiv.2010.00581arXiv 2010.00581
Open this paper in CoChat — free accountOpen PDF

Abstract

Social learning is a key component of human and animal intelligence. By taking cues from the behavior of experts in their environment, social learners can acquire sophisticated behavior and rapidly adapt to new circumstances. This paper investigates whether independent reinforcement learning (RL) agents in a multi-agent environment can learn to use social learning to improve their performance. We find that in most circumstances, vanilla model-free RL agents do not use social learning. We analyze the reasons for this deficiency, and show that by imposing constraints on the training environment and introducing a model-based auxiliary loss we are able to obtain generalized social learning policies which enable agents to: i) discover complex skills that are not learned from single-agent training, and ii) adapt online to novel environments by taking cues from experts present in the new environment. In contrast, agents trained with model-free RL or imitation learning generalize poorly and do not succeed in the transfer tasks. By mixing multi-agent and solo training, we can obtain agents that use social learning to gain skills that they can deploy when alone, even out-performing agents trained alone from the start.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.00581

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Expert dropout encourages the learning of within-episode memorisation, a capability that was absent from previous agents and which confers a higher cultural transmission score.

    “This is because dropout encourages the learning of within-episode memorisation, a capability that was absent from previous agents 33 and which confers a higher cultural transmission score”

    From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [33]

    The Introduction (p. 2) and §4.3 (p. 8) report that interleaving episodes with and without experts improves agents’ solo performance. They do not establish that this training produces within-episode memorisation or a higher cultural transmission score. The claim that the capability was absent from these earlier agents is also not established: §3.2 (p. 4) says their recurrent policy models observation history within each episode. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • To use social information, agents need to notice that there are other players in the world that have similar abilities and intentions as themselves.

    “To use social information, agents need to notice that there are other players in the world that have similar abilities and intentions as themselves 33,68”

    From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [33]

    The Introduction (p. 1) says novices can see experts and must learn to use cues from them. But the paper does not say agents must recognize other players as having similar abilities and intentions. Related Work (p. 2) says its agents do not explicitly model other agents, and Limitations (p. 9) leaves experts with different goals for future work. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The MEAL variant does not learn a within-episode recall ability, even in a gridworld.

    “We know that the MEAL variant 33 does not learn a within-episode recall ability, even in a gridworld”

    From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [33]

    Section 4.2 (pp. 7–8) discusses within-episode adaptation through social learning in grid environments, but does not assess within-episode recall. Searches for “MEAL” and “within-episode recall” found no passage identifying or evaluating that variant, so our copy does not establish the claimed failure. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., and Bachem, O. What matters in on-policy reinforcement learning? a largescale empirical study. arXiv preprint arXiv:2006.05990 , 2020. · arXiv
  2. Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems , 57(5):469-483, 2009.
  3. Bain, M. and Sammut, C. A framework for behavioural cloning. In Machine Intelligence 15 , pp. 103-129, 1995.
  4. Barkow, J. H., Akiwowo, A. A., Barua, T. K., Chance, M., Chapple, E. D., Chattopadhyay, G. P., Freedman, D. G., Geddes, W., Goswami, B., Isichei, P., et al. Prestige and culture: a biosocial interpretation [and comments and replies]. Current Anthropology , 16(4):553-572, 1975.
  5. Borsa, D., Heess, N., Piot, B., Liu, S., Hasenclever, L., Munos, R., and Pietquin, O. Observational learning by reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pp. 1117-1124. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
  6. Boyd, R., Richerson, P. J., and Henrich, J. The cultural niche: Why social learning is essential for human adaptation. Proceedings of the National Academy of Sciences , 108 (Supplement\_2):10918-10925, June 2011.
  7. Calinon, S., Guenter, F., and Billard, A. On learning, representing, and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 37(2):286-298, 2007.
  8. Cheng, C.-A., Kolobov, A., and Agarwal, A. Policy improvement from multiple experts. arXiv preprint arXiv:2007.00795 , 2020. · arXiv
  9. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pp. 4299-4307, 2017.
  10. Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning , pp. 1282-1289, 2019.
  11. Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, S. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv:2012.02096 , 2020. · arXiv
  12. Guo, X., Chang, S., Yu, M., Tesauro, G., and Campbell, M. Hybrid reinforcement learning with expert state sequences. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 3739-3746, 2019.
  13. Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A. Cooperative inverse reinforcement learning. In Advances in neural information processing systems , pp. 3909-3917, 2016.
  14. Henrich, J. and McElreath, R. The evolution of cultural evolution. Evolutionary Anthropology: Issues, News, and Reviews , 12(3):123-135, May 2003.
  15. Hernandez-Leal, P., Kartal, B., and Taylor, M. E. Agent modeling as auxiliary task for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume 15, pp. 31-37, 2019.
  16. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.
  17. Horner, V., Proctor, D., Bonnie, K. E., Whiten, A., and de Waal, F. B. Prestige affects cultural learning in chimpanzees. PloS one , 5(5):e10625, 2010.
  18. Humphrey, N. K. The social function of intellect. In Growing points in ethology , pp. 303-317. Cambridge University Press, 1976.
  19. Jacq, A., Geist, M., Paiva, A., and Pietquin, O. Learning from a learner. In International Conference on Machine Learning , pp. 2990-2999, 2019.
  20. Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 , 2016. · arXiv
  21. Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P. A., Strouse, D., Leibo, J. Z., and de Freitas, N. Intrinsic social motivation via causal influence in multi-agent rl. arXiv preprint arXiv:1810.08647 , 2018. · arXiv
  22. Jiménez, Á. V. and Mesoudi, A. Prestige-biased social learning: current evidence and outstanding questions. Palgrave Communications , 5(1):1-12, 2019.
  23. Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2018.
  24. Ke, N. R., Singh, A., Touati, A., Goyal, A., Bengio, Y., Parikh, D., and Batra, D. Learning dynamics model in reinforcement learning by incorporating the long term future. arXiv preprint arXiv:1903.01599 , 2019. · arXiv
  25. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations , 12 2014.
  26. Krupnik, O., Mordatch, I., and Tamar, A. Multi-agent reinforcement learning with multi-step generative models. In Conference on Robot Learning , pp. 776-790, 2020.
  27. Laland, K. N. Social learning strategies. Animal Learning &Behavior , 32(1):4-14, February 2004.
  28. Laland, K. N. Darwin's unfinished symphony: How culture made the human mind . Princeton University Press, 2018.
  29. Landolfi, N. C. and Dragan, A. D. Social cohesion in autonomous driving. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 8118-8125. IEEE, 2018.
  30. Lerer, A. and Peysakhovich, A. Learning existing social conventions via observationally augmented self-play. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pp. 107-114, 2019.
  31. Ng, A. Y., Russell, S. J., et al. Algorithms for inverse reinforcement learning. In Icml , volume 1, pp. 2, 2000.
  32. Omidshafiei, S., Kim, D.-K., Liu, M., Tesauro, G., Riemer, M., Amato, C., Campbell, M., and How, J. P. Learning to teach in cooperative multiagent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 6128-6136, 2019.
  33. Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., and Song, D. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282 , 2018. · arXiv
  34. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d ' Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 8024-8035. Curran Associates, Inc., 2019.
  35. Pomerleau, D. A. Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems , pp. 305-313, 1989.
  36. Ramachandran, D. and Amir, E. Bayesian inverse reinforcement learning. In IJCAI , volume 7, pp. 2586-2591, 2007.
  37. Reddy, S., Dragan, A. D., and Levine, S. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11108 , 2019. · arXiv
  38. Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
  39. Schaal, S. Is imitation learning the route to humanoid robots? Trends in cognitive sciences , 3(6):233-242, 1999.
  40. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR) , 2016.
  41. Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307 , 2016. · arXiv
  42. Shon, A., Grochow, K., Hertzmann, A., and Rao, R. P. Learning shared latent structure for image synthesis and robotic imitation. In Advances in neural information processing systems , pp. 1233-1240, 2006.
  43. Stadie, B. C., Abbeel, P., and Sutskever, I. Third-person imitation learning. arXiv preprint arXiv:1703.01703 , 2017. · arXiv
  44. Sun, L., Zhan, W., Chan, C.-Y., and Tomizuka, M. Behavior planning of autonomous cars with social perception. In 2019 IEEE Intelligent Vehicles Symposium (IV) , pp. 207213. IEEE, 2019.
  45. Torabi, F., Warnell, G., and Stone, P. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954 , 2018. · arXiv
  46. Weber, T., Racanière, S., Reichert, D. P., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. Imagination-augmented agents for deep reinforcement learning. arXiv preprint arXiv:1707.06203 , 2017. · arXiv
  47. Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.

Cited by

Sorted by citations

Authors
Ndousse, K., Eck, D., Levine, S., & Jaques, N.
Affiliations
  • OpenAI (United States) (US)
  • University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
7991-8004
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.00581
arXiv
2010.00581
OpenAlex
W3171853008
Cited by
23
References
2
Citations by year
2025: 4 · 2024: 3 · 2023: 6 · 2022: 8 · 2021: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
social learning, multi-agent reinforcement learning, imitation learning, online adaptation, transfer learning, skill discovery, model-free reinforcement learning, human-robot interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review