CoChat
Sign Up
PreprintCited by 64Reinforcement Learning in RoboticsPsychiatry, Mental Health, NeuroscienceAI-based Problem Solving and Planning

Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

Open access
Colas, Cédric, Karch, Tristan, Lair, Nicolas, Dussoux, Jean-Michel, Moulin-Frier, Clément, Dominey, Peter Ford, Oudeyer, Pierre-Yves·2020·arXiv
DOI 10.48550/arxiv.2002.09253arXiv 2002.09253
Open this paper in CoChat — free accountOpen PDF

Abstract

Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2002.09253

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Language abstractions can support efficient learning and generalization.

    “Language abstractions can enable efficient learning and generalization (Hill et al., 2020;Colas et al., 2020;Lampinen et al., 2022;Tam et al., 2022;Hu and Clune, 2023).”

    From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Colas et al., 2020

  • Language abstractions can speed up grounded learning, for instance by improving state abstractions for novelty-based exploration or by composing known goals into new ones.

    “Language abstractions can accelerate grounded learning, for example accelerating novelty-based exploration in reinforcement learning by providing better state abstractions (Tam et al., 2022;Mu et al., 2022), or composing known goals into new ones (Colas et al., 2020;Nottingham et al., 2023).”

    From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Colas et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 67

  1. [1][1] Jacob Andreas. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7556-7566, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.676. URL https://www.aclweb.org/anthology/2020.acl-main.676 . - · DOI
  2. [2][2] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems , pages 5048-5058, 2017. -
  3. [3][3] Minoru Asada, Koh Hosoda, Yasuo Kuniyoshi, Hiroshi Ishiguro, Toshio Inui, Yuichiro Yoshikawa, Masaki Ogino, and Chisato Yoshida. Cognitive developmental robotics: A survey. IEEE transactions on autonomous mental development , 1(1):12-34, 2009. -
  4. [4][4] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning to Understand Goal Specifications by Modelling Reward. In International Conference on Learning Representations , jun 2019. -
  5. [5][5] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In ICLR , 2019. -
  6. [6][6] Gianluca Baldassarre and Marco Mirolli. Intrinsically motivated learning in natural and artificial systems . Springer, 2013. -
  7. [7][7] Adrien Baranes and Pierre-Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems , 61(1):49-73, 2013. -
  8. [8][8] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems , pages 1471-1479, 2016. -
  9. [9][9] Marc H Bornstein, Catherine S Tamis-LeMonda, Joseph Tal, Pamela Ludemann, Sueko Toda, Charles W Rahn, Marie-Germaine Pêcheux, Hiroshi Azuma, and Danya Vardi. Maternal responsiveness to infants in three societies: The united states, france, and japan. Child development , 63(4):808-821, 1992. -
  10. [10][10] Jerome Bruner. The Narrative Construction of Reality. Critical Inquiry , 18(1):1-21, oct 1991. ISSN 0093-1896. doi: 10.1086/448619. - · DOI · Paper page
  11. [11][11] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390 , 2019. - · arXiv
  12. [12][12] Angelo Cangelosi and Matthew Schlesinger. Developmental robotics: From babies to robots . MIT press, 2015. -
  13. [15][15] David L. Chen and Raymond J. Mooney. Learning to Interpret Natural Language Navigation Instructions from Observations. In AAAI Conference on Artificial Intelligence (AAAI), 2011 , 2011. -
  14. [16][16] Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems , pages 1281-1288, 2005. -
  15. [17][17] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby{AI}: First Steps Towards Grounded Language Learning With a Human In the Loop. In International Conference on Learning Representations , 2019. -
  16. [20][20] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Self-educated language agent with hindsight experience replay for instruction following. arXiv preprint arXiv:1910.09451 , 2019. - · arXiv
  17. [22][22] Cédric Colas, Pierre-Yves Oudeyer, Olivier Sigaud, Pierre Fournier, and Mohamed Chetouani. CURIOUS: intrinsically motivated modular multi-goal reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , pages 1331-1340, 2019. -
  18. [23][23] Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. arXiv preprint arXiv:1904.06979 , 2019. - · arXiv
  19. [24][24] Peter Ford Dominey. Emergence of grammatical constructions: evidence from simulation and grounded agent experiments. Connection Science , 17(3-4):289-306, sep 2005. ISSN 0954-0091. doi: 10.1080/09540090500270714. - · DOI
  20. [25][25] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019. - · arXiv
  21. [26][26] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. ICML , 2018. -
  22. [27][27] Sébastien Forestier and Pierre-Yves Oudeyer. Modular active curiosity-driven discovery of tool use. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on , pages 3965-3972. IEEE, 2016. -
  23. [28][28] Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. CoRR , abs/1708.02190, 2017. URL http://arxiv.org/abs/1708.02190 . - · arXiv
  24. [29][29] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following. In International Conference on Learning Representations , 2019. -
  25. [30][30] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900 , 2018. - · arXiv
  26. [31][31] Arthur M. Glenberg and Michael P. Kaschak. Grounding language in action. Psychonomic Bulletin & Review , 9(3):558-565, sep 2002. ISSN 1069-9384. doi: 10.3758/BF03196313. - · DOI · Paper page
  27. [32][32] Adele E Goldberg. Constructions: A new theoretical approach to language. Trends in cognitive sciences , 7(5):219-224, 2003. -
  28. [34][34] Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. Using Natural Language for Reward Shaping in Reinforcement Learning. In IJCAI 2019 , mar 2019. URL http://arxiv.org/ abs/1903.02020 . - · arXiv
  29. [35][35] Edwin James Green and Jake Quilty-Dunn. What is an object file? The British Journal for the Philosophy of Science , 2017. -
  30. [36][36] Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.00450 , 2019. - · arXiv
  31. [37][37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015. -
  32. [38][38] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded Language Learning in a Simulated 3D World. jun 2017. URL http://arxiv.org/abs/1706.06551 . - · arXiv
  33. [40][40] Xavier Hinaut and Peter Ford Dominey. Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network simulation study using reservoir computing. PloS one , 8(2), 2013. -
  34. [41][41] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. , 9 (8):1735-1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735 . - · DOI · Paper page
  35. [42][42] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an Abstraction for Hierarchical Deep Reinforcement Learning. In Workshop on 'Structure & Priors in Reinforcement Learning'at ICLR 2019 , jun 2019. URL http://arxiv.org/abs/1906. 07343 . -
  36. [43][43] Scott P Johnson, Dima Amso, and Jonathan A Slemmer. Development of object concepts in infancy: Evidence for early learning in an eye-tracking paradigm. Proceedings of the National Academy of Sciences , 100(18):10568-10573, 2003. -
  37. [49][49] Adrien Laversanne-Finot, Alexandre Pere, and Pierre-Yves Oudeyer. Curiosity driven exploration of learned disentangled goal spaces. volume 87 of Proceedings of Machine Learning Research , pages 487-504. PMLR, 29-31 Oct 2018. URL http://proceedings.mlr.press/ v87/laversanne-finot18a.html . -
  38. [50][50] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015. - · arXiv
  39. [51][51] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A Survey of Reinforcement Learning Informed by Natural Language. IJCAI'19 , jun 2019. URL http://arxiv.org/abs/1906. 03926 . -
  40. [52][52] Carol Madden, Michel Hoen, and Peter Ford Dominey. A cognitive neuroscience perspective on embodied language for human-robot cooperation. Brain and Language , 112(3):180-188, mar 2010. ISSN 0093-934X. doi: 10.1016/J.BANDL.2009.07.001. - · DOI
  41. [53][53] Daniel J. Mankowitz, Augustin Zídek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, and Tom Schaul. Unicorn: Continual learning with a universal, off-policy agent. CoRR , abs/1802.08294, 2018. URL http://arxiv.org/ abs/1802.08294 . - · arXiv
  42. [54][54] James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. Extending machine language models toward human-level language understanding. arXiv preprint arXiv:1912.05877 , 2019. - · arXiv
  43. [55][55] Toben H Mintz. Frequent frames as a cue for grammatical categories in child directed speech. Cognition , 90(1):91-117, 2003. -
  44. [56][56] Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. arXiv preprint arXiv:1910.11670 , 2019. - · arXiv
  45. [57][57] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems , pages 9191-9200, 2018. -
  46. [58][58] Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 12527-12537, 2019. -
  47. [59][59] Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation , 11(2): 265-286, 2007. -
  48. [60][60] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 16-17, 2017. -
  49. [61][61] Jean Piaget. The language and thought of the child . Routledge, 1926. ISBN 0415267501. -
  50. [62][62] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464 , 2018. - · arXiv
  51. [63][63] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698 , 2019. - · arXiv
  52. [64][64] S R. K. Branavan, Luke S. Zettlemoyer, and Regina Barzilay. Reading Between the Lines: Learning to Map High-level Instructions to Commands. In ACL 2010 - 48th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference , pages 12681277, 2010. -
  53. [65][65] Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv:1909.12892 , 2019. - · arXiv
  54. [66][66] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019. -
  55. [67][67] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning , pages 1312-1320, 2015. -
  56. [68][68] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development , 2(3):230-247, 2010. -
  57. [69][69] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10740-10749, 2020. -
  58. [70][70] Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge. Psychological review , 99(4):605, 1992. -
  59. [71][71] Luc Steels. Semiotic dynamics for embodied agents. IEEE Intelligent Systems , 21(3):32-38, 2006. -
  60. [72][72] Michael Tomasello. The cultural origins of human cognition . Harvard University Press, 1999. ISBN 9780674005822. -
  61. [73][73] Michael Tomasello. The item-based nature of children's early syntactic development. Trends in cognitive sciences , 4(4):156-163, 2000. -
  62. [74][74] Michael Tomasello. Constructing a language . Harvard university press, 2009. -
  63. [75][75] Michael Tomasello and Raquel Olguin. Twenty-three-month-old children have a grammatical category of noun. Cognitive development , 8(4):451-464, 1993. -
  64. [76][76] Srinivas Venkattaramanujam, Eric Crawford, Thang Doan, and Doina Precup. Self-supervised learning of distance functions for goal-conditioned reinforcement learning. arXiv preprint arXiv:1907.02998 , 2019. - · arXiv
  65. [77][77] L. S. Vygotsky. Tool and Symbol in Child Development. In Mind in Society , chapter Tool and Symbol in Child Development, pages 19-30. Harvard University Press, 1978. ISBN 0674576292. doi: 10.2307/j.ctvjf9vz4.6. - · DOI
  66. [78][78] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems , pages 3391-3401, 2017. -
  67. [80][80] Rolf Zwaan and Carol Madden. Embodied sentence comprehension. Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking , pages 224-245, 2005. doi: 10.1017/CBO9780511499968.010. · DOI

Cited by

Sorted by citations

Authors
Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin-Frier, C., Dominey, P. F., & Oudeyer, P.-Y.
Affiliations
  • Inserm (FR)
  • Université de Bordeaux (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2002.09253
arXiv
2002.09253
OpenAlex
W3007295596
Cited by
64
References
3
Citations by year
2026: 1 · 2025: 3 · 2024: 8 · 2023: 22 · 2022: 11 · 2021: 13 · 2020: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
curiosity-driven exploration, deep reinforcement learning, intrinsic motivation, modularity, social interaction, open-ended learning, Deep Sets, gated attention, object-centered representation, children's play

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review