CoChat
Sign Up
PreprintCited by 3Speech and dialogue systemsDomain Adaptation and Few-Shot LearningReinforcement Learning in Robotics

Intra-agent speech permits zero-shot task acquisition

Open access
Yan, Chen, Carnevale, Federico, Georgiev, Petko, Santoro, Adam, Guy, Aurelia, Muldal, Alistair, Hung, Chia-Chun, Abramson, Josh +2 more·2022·arXiv
DOI 10.48550/arxiv.2206.03139arXiv 2206.03139
Open this paper in CoChat — free accountOpen PDF

Abstract

Human language learners are exposed to a trickle of informative, context-sensitive language, but a flood of raw sensory data. Through both social language use and internal processes of rehearsal and practice, language learners are able to build high-level, semantic representations that explain their perceptions. Here, we take inspiration from such processes of "inner speech" in humans (Vygotsky, 1934) to better understand the role of intra-agent speech in embodied behavior. First, we formally pose intra-agent speech as a semi-supervised problem and develop two algorithms that enable visually grounded captioning with little labeled language data. We then experimentally compute scaling curves over different amounts of labeled data and compare the data efficiency against a supervised learning baseline. Finally, we incorporate intra-agent speech into an embodied, mobile manipulator agent operating in a 3D virtual world, and show that with as few as 150 additional image captions, intra-agent speech endows the agent with the ability to manipulate and answer questions about a new object without any related task-directed experience (zero-shot). Taken together, our experiments suggest that modelling intra-agent speech is effective in enabling embodied agents to learn new tasks efficiently and without direct interaction experience.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.03139

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Intra-agent speech in an agent exploring a 3D virtual world enabled the agent to manipulate and answer questions about a novel object without any direct experience of it.

    “Intra-agent speech was modeled in an agent exploring a 3D virtual world, showing that the agent was able to manipulate and answer questions about a novel object without any direct experience of it [79].”

    From Inner speech as language process and cognitive tool (2023) · cites this paper as [79]

    The Abstract reports that an agent in a 3D virtual world could manipulate and answer questions about a new object using intra-agent speech. But it specifies no related task-directed experience, not no direct experience of the object: the agent received image captions about it. Figure 1 in the Introduction likewise describes labeled captions for the new object category. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Modeling inner speech in an artificial agent has been shown to enhance learning from external information.

    “Modeling IS in an artificial agent has also been shown to enhance learning from external information.”

    From Inner speech as language process and cognitive tool (2023) · cites this paper as [79]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 61

  1. [1][1] Lev S Vygotsky. Thought and language. MIT press, 1934.
  2. [2][2] Adam Ed Winsler, Charles Ed Fernyhough, and Ignacio Ed Montero. Private speech, executive functioning, and the development of verbal self-regulation. Cambridge University Press, 2009.
  3. [3][3] Patti Adank, Peter Hagoort, and Harold Bekkering. Imitation improves language comprehension. Psychological Science, 21(12):1903–1909, 2010.
  4. [4][4] Conchi San Martín Martínez, Humbert Boada i Calbet, and Peter Feigenbaum. Private and inner speech and the regulation of social speech communication. Cognitive Development, 26(3): 214–229, 2011.
  5. [5][5] Alan D Baddeley, Susan E Gathercole, and Costanza Papagno. The phonological loop as a language learning device. Exploring Working Memory, pages 164–198, 2017.
  6. [6][6] Adam Winsler, Rafael M Diaz, and Ignacio Montero. The role of private speech in the transition from collaborative to independent task performance in young children. Early Childhood Research Quarterly, 12(1):59–79, 1997.
  7. [7][7] Ben Alderson-Day and Charles Fernyhough. Inner speech: development, cognitive functions, phenomenology, and neurobiology. Psychological bulletin, 141(5):931, 2015.
  8. [8][8] Alan D Baddeley. Short-term memory for word sequences as a function of acoustic, semantic and formal similarity. Quarterly journal of experimental psychology, 18(4):362–365, 1966.
  9. [9][9] Diane J Schiano and Michael J Watkins. Speech-like coding of pictures in short-term memory. Memory & Cognition, 9(1):110–114, 1981.
  10. [10][10] Antonis Hatzigeorgiadis, Nikos Zourbanos, Evangelos Galanis, and Yiannis Theodorakis. Self- talk and sports performance: A meta-analysis. Perspectives on Psychological Science, 6(4): 348–356, 2011.
  11. [11][11] Virginia S Tinsley and Harriet Salatas Waters. The development of verbal control over motor behavior: A replication and extension of luria's findings. Child Development, pages 746–753, 1982.
  12. [12][12] Charles Fernyhough and Emma Fradley. Private speech on an executive task: Relations with task difficulty and task performance. Cognitive development, 20(1):103–120, 2005.
  13. [13][13] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
  14. [14][14] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017. · arXiv
  15. [15][15] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018. · arXiv
  16. [16][16] H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. arXiv preprint arXiv:1909.12238, 2019. · arXiv
  17. [17][17] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv–1807, 2018.
  18. [18][18] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29, 2016.
  19. [19][19] Chris Dyer. Notes on noise contrastive estimation and negative sampling. arXiv preprint arXiv:1410.8251, 2014. 11 · arXiv
  20. [20][20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  21. [21][21] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  22. [22][22] DeepMind Interactive Agents Team. Creating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint arXiv:2112.03763, 2021. · arXiv
  23. [23][23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  24. [24][24] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. · arXiv
  25. [25][25] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
  26. [26][26] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  27. [27][27] Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pages 1–2. IEEE, 2018.
  28. [29][29] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25–37, 2020.
  29. [30][30] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  30. [32][32] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
  31. [33][33] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017. · arXiv
  32. [34][34] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272, 2018. · arXiv
  33. [35][35] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113–1132. PMLR, 2020.
  34. [36][36] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382, 2020. 12 https://cloud.google.com/tpu/docs/tpus https://cloud.google.com/tpu/docs/tpus · arXiv
  35. [37][37] Ziad Al-Halah, Santhosh K Ramakrishnan, and Kristen Grauman. Zero experience re- quired: Plug & play modular transfer learning for semantic visual navigation. arXiv preprint arXiv:2202.02440, 2022. · arXiv
  36. [38][38] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303–1315. PMLR, 2022.
  37. [39][39] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020. · arXiv
  38. [40][40] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022.
  39. [41][41] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050, 2021. · arXiv
  40. [42][42] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. Advances in neural information processing systems, 33:19861–19872, 2020.
  41. [43][43] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pages 357–368. PMLR, 2017.
  42. [44][44] Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
  43. [45][45] Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. arXiv preprint arXiv:1906.03352, 2019. · arXiv
  44. [46][46] Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hi- erarchical few-shot imitation with skill transition models. arXiv preprint arXiv:2107.08981, 2021. · arXiv
  45. [47][47] Yishu Miao and Phil Blunsom. Language as a latent variable: Discrete generative models for sentence compression. arXiv preprint arXiv:1609.07317, 2016. · arXiv
  46. [48][48] Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. arXiv preprint arXiv:1909.04499, 2019. · arXiv
  47. [49][49] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. arXiv preprint arXiv:2005.07064, 2020. · arXiv
  48. [50][50] Wenhu Chen, Aurelien Lucchi, and Thomas Hofmann. A semi-supervised framework for image captioning. arXiv preprint arXiv:1611.05321, 2016. · arXiv
  49. [51][51] Bicheng Xu, Weirui Kong, and Jiaxuan Chen. Semi-supervised image captioning via recon- struction. In Proceedings of the Intern. Conf. on Comp. Vis.(ICCV), pages 4135–4144, 2017.
  50. [52][52] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. Image captioning with very scarce supervised data: Adversarial semi-supervised learning approach. arXiv preprint arXiv:1909.02201, 2019. · arXiv
  51. [53][53] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  52. [54][54] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015. 13
  53. [55][55] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  54. [56][56] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  55. [57][57] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. · arXiv
  56. [58][58] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021. · arXiv
  57. [59][59] Tiago Ramalho, Tomáš Kočiskỳ, Frederic Besse, SM Eslami, Gábor Melis, Fabio Viola, Phil Blunsom, and Karl Moritz Hermann. Encoding spatial relations from natural language. arXiv preprint arXiv:1807.01670, 2018. · arXiv
  58. [60][60] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pages 1060–1069. PMLR, 2016.
  59. [61][61] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  60. [62][62] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802–5810, 2019.
  61. [63][63] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 833–842, 2021.

Cited by

Sorted by citations

Related

Authors
Yan, C., Carnevale, F., Georgiev, P., Santoro, A., Guy, A., Muldal, A., Hung, C.-C., Abramson, J., Lillicrap, T., & Wayne, G.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
2423-2438
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2206.03139
arXiv
2206.03139
OpenAlex
W7133241669
Cited by
3
References
0
Citations by year
2023: 1 · 2022: 2
FWCI
0.41
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
chatbots, human-agent interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review