CoChat
Sign Up
PreprintCited by 43Reinforcement Learning in RoboticsSocial Robot Interaction and HRISpeech and dialogue systems

Imitating Interactive Intelligence

Open access
Abramson, Josh, Ahuja, Arun, Barr, Iain, Brussee, Arthur, Carnevale, Federico, Cassin, Mary, Chhaparia, Rachita, Clark, Stephen +21 more·2020·arXiv
DOI 10.48550/arxiv.2012.05672arXiv 2012.05672
Open this paper in CoChat — free accountOpen PDF

Abstract

A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central challenges of artificial intelligence (AI) research: complex visual perception and goal-directed physical control, grounded language comprehension and production, and multi-agent social interaction. To build agents that can robustly interact with humans, we would ideally train them while they interact with humans. However, this is presently impractical. Therefore, we approximate the role of the human with another learned agent, and use ideas from inverse reinforcement learning to reduce the disparities between human-human and agent-agent interactive behaviour. Rigorously evaluating our agents poses a great challenge, so we develop a variety of behavioural tests, including evaluation by humans who watch videos of agents or interact directly with them. These evaluations convincingly demonstrate that interactive training and auxiliary losses improve agent behaviour beyond what is achieved by supervised learning of actions alone. Further, we demonstrate that agent capabilities generalise beyond literal experiences in the dataset. Finally, we train evaluation models whose ratings of agents agree well with human judgement, thus permitting the evaluation of new agent models without additional effort. Taken together, our results in this virtual environment provide evidence that large-scale human behavioural imitation is a promising tool to create intelligent, interactive agents, and the challenge of reliably evaluating such agents is possible to surmount.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.05672

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SIMA tries to solve the problem across many simulated environments in a general and scalable manner, assuming little beyond human-like interaction with those environments.

    “Thus, in contrast to prior works (e.g., Abramson et al., 2020;Vinyals et al., 2019;Berner et al., 2019;Lifshitz et al., 2023), we are attempting to tackle this problem across many simulated environments, in the most general and scalable way possible, by making few assumptions beyond interacting with the environments in the same way as humans do.”

    From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Abramson et al., 2020

  • Their data can augment existing training data or fine-tune pretrained models to give them more situated understanding.

    “Our data can be used to augment and leverage existing training data (e.g., Abramson et al., 2020), or to fine-tune pretrained models to endow them with more situated understanding.”

    From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Abramson et al., 2020

  • The agent architecture builds on prior related work (Abramson et al., 2020, 2022a), but with various changes and adaptations to more general goals.

    “Our agent architecture builds on prior related work (Abramson et al., 2020(Abramson et al., , 2022a)), but with various changes and adaptations to our more general goals.”

    From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Abramson et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. Adam, J. et al. (1902). The Republic of Plato . University Press. 3
  2. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., S¨ underhauf, N., Reid, I. D., Gould, S., and van den Hengel, A. (2017). Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. CoRR , abs/1711.07280. 37 · arXiv
  3. Aytar, Y., Pfaff, T., Budden, D., Paine, T., Wang, Z., and de Freitas, N. (2018). Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems , pages 2930-2941. 36
  4. Borsa, D., Piot, B., Munos, R., and Pietquin, O. (2017). Observational learning by reinforcement learning. arXiv preprint arXiv:1706.06617 . 36 · arXiv
  5. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . 5, 36 · arXiv
  6. Byrne, R. W. (2009). Animal imitation. Current Biology , 19(3):R111-R114. 38
  7. Byrne, R. W. and Russon, A. E. (1998). Learning by imitation: A hierarchical approach. Behavioral and brain sciences , 21(5):667-684. 38
  8. Cabi, S., G´ omez Colmenarejo, S., Novikov, A., Konyushkova, K., Reed, S., Jeong, R., Zolna, K., Aytar, Y., Budden, D., Vecerik, M., et al. (2019). Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv , pages arXiv-1909. 26, 55
  9. Card, S. K., Moran, T. P., and Newell, A. (1983). The psychology of human-computer interaction . CRC Press. 2
  10. Chollet, F. (2019). On the measure of intelligence. arXiv preprint arXiv:1911.01547 . 20 · arXiv
  11. Chomsky, N. (1959). Chomsky, n. 1959. a review of bf skinner's verbal behavior. language, 35 (1), 26-58. 38
  12. Chopra, S., Hadsell, R., and LeCun, Y. (2005). Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on , volume 1, pages 539-546. IEEE. 65
  13. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pages 4299-4307. 4
  14. Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 702-703. 16, 67
  15. Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., and Batra, D. (2018). Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 2054-2063. 37
  16. Dunbar, R. I. (1993). Coevolution of neocortical size, group size and language in humans. Behavioral and brain sciences , 16(4):681-694. 2, 20
  17. Duncan, J. (2010). How intelligence happens . Yale University Press. 20
  18. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 . 69 · arXiv
  19. Finn, C., Christiano, P., Abbeel, P., and Levine, S. (2016). A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 . 14, 67 · arXiv
  20. Galashov, A., Jayakumar, S. M., Hasenclever, L., Tirumala, D., Schwarz, J., Desjardins, G., Czarnecki, W. M., Teh, Y. W., Pascanu, R., and Heess, N. (2019). Information asymmetry in kl-regularized rl. arXiv preprint arXiv:1905.01240 . 5 · arXiv
  21. Ghasemipour, S. K. S., Zemel, R., and Gu, S. (2020). A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning , pages 1259-1277. PMLR. 15
  22. Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448. 13
  23. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680. 15
  24. Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 . 11 · arXiv
  25. Guo, M., Dai, Z., Vrandecic, D., and Al-Rfou, R. (2020). Wiki-40b: Multilingual language model dataset. In LREC 2020 . 25, 26
  26. Gutmann, M. and Hyv¨ arinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 297-304. 13, 65
  27. Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena , 42(1-3):335-346. 36
  28. He, K., Gkioxari, G., Doll´ ar, P., and Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969. 13
  29. H´ enaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S., and Oord, A. v. d. (2019). Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272 . 13, 66 · arXiv
  30. Hermann, K. M., Hill, F., Green, S., Wang, F., Faulkner, R., Soyer, H., Szepesvari, D., Czarnecki, W. M., Jaderberg, M., Teplyashin, D., et al. (2017). Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551 . 2 · arXiv
  31. Heyes, C. M. and Galef Jr, B. G. (1996). Social learning in animals: the roots of culture . Elsevier. 38
  32. Hu, H., Yarats, D., Gong, Q., Tian, Y., and Lewis, M. (2019). Hierarchical decision making by generating and following natural language instructions. In Advances in neural information processing systems , pages 10025-10034. 37
  33. Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017). In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture , pages 1-12. 69
  34. Kakade, S. M. et al. (2003). On the sample complexity of reinforcement learning . PhD thesis, University of London London, England. 4
  35. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . 64 · arXiv
  36. Lake, B. M. and Murphy, G. L. (2020). Word meaning in minds and machines. arXiv preprint arXiv:2008.01766 . 2, 36 · arXiv
  37. Laland, K. N. (2004). Social learning strategies. Animal Learning & Behavior , 32(1):4-14. 38
  38. Li, Y., Song, J., and Ermon, S. (2017). Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems , pages 3812-3822. 16
  39. Lin, J., Gan, C., and Han, S. (2019). Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision , pages 7083-7093. 74
  40. Lynch, C. and Sermanet, P. (2020). Grounding language in play. arXiv preprint arXiv:2005.07648 . 2, 37 · arXiv
  41. McClelland, J. L., Hill, F., Rudolph, M., Baldridge, J., and Sch¨ utze, H. (2019). Extending machine language models toward human-level language understanding. arXiv preprint arXiv:1912.05877 . 2, 36 · arXiv
  42. Merel, J., Tassa, Y., TB, D., Srinivasan, S., Lemmon, J., Wang, Z., Wayne, G., and Heess, N. (2017). Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 . 15, 36 · arXiv
  43. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. 69
  44. Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., and Peters, J. (2018). An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711 . 10 · arXiv
  45. Pomerleau, D. A. (1989). Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems , pages 305-313. 5, 10
  46. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9. 36, 74
  47. Ross, S., Gordon, G., and Bagnell, D. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. 14
  48. Roy, D., Patel, R., DeCamp, P., Kubat, R., Fleischman, M., Roy, B., Mavridis, N., Tellex, S., Salata, A., Guinness, J., et al. (2006). The human speechome project. In International Workshop on Emergence and Evolution of Linguistic Communication , pages 192-196. Springer. 36
  49. Schaal, S. (1999). Is imitation learning the route to humanoid robots? Trends in cognitive sciences , 3(6):233-242. 5
  50. Shannon, C. E. (1951). Prediction and entropy of printed English. Bell system technical journal , 30(1):50-64. 5
  51. Shaw, P., Uszkoreit, J., and Vaswani, A. (2018). Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 . 61, 69 · arXiv
  52. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489. 5
  53. Stadie, B. C., Abbeel, P., and Sutskever, I. (2017). Third-person imitation learning. arXiv preprint arXiv:1703.01703 . 36 · arXiv
  54. Sullivan, J., Mei, M., Perfors, A., Wojcik, E. H., and Frank, M. C. (2020). Saycam: A large, longitudinal audiovisual dataset recorded from the infant's perspective. PsyArXiv . 36
  55. Tellex, S. A., Kollar, T. F., Dickerson, S. R., Walter, M. R., Banerjee, A., Teller, S., and Roy, N. (2011). Understanding natural language commands for robotic navigation and mobile manipulation. AAAI Publications . 37
  56. Tomasello, M. (2010). Origins of human communication . MIT press. 2
  57. van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 . 13, 66 · arXiv
  58. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems , pages 5998-6008. 10, 61, 69
  59. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354. 5, 37
  60. Ward, T., Bolt, A., Hemmings, N., Carter, S., Sanchez, M., Barreira, R., Noury, S., Anderson, K., Lemmon, J., Coe, J., et al. (2020). Using unity to help solve intelligence. arXiv preprint arXiv:2011.09294 . 2, 47 · arXiv
  61. Winograd, T. (1972). Understanding natural language. Cognitive psychology , 3(1):1-191. 2, 36
  62. Winograd, T. (2006). Shifting viewpoints: Artificial intelligence and human-computer interaction. Artificial intelligence , 170(18):1256-1258. 36
  63. Yoshida, H. and Smith, L. B. (2008). What's in view for toddlers? using a head camera to study visual experience. Infancy , 13(3):229-248. 36
  64. Ziebart, B. D. (2010). Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Thesis for Carnegie Mellon University . 14, 67
  65. Zolna, K., Reed, S., Novikov, A., Colmenarej, S. G., Budden, D., Cabi, S., Denil, M., de Freitas, N., and Wang, Z. (2019). Task-relevant adversarial imitation learning. arXiv preprint arXiv:1910.01077 . 16 · arXiv

Cited by

Sorted by citations

Authors
Abramson, J., Ahuja, A., Barr, I., Brussee, A., Carnevale, F., Cassin, M., Chhaparia, R., Clark, S., Damoc, B., Dudzik, A., Georgiev, P., Guy, A., Harley, T., Hill, F., Hung, A., Kenton, Z., Landon, J., Lillicrap, T., Mathewson, K., Mokrá, S., Muldal, A., Santoro, A., Savinov, N., Varma, V., Wayne, G., Williams, D., Wong, N., Yan, C., & Zhu, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2012.05672
arXiv
2012.05672
OpenAlex
W3113184484
Cited by
43
References
6
Citations by year
2025: 1 · 2024: 2 · 2023: 7 · 2022: 18 · 2021: 15
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
interactive training, inverse reinforcement learning, human-robot interaction, virtual worlds, agent evaluation, auxiliary loss, visual perception, physical control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review