CoChat
Sign Up
Cited by 235

First return, then explore

Ecoffet, Adrien, Huizinga, Joost, Lehman, Joel, Stanley, Kenneth O., Clune, Jeff·2021·Nature
DOI 10.1038/s41586-020-03157-9
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41586-020-03157-9

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Goal discovery is a Go-Explore-style novelty search serving WorldCoder's optimism constraint: propose a reward believed to be reachable, then use it to direct exploration.

    “Goal discovery is a Go-Explore-style novelty search (Ecoffet et al. 2021) serving WorldCoder's optimism constraint (Tang, Key, and Ellis 2024): propose a reward believed to be reachable, then use it to direct exploration.”

    From Twin: Playing an Unknown Game with a Test-Time Digital Twin (2026) · cites this paper as Ecoffet et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Bellemare, M. et al. Unifying count-based exploration and intrinsic motivation in NIPS (2016), 1471-1479.
  2. OpenAI et al. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv preprint arXiv:1912.06680 (2019). · arXiv
  3. Merel, J. et al. Hierarchical visuomotor control of humanoids. arXiv preprint arXiv:1811.09656 (2018). · arXiv
  4. OpenAI et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research 39, 3-20 (2020).
  5. Lehman, J. et al. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolution- ary Computation and Artificial Life Research Communities. CoRR abs/1803.03453 (2018). · arXiv
  6. Amodei, D. et al. Concrete Problems in AI Safety. CoRR abs/1606.06565 (2016). · arXiv
  7. Conti, E. et al. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents in Advances in neural information processing systems (2018), 5027-5038.
  8. Puigdomènech Badia, A. et al. Agent57: Outperforming the Atari Human Benchmark. arXiv preprint arXiv:2003.13350 (2020). · arXiv
  9. Aytar, Y. et al. Playing hard exploration games by watching YouTube. arXiv preprint arXiv:1805.11592 (2018). · arXiv
  10. Mnih, V. et al. Asynchronous methods for deep reinforcement learning in International conference on machine learning (2016), 1928-1937.
  11. Mouret, J.-B. & Clune, J. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 (2015). · arXiv
  12. Lehman, J. & Stanley, K. O. Evolving a diversity of virtual creatures through novelty search and local com- petition in GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computation (2011), 211-218.
  13. Tan, J. et al. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332 (2018). · arXiv
  14. Hester, T. et al. Deep Q-learning From Demonstrations in AAAI (2018).
  15. Horgan, D. et al. Distributed Prioritized Experience Replay. CoRR abs/1803.00933 (2018). · arXiv
  16. Espeholt, L. et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architec- tures in ICML (2018).
  17. Salimans, T. & Chen, R. Learning Montezuma's Revenge from a Single Demonstration. arXiv preprint arXiv:1812.03381 (2018). · arXiv
  18. Puigdomènech Badia, A. et al. Never Give Up: Learning Directed Exploration Strategies. arXiv (2020).
  19. Such, F. P. et al. Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567 (2017). · arXiv
  20. Brockman, G. et al. OpenAI gym. arXiv preprint arXiv:1606.01540 (2016). · arXiv
  21. Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W. & Abbeel, P. Overcoming exploration in reinforce- ment learning with demonstrations in 2018 IEEE International Conference on Robotics and Automation (ICRA) (2018), 6292-6299.
  22. Kraft, D. et al. Development of object and grasping knowledge by robot exploration. IEEE Transactions on Autonomous Mental Development 2, 368-383 (2010).
  23. Andrychowicz, M. et al. Hindsight experience replay in Advances in Neural Information Processing Systems (2017), 5048-5058.
  24. Eysenbach, B., Salakhutdinov, R. R. & Levine, S. Search on the replay buffer: Bridging planning and reinforce- ment learning in Advances in Neural Information Processing Systems (2019), 15220-15231.
  25. Oh, J., Guo, Y., Singh, S. & Lee, H. Self-Imitation Learning in ICML (2018).
  26. Guo, Y. et al. Efficient Exploration with Self-Imitation Learning via Trajectory-Conditioned Policy. arXiv preprint arXiv:1907.10247 (2019). · arXiv
  27. Liu, H., Simonyan, K. & Yang, Y. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018). · arXiv
  28. Madotto, A. et al. Exploration Based Language Learning for Text-Based Games. arXiv preprint arXiv:2001.08868 (2020). · arXiv
  29. Zhang, W. & Dietterich, T. G. A reinforcement learning approach to job-shop scheduling in IJCAI 95 (1995), 1114-1120.
  30. Toromanoff, M., Wirbel, E. & Moutarde, F. Is Deep Reinforcement Learning Really Superhuman on Atari? arXiv preprint arXiv:1908.04683 (2019). · arXiv
  31. Burda, Y., Edwards, H., Storkey, A. & Klimov, O. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 (2018). · arXiv
  32. Choi, J. et al. Contingency-Aware Exploration in Reinforcement Learning. CoRR abs/1811.01483 (2018). · arXiv
  33. Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G. & Larochelle, H. Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865 (2019). · arXiv
  34. Taiga, A. A., Fedus, W., Machado, M. C., Courville, A. & Bellemare, M. G. On Bonus Based Exploration Methods In The Arcade Learning Environment in International Conference on Learning Representations (2020).
  35. Tang, Y., Valko, M. & Munos, R. Taylor expansion policy optimization. arXiv preprint arXiv:2003.06259 (2020). · arXiv
  36. Todorov, E., Erez, T. & Tassa, Y. MuJoCo: A physics engine for model-based control in IROS (2012), 5026-5033.
  37. Strehl, A. L. & Littman, M. L. An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences 74, 1309-1331 (2008).
  38. Tang, H. et al. # Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning in NIPS (2017), 2750-2759.
  39. Cho, K., Van Merriënboer, B., Bahdanau, D. & Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014). References · arXiv
  40. Nguyen, A., Yosinski, J. & Clune, J. Deep neural networks are easily fooled: High confidence predictions for un- recognizable images in Proceedings of the IEEE conference on computer vision and pattern recognition (2015), 427-436.
  41. Szegedy, C. et al. Intriguing properties of neural networks. ArXiv e-prints abs/1312.6199 (2013). · arXiv
  42. Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O. & Clune, J. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 (2019). · arXiv
  43. O'Donoghue, B., Munos, R., Kavukcuoglu, K. & Mnih, V. Combining policy gradient and Q-learning. arXiv preprint arXiv:1611.01626 (2016). · arXiv

Cited by

Sorted by citations

Authors
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J.
ORCID
Venue
Nature
Publisher
Springer Science and Business Media LLC
Published
2021-02-24
Volume
590
Issue
7847
Pages
580-586
ISSN
  • 0028-0836
  • 1476-4687
DOI
10.1038/s41586-020-03157-9
Cited by
235
References
5
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review