CoChat
Sign Up
PreprintCited by 3Reinforcement Learning in RoboticsAI-based Problem Solving and PlanningDomain Adaptation and Few-Shot Learning

Training a Generally Curious Agent

Open access
Tajwar, Fahim, Jiang, Yiding, Thankaraj, Abitha, Rahman, Sumaita Sadia, Kolter, J Zico, Schneider, Jeff, Salakhutdinov, Ruslan·2025·arXiv
DOI 10.48550/arxiv.2502.17543arXiv 2502.17543
Open this paper in CoChat — free accountOpen PDF

Abstract

Efficient exploration is essential for intelligent systems interacting with their environment, but existing language models often fall short in scenarios that require strategic information gathering. In this paper, we present Paprika, a fine-tuning approach that enables language models to develop general decision-making capabilities that are not confined to particular environments. By training on synthetic interaction data from different tasks that require diverse strategies, Paprika teaches models to explore and adapt their behavior on a new task based on environment feedback in-context without more gradient updates. Experimental results show that models fine-tuned with Paprika can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike traditional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems that require interactions with the external world.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.17543

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In their Llama-3.1-8B-Instruct implementation, Tajwar et al. selected min-p sampling with temperature 1.5 and min-p parameter 0.3 after observing that this configuration consistently generated diverse training data while maintaining higher real-time accuracy compared to other sampling methods.

    “In their Llama-3.1-8B-Instructmodel implementation, they selected min-p sampling with temperature 1.5 and min-p parameter 0.3 after observing that this configuration consistently generated diverse training data while maintaining higher real-time accuracy compared to other sampling methods.”

    From Turning up the heat: Min-p sampling for creative and coherent llm outputs (2024) · cites this paper as Tajwar et al., 2025

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Tajwar et al. implemented min-p sampling with temperature 1.5 and min-p parameter 0.3 in their Llama-3.1-8B-Instruct model, finding that this configuration consistently produced more diverse trajectory data while maintaining higher real-time accuracy compared to alternative sampling methods.

    “Specifically, they implemented min-p sampling with temperature 1.5 and min-p parameter 0.3 in their Llama-3.1-8B-Instructmodel, finding that this configuration consistently produced more diverse trajectory data while maintaining higher real-time accuracy compared to alternative sampling methods.”

    From Turning up the heat: Min-p sampling for creative and coherent llm outputs (2024) · cites this paper as Tajwar et al., 2025

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Recent work in reinforcement learning has begun adopting min-p for trajectory data generation, specifically citing its benefits for generating high-quality and diverse training data.

    “Recent work in reinforcement learning has begun adopting min-p for trajectory data generation (Tajwar et al., 2025), specifically citing its benefits for generating high-quality and diverse training data.”

    From Turning up the heat: Min-p sampling for creative and coherent llm outputs (2024) · cites this paper as Tajwar et al., 2025

  • Recent research in reinforcement learning has begun utilizing min-p sampling for generating high-quality and diverse training data for curious agents.

    “Recent research in reinforcement learning has begun utilizing min-p sampling for generating high-quality and diverse training data for curious agents (Tajwar et al., 2025).”

    From Turning up the heat: Min-p sampling for creative and coherent llm outputs (2024) · cites this paper as Tajwar et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 104

  1. Abdulhai, M., White, I., Snell, C., Sun, C., Hong, J., Zhai, Y., Xu, K., and Levine, S. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:2311.18232 , 2023. · arXiv
  2. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.
  3. Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search, 2017. URL https://arxiv.org/abs/1705.08439 . · arXiv
  4. Anthony, T., Nishihara, R., Moritz, P., Salimans, T., and Schulman, J. Policy gradient search: Online planning and expert iteration without search trees, 2019. URL https://arxiv.org/abs/1904.03646 . · arXiv
  5. Audibert, J.-Y. and Bubeck, S. Best Arm Identification in Multi-Armed Bandits. In COLT 2010 - Proceedings , pp. 13 p., Haifa, Israel, June 2010. URL https://enpc. hal.science/hal-00654404 .
  6. Auer, P. Using upper confidence bounds for online learning. In Proceedings 41st annual symposium on foundations of computer science , pp. 270-279. IEEE, 2000.
  7. Beck, J., Vuorio, R., Liu, E. Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 , 2023. · arXiv
  8. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pp. 41-48, 2009.
  9. Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. L., and Towns, J. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good , PEARC '23, pp. 173-176, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399852. doi: 10.1145/ 3569951.3597559. URL https://doi.org/10. 1145/3569951.3597559 .
  10. Brown, S. T., Buitrago, P., Hanna, E., Sanielevici, S., Scibek, R., and Nystrom, N. A. Bridges-2: A platform for rapidly-evolving and data intensive research. In Practice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions , PEARC '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450382922. doi: 10. 1145/3437359.3465593. URL https://doi.org/ 10.1145/3437359.3465593 . · DOI
  11. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems , 33: 1877-1901, 2020.
  12. Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning , 2024.
  13. Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018. · arXiv
  14. Chen, J., Qadri, R., Wen, Y., Jain, N., Kirchenbauer, J., Zhou, T., and Goldstein, T. Genqa: Generating millions of instructions from a handful of prompts, 2024. URL https://arxiv.org/abs/2406.10323 . · arXiv
  15. Chen, R. Y., Sidor, S., Abbeel, P., and Schulman, J. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502 , 2017. · arXiv
  16. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.
  17. Cook, M. et al. Universality in elementary cellular automata. Complex systems , 15(1):1-40, 2004.
  18. Cˆ ot´ e, M.-A., ´ Akos K´ ad´ ar, Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Tao, R. Y ., Hausknecht, M., Asri, L. E., Adada, M., Tay, W., and Trischler, A. Textworld: Alearning environment for text-based games, 2019. URL https://arxiv.org/abs/1806.11532 . · arXiv
  19. Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR) , 2024.
  20. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R´ e, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.
  21. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https:// arxiv.org/abs/2501.12948 . · arXiv
  22. Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, S. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems , 33: 13049-13061, 2020.
  23. Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https: //openreview.net/forum?id=m7p5O7zblY .
  24. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 , 2024. · arXiv
  25. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization, 2024. URL https://arxiv.org/abs/ 2402.01306 .
  26. Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018. · arXiv
  27. Fang, M., Zhou, T., Du, Y., Han, L., and Zhang, Z. Curriculum-guided hindsight experience replay. Advances in neural information processing systems , 32, 2019.
  28. Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. Reverse curriculum generation for reinforcement learning. In Conference on robot learning , pp. 482495. PMLR, 2017.
  29. Foster, T. and Foerster, J. Learning to reason at the frontier of learnability, 2025. URL https://arxiv.org/ abs/2502.12272 . · arXiv
  30. Gemma-Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram´ e, A., Rivi` ere, M., Rouillard, L., Mesnard, T., et al. Gemma 3 technical report, 2025. URL https://arxiv.org/ abs/2503.19786 . · arXiv
  31. Grand, G., Pepe, V., Andreas, J., and Tenenbaum, J. B. A llama sunk my battleship! asking rational questions with LLMs via bayesian inference. In The First Workshop on System-2 Reasoning at Scale, NeurIPS'24 , 2024. URL https://openreview.net/forum? id=DCer8frGLP .
  32. Grand, G., Pepe, V., Andreas, J., and Tenenbaum, J. B. Shoot first, ask questions later? building rational agents that explore and act like people, 2025. URL https: //arxiv.org/abs/2510.20886 .
  33. Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N. Reinforced self-training (rest) for language modeling, 2023. URL https://arxiv.org/abs/ 2308.08998 .
  34. Harris, K. and Slivkins, A. Should you use your large language model to explore or exploit?, 2025. URL https://arxiv.org/abs/2502.00225 . · arXiv
  35. Hausknecht, M., Ammanabrolu, P., Cˆ ot´ e, M.-A., and Yuan, X. Interactive fiction games: A colossal adventure, 2020a. URL https://arxiv.org/abs/1909.05398 . · arXiv
  36. Hausknecht, M., Ammanabrolu, P., Cˆ ot´ e, M.-A., and Yuan, X. Interactive fiction games: A colossal adventure, 2020b. URL https://arxiv.org/abs/1909.05398 . · arXiv
  37. Havrilla, A., Du, Y., Raparthy, S. C., Nalmpantis, C., Dwivedi-Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S., and Raileanu, R. Teaching large language models to reason with reinforcement learning, 2024. URL https://arxiv.org/abs/2403.04642 . · arXiv
  38. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874 . · arXiv
  39. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024. · arXiv
  40. Jansen, P., Cˆ ot´ e, M.-A., Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents, 2024. URL https://arxiv.org/abs/2406.06769 . · arXiv
  41. Jansen, P. A. A systematic survey of text worlds as embodied natural language environments, 2021. URL https:// arxiv.org/abs/2107.04132 . · arXiv
  42. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023a. URL https: //arxiv.org/abs/2310.06825 .
  43. Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefenstette, E., and Rockt¨ aschel, T. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems , 34:1884-1897, 2021a.
  44. Jiang, M., Grefenstette, E., and Rockt¨ aschel, T. Prioritized level replay. In International Conference on Machine Learning , pp. 4940-4950. PMLR, 2021b.
  45. Jiang, Y., Kolter, J. Z., and Raileanu, R. On the importance of exploration for generalization in reinforcement learning. Advances in Neural Information Processing Systems , 36:12951-12986, 2023b.
  46. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35: 22199-22213, 2022.
  47. Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., and Slivkins, A. Can large language models explore incontext?, 2024. URL https://arxiv.org/abs/ 2403.15371 .
  48. Kwan, W.-C., Zeng, X., Jiang, Y., Wang, Y., Li, L., Shang, L., Jiang, X., Liu, Q., and Wong, K.-F. Mt-eval: A multiturn capabilities evaluation benchmark for large language models. arXiv preprint arXiv:2401.16745 , 2024. · arXiv
  49. Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215 , 2022. · arXiv
  50. Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.
  51. Li, S. S., Mun, J., Brahman, F., Ilgen, J. S., Tsvetkov, Y ., and Sap, M. Aligning llms to ask good questions a case study in clinical reasoning, 2025. URL https://arxiv. org/abs/2502.14860 .
  52. Lin, L., Bai, Y ., and Mei, S. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=yN4Wv17ss3 .
  53. Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL https: //arxiv.org/abs/1608.03983 .
  54. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/ 1711.05101 .
  55. MetaAI, Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., et al. The llama 3 herd of models, 2024. URL https:// arxiv.org/abs/2407.21783 . · arXiv
  56. Monea, G., Bosselut, A., Brantley, K., and Artzi, Y. Llms are in-context bandit reinforcement learners, 2025. URL https://arxiv.org/abs/2410.05362 . · arXiv
  57. Mukobi, G., Chatain, P., Fong, S., Windesheim, R., Kutyniok, G., Bhatia, K., and Alberti, S. Superhf: Supervised iterative learning from human feedback, 2023. URL https://arxiv.org/abs/2310.16763 . · arXiv
  58. Murty, S., Manning, C. D., Shaw, P., Joshi, M., and Lee, K. BAGEL: Bootstrapping agents by guiding exploration with language. In Forty-first International Conference on Machine Learning , 2024. URL https: //openreview.net/forum?id=VsvfSMI5bs .
  59. Narayanan, S., Braza, J. D., Griffiths, R.-R., Ponnapati, M., Bou, A., Laurent, J., Kabeli, O., Wellawatte, G., Cox, S., Rodriques, S. G., and White, A. D. Aviary: training language agents on challenging scientific tasks, 2024. URL https://arxiv.org/abs/2412.21154 . · arXiv
  60. Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv, R. Turning up the heat: Min-p sampling for creative and coherent llm outputs. arXiv preprint arXiv:2407.01082 , 2024. · arXiv
  61. Nie, A., Su, Y., Chang, B., Lee, J. N., Chi, E. H., Le, Q. V ., and Chen, M. Evolve: Evaluating and optimizing llms for exploration, 2024. URL https://arxiv.org/ abs/2410.06238 . · arXiv
  62. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., et al. Gpt-4 technical report, 2024a. URL https: //arxiv.org/abs/2303.08774 .
  63. OpenAI, Jaech, A., Kalai, A., Lerer, A., Richardson, A., ElKishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024b. · arXiv
  64. Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.
  65. Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https:// arxiv.org/abs/2402.13228 . · arXiv
  66. Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization, 2024. URL https://arxiv.org/abs/2404. 19733 .
  67. Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pp. 2778-2787. PMLR, 2017.
  68. Pathak, D., Gandhi, D., and Gupta, A. Self-supervised exploration via disagreement. In International conference on machine learning , pp. 5062-5071. PMLR, 2019.
  69. Portelas, R., Colas, C., Hofmann, K., and Oudeyer, P.-Y. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning , pp. 835-853. PMLR, 2020a.
  70. Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P.-Y. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664 , 2020b. · arXiv
  71. Qwen, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/ abs/2412.15115 . · arXiv
  72. Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q ∗ : Your language model is secretly a q-function, 2024a. URL https://arxiv.org/abs/2404.12358 . · arXiv
  73. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024b.
  74. Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., and Raileanu, R. Generalization to new sequential decision making tasks with in-context learning. arXiv preprint arXiv:2312.03801 , 2023. · arXiv
  75. Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization, 2024. URL https://arxiv.org/abs/2410.08847 . · arXiv
  76. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022 . · arXiv
  77. Rothe, A., Lake, B., and Gureckis, T. Asking and evaluating natural language questions. In Papafragou, A., Grodner, D., Mirman, D., and Trueswell, J. (eds.), Proceedings of the 38th Annual Meeting of the Cognitive Science Society, CogSci 2016 , Proceedings of the 38th Annual Meeting of the Cognitive Science Society, CogSci 2016, pp. 20512056. The Cognitive Science Society, 2016. Publisher Copyright: © 2016 Proceedings of the 38th Annual Meeting of the Cognitive Science Society, CogSci 2016. All rights reserved.; 38th Annual Meeting of the Cognitive Science Society: Recognizing and Representing Events, CogSci 2016 ; Conference date: 10-08-2016 Through 13-08-2016.
  78. Rothe, A., Lake, B. M., and Gureckis, T. M. Question asking as program generation, 2017. URL https:// arxiv.org/abs/1711.06351 . · arXiv
  79. Rothe, A., Lake, B. M., and Gureckis, T. M. Do people ask good questions? Computational Brain & Behavior , 1: 69-89, 2018. doi: 10.1007/s42113-018-0005-5. · DOI
  80. Rothe, A., Lake, B. M., and Gureckis, T. M. Asking goaloriented questions and learning from answers. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society: Creativity + Cognition + Computation, CogSci 2019 , pp. 981-986, Montreal, QC, Canada, 2019. The Cognitive Science Society.
  81. Schmidhuber, J. Curious model-building control systems. In Proc. international joint conference on neural networks , pp. 1458-1463, 1991.
  82. Schmidhuber, J. G¨ odel machines: Fully self-referential optimal universal self-improvers. In Artificial general intelligence , pp. 199-226. Springer, 2007.
  83. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. · arXiv
  84. Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657 , 2019. · arXiv
  85. Skinner, B. F. Reinforcement today. American Psychologist , 13(3):94, 1958.
  86. Slivkins, A. Introduction to multi-armed bandits, 2024. URL https://arxiv.org/abs/1904.07272 . · arXiv
  87. Sun, Y., Liu, C., Huang, J., Song, R., Zhang, F., Zhang, D., Wang, Z., and Gai, K. Parrot: Enhancing multi-turn chat models by learning to ask questions. arXiv preprint arXiv:2310.07301 , 2023. · arXiv
  88. Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.
  89. Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of llms should leverage suboptimal, on-policy data, 2024. URL https://arxiv.org/abs/2404. 14367 .
  90. Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.
  91. Wang, P.-A., Tzeng, R.-C., and Proutiere, A. Best arm identification with fixed budget: A large deviation perspective, 2024a. URL https://arxiv.org/abs/ 2312.12137 .
  92. Wang, R., Lehman, J., Clune, J., and Stanley, K. O. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 , 2019. · arXiv
  93. Wang, R., Jansen, P., Cˆ ot´ e, M.-A., and Ammanabrolu, P. Scienceworld: Is your agent smarter than a 5th grader?, 2022. URL https://arxiv.org/abs/2203.07540 . · arXiv
  94. Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations , 2024b. URL https://openreview.net/forum? id=jp3gWrMuIZ .
  95. Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multitask language understanding benchmark, 2024c. URL https://arxiv.org/abs/2406.01574 . · arXiv
  96. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.
  97. Wolfram, S. Statistical mechanics of cellular automata. Reviews of modern physics , 55(3):601, 1983.
  98. Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024. URL https: //arxiv.org/abs/2404.10719 .
  99. Yang, Z., Li, P., Yan, M., Zhang, J., Huang, F., and Liu, Y. React meets actre: Autonomous annotation of agent trajectories for contrastive self-training. In First Conference on Language Modeling , 2024. URL https: //openreview.net/forum?id=0VLBwQGWpA .
  100. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models, 2023. URL https://arxiv. org/abs/2210.03629 .
  101. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203.14465 . · arXiv
  102. Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y ., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild, 2024. URL https://arxiv.org/abs/2405.01470 . · arXiv
  103. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https: //arxiv.org/abs/2306.05685 .
  104. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models, 2023. URL https: //arxiv.org/abs/2311.07911 .

Cited by

Sorted by citations

Authors
Tajwar, F., Jiang, Y., Thankaraj, A., Rahman, S. S., Kolter, J. Z., Schneider, J., & Salakhutdinov, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.17543
arXiv
2502.17543
OpenAlex
W4415186413
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
language model fine-tuning, exploration, sequential decision making, in-context learning, zero-shot transfer, curriculum learning, sample efficiency, paprika, autonomous agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review