BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Open accessAbstract
Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortunately, LM agents often fail to generalize to new environments without human demonstrations. This work presents BAGEL, a method for bootstrapping LM agents without human supervision. BAGEL converts a seed set of randomly explored trajectories or synthetic instructions, into demonstrations, via round-trips between two noisy LM components: an LM labeler which converts a trajectory into a synthetic instruction, and a zero-shot LM agent which maps the synthetic instruction into a refined trajectory. By performing these round-trips iteratively, BAGEL quickly converts the initial distribution of trajectories towards those that are well-described by natural language. We use BAGEL demonstrations to adapt a zero shot LM agent at test time via in-context learning over retrieved demonstrations, and find improvements of over 2-13% absolute on ToolQA and MiniWob++, with up to 13x reduction in execution failures.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 35
- Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. · arXiv
- Allen, J., Chambers, N., Ferguson, G., Galescu, L., Jung, H., Swift, M., and Taysom, W. Plow: a collaborative task learning agent. In Proceedings of the 22nd National Con- ference on Artificial Intelligence -Volume 2, AAAI'07, pp. 1514-1519. AAAI Press, 2007. ISBN 9781577353232.
- Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
- Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. · arXiv
- Branavan, S., Chen, H., Zettlemoyer, L., and Barzilay, R. Reinforcement learning for mapping instructions to actions. In Su, K.-Y., Su, J., Wiebe, J., and Li, H. (eds.), Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th Interna- tional Joint Conference on Natural Language Processing of the AFNLP, pp. 82-90, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https://aclanthology.org/P09-1010.
- Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Ra- jagopal, D., and Salakhutdinov, R. Gated-attention archi- tectures for task-oriented language grounding. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018.
- Cideron, G., Seurin, M., Strub, F., and Pietquin, O. Higher: Improving instruction following with hindsight genera- tion for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 225- 232. IEEE, 2020.
- Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692, 2023. · arXiv
- Furuta, H., Nachum, O., Lee, K.-H., Matsuo, Y., Gu, S. S., and Gur, I. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854, 2023. · arXiv
- Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. · arXiv
- Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting ac- tionable knowledge for embodied agents. In Interna- tional Conference on Machine Learning, pp. 9118-9147.
- PMLR, 2022.
- Humphreys, P. C., Raposo, D., Pohlen, T., Thornton, G., Chhaparia, R., Muldal, A., Abramson, J., Georgiev, P., Santoro, A., and Lillicrap, T. A data-driven approach for learning to control computers. In International Con- ference on Machine Learning, pp. 9466-9482. PMLR, 2022.
- Jiang, Y., Gu, S. S., Murphy, K. P., and Finn, C. Language as an abstraction for hierarchical deep reinforcement learn- ing. Advances in Neural Information Processing Systems, 32, 2019.
- Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491, 2023. · arXiv
- Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Re- ward design with language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= 10uNUgI5Kl.
- Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 6086-6096, 2019.
- Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
- Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Rein- forcement learning on web interfaces using workflow- guided exploration. In International Conference on Learning Representations, 2018.
- Logeswaran, L., Fu, Y., Lee, M., and Lee, H. Few-shot subgoal planning with language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5493-5506, 2022.
- Lù, X. H., Kasner, Z., and Reddy, S. Weblinx: Real- world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930, 2024. · arXiv
- Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022. · arXiv
- Misra, D., Langford, J., and Artzi, Y. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795, 2017. · arXiv
- Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with hu- man feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
- Parisi, A., Zhao, Y., and Fiedel, N. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022. · arXiv
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21: 1-67, 2020.
- Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., and Toutanova, K. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306.00245, 2023. · arXiv
- Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. World of bits: An open-domain platform for web- based agents. In International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
- Shinn, N., Labash, B., and Gopinath, A. Reflexion: an au- tonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. · arXiv
- Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023. · arXiv
- Sodhi, P., Branavan, S., and McDonald, R. Heap: Hierar- chical policies for web actions using llms. arXiv preprint arXiv:2310.03720, 2023. · arXiv
- Sun, H., Zhuang, Y., Kong, L., Dai, B., and Zhang, C. Ada- planner: Adaptive planning from feedback with language models. arXiv preprint arXiv:2305.16653, 2023. · arXiv
- Xiao, T., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., and Tompson, J. Robotic skill acquisition via instruction augmentation with vision- language models. arXiv preprint arXiv:2211.11736, 2022. · arXiv
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
- Zhuang, Y., Yu, Y., Wang, K., Sun, H., and Zhang, C. Toolqa: A dataset for llm question answering with exter- nal tools. arXiv preprint arXiv:2306.13304, 2023. · arXiv
Cited by
Sorted by citations
Wang, Z. Z., Mao, J., Fried, D., & Neubig, G. (2024) · arXiv · doi:10.48550/arxiv.2409.07429 · Cited by 5 · see citing works
“Current agents mostly integrate a fixed set of given examples via training (Fu et al., 2024; Murty et al., 2024) or in-context learning (Zheng et al., 2024).”
BackgroundKanellopoulos, K., Sgouras, K., Bostanci, F. N., Kakolyris, A. K., Konar, B. K., Bera, R., Sadrosadati, M., Kumar, R., Vijaykumar, N., & Mutlu, O. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716027 · Cited by 8 · see citing works
ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow DataPreprint
Shen, J.-H., Jain, A., Xiao, Z., Amlekar, I., Hadji, M., Podolny, A., & Talwalkar, A. (2024) · Qeios · doi:10.32388/8vog0o · Cited by 13 · see citing works
SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
Zheng, B., Fatemi, M. Y., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y., Gu, Y., Srinivasa, J., Liu, G., Neubig, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.07079
Self-Challenging Language Model Agents
Zhou, Y., Levine, S., Weston, J., Li, X., & Sukhbaatar, S. (2025) · arXiv · doi:10.48550/arxiv.2506.01716 · Cited by 1
- Authors
- Murty, S., Manning, C., Shaw, P., Joshi, M., & Lee, K.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.08140
- OpenAlex
- W4392822311
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 1 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- DeepMind
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- language model agents, in-context learning, retrieval-augmented learning, exploration guidance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
