CoChat
Sign Up
PreprintCited by 47Reinforcement Learning in RoboticsWeb Data Mining and AnalysisInformation Retrieval and Search Behavior

Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

Open access
Liu, Evan Zheran, Guu, Kelvin, Pasupat, Panupong, Shi, Tianlin, Liang, Percy·2018·arXiv
DOI 10.48550/arxiv.1802.08802arXiv 1802.08802
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates. This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking flights or replying to emails, where a single mistake can ruin the entire sequence of actions. A common remedy is to "warm-start" the agent by pre-training it to mimic expert demonstrations, but this is prone to overfitting. Instead, we propose to constrain exploration using demonstrations. From each demonstration, we induce high-level "workflows" which constrain the allowable actions at each time step to be similar to those in the demonstration (e.g., "Step 1: click on a textbox; Step 2: enter some text"). Our exploration policy then learns to identify successful workflows and samples actions that satisfy these workflows. Workflows prune out bad exploration directions and accelerate the agent's ability to discover rewards. We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites, and evaluate on a suite of web tasks, including the recent World of Bits benchmark. We achieve new state-of-the-art results, and show that workflow-guided exploration improves sample efficiency over behavioral cloning by more than 100x.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1802.08802

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Unlike WebArena, MiniWoB++ tasks are easier without page changes, require fewer steps, and provide low-level step-by-step task directions.

    “Unlike WebArena, tasks are easier without page changes, require fewer steps, and provide low-level step-by-step task directions.”

    From OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024) · cites this paper as Liu et al., 2018

    Section 6.1 (pp. 5–6) identifies MiniWoB++ as a benchmark with compact web environments and text-specified goals. Section 6.3.2 (p. 7) describes MiniWoB goals as structured, but does not say they give step-by-step directions. The paper does not compare MiniWoB++ with WebArena or establish that its tasks avoid page changes or require fewer steps; Section 6.3.1 (p. 7) explicitly includes longer time horizons among MiniWoB++'s challenges. (LLM, conf=0.80)

    Experimental — accuracy not yet measured

  • MiniWoB++ tasks are synthetically initialized on 125 different minimalist web interfaces.

    “The tasks are synthetically initialized on 125 different minimalist web interfaces.”

    From OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024) · cites this paper as Liu et al., 2018

    Section 6.1 (pp. 5–6) says the authors constructed MiniWoB++ tasks and describes their small, 160 × 210 px web environments. It does not say the tasks are synthetically initialized or run on 125 different interfaces; the only benchmark count given there is 80 tasks for MiniWoB, a separate benchmark. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Existing research has shown that language models, without intervention, struggle to express linguistic intent in formal instruction that can control an extra-linguistic environment such as web site navigation.

    “Existing research has shown that language models, without intervention, struggle to express linguistic intent in formal instruction that can control an extra-linguistic environment, such as web site navigation (Shi et al., 2017; Liu et al., 2018; Yao et al., 2022; Zhou et al., 2024b; Deng et al., 2023).”

    From R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025) · cites this paper as Liu et al., 2018

    The Introduction (p. 1) says that reinforcement-learning agents struggle to learn web tasks through unaided trial and error because successful action sequences are hard to discover. The paper does not establish that language models struggle to express linguistic intent as formal instructions; it studies agents learning to click and type in response to goals. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • MiniWoB++ is an interactive web benchmark, with built-in reward functions.

    “MiniWoB++ (Liu et al., 2018) is an interactive web benchmark, with built-in reward functions.”

    From OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024) · cites this paper as Liu et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. P. Abbeel and A. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML) , 2004.
  2. J. Allen, N. Chambers, G. Ferguson, L. Galescu, H. Jung, M. Swift, and W. Taysom. PLOW: A collaborative task learning agent. In Association for the Advancement of Artificial Intelligence (AAAI) , pp. 1514-1519, 2007.
  3. D. Andre. Programmable reinforcement learning agents . PhD thesis, University of California, Berkeley, 2003.
  4. J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. arXiv preprint arXiv:1611.01796 , 2016. · arXiv
  5. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv:1707.01495 , 2017. · arXiv
  6. C. G. Atkeson and S. Schaal. Robot learning from demonstration. In International Conference on Machine Learning (ICML) , volume 97, pp. 12-20, 1997.
  7. P. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Association for the Advancement of Artificial Intelligence (AAAI) , pp. 1726-1734, 2017.
  8. N. Baram, O. Anschel, I. Caspi, and S. Mannor. End-to-end differentiable adversarial imitation learning. In International Conference on Machine Learning (ICML) , pp. 390-399, 2017.
  9. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research (JAIR) , 47: 253-279, 2013.
  10. S. Bitzer, M. Howard, and S. Vijayakumar. Using dimensionality reduction to exploit constraints in reinforcement learning. In International Conference on Intelligent Robots and Systems (IROS) , pp. 3219-3225, 2010.
  11. T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Now'e. Reinforcement learning from demonstration through shaping. In International Joint Conference on Artificial Intelligence (IJCAI) , 2015.
  12. N. Chentanez, A. G. Barto, and S. P. Singh. Intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems (NIPS) , pp. 1281-1288, 2005.
  13. C. Daniel, G. Neumann, O. Kroemer, and J. Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research (JMLR) , 17:3190-3239, 2016.
  14. B. Deka, Z. Huang, and R. Kumar. Erica: Interaction mining mobile apps. In User Interface Software and Technology (UIST) , pp. 767-776, 2016.
  15. T. G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In International Conference on Machine Learning (ICML) , 1998.
  16. C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML) , pp. 49-58, 2016.
  17. W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (NIPS) , 2017.
  18. T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys. Deep Q-learning from demonstrations. In Association for the Advancement of Artificial Intelligence (AAAI) , 2018.
  19. A. Hussein, E. Elyan, M. M. Gaber, and C. Jayne. Deep reward shaping from demonstrations. In International Joint Conference on Neural Networks , 2017.
  20. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 , 2016. · arXiv
  21. B. Kim, A. massoud Farahmand, J. Pineau, and D. Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems (NIPS) , pp. 2859-2867, 2013.
  22. T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.
  23. G. Konidaris and A. G. Barto. Building portable options: Skill transfer in reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI) , 2007.
  24. S. Levine and V. Koltun. Guided policy search. In International Conference on Machine Learning (ICML) , 2013.
  25. L. Li, T. J. Walsh, and M. L. Littman. Towards a unified theory of state abstraction for mdps. In International Symposium on Artificial Intelligence and Mathematics (ISAIM) , 2006.
  26. B. Marthi and C. Guestrin. Concurrent hierarchical reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI) , 2005.
  27. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML) , 2016.
  28. A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML) , volume 99, pp. 278-287, 1999.
  29. I. Osband, C. Blundell, A. Pritzel, and B. V. Roy. Deep exploration via bootstrapped DQN. In Advances In Neural Information Processing Systems , pp. 4026-4034, 2016.
  30. R. Parr and S. J. Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems (NIPS) , pp. 1043-1049, 1998.
  31. C. Perez-D'Arpino and J. A. Shah. C-learn: Learning geometric constraints from demonstrations for multi-step manipulation in shared autonomy. In International Conference on Robotics and Automation (ICRA) , pp. 4058-4065, 2017.
  32. T. Pham, T. Tran, D. Phung, and S. Venkatesh. Column networks for collective classification. In Association for the Advancement of Artificial Intelligence (AAAI) , 2017.
  33. M. Phillips, V. Hwang, S. Chitta, and M. Likhachev. Learning to plan for constrained manipulation from demonstrations. Autonomous Robots , 40(1):109-124, 2016.
  34. D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation , 3(1):88-97, 1991.
  35. M. Roderick, C. Grimm, and S. Tellex. Deep abstract Q-networks. arXiv preprint arXiv:1710.00459 , 2017. · arXiv
  36. S. Ross and J. A. Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979 , 2014. · arXiv
  37. S. Ross, G. Gordon, and A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS) , 2011.
  38. T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning (ICML) , 2017.
  39. W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International Conference on Machine Learning (ICML) , 2017.
  40. R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction , volume 1. MIT Press MIT press Cambridge, 1998.
  41. M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothorl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 , 2017. · arXiv
  42. Y. Wang, J. Berant, and P. Liang. Building a semantic parser overnight. In Association for Computational Linguistics (ACL) , 2015.
  43. T. Weber, S. Racanière, D. P. Reichert, L. Buesing, A. Guez, D. J. Rezende, A. P. Badia, O. Vinyals, N. Heess, Y. Li, et al. Imagination-augmented agents for deep reinforcement learning. arXiv preprint arXiv:1707.06203 , 2017. · arXiv
  44. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.
  45. T. Yeh, T. Chang, and R. Miller. Sikuli: using GUI screenshots for search and automation. In User Interface Software and Technology (UIST) , 2009.
  46. B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence (AAAI) , 2008.

Cited by

Sorted by citations

Authors
Liu, E. Z., Guu, K., Pasupat, P., Shi, T., & Liang, P.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1802.08802
arXiv
1802.08802
OpenAlex
W2786616516
Cited by
47
References
3
Citations by year
2026: 2 · 2025: 2 · 2024: 7 · 2023: 2 · 2022: 1 · 2021: 16 · 2020: 8 · 2019: 5 · 2018: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1552635, CAREER)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
reinforcement learning, web interface, sparse rewards, sample efficiency, behavior cloning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review