CoChat
Sign Up
PreprintCited by 97Software Testing and Debugging TechniquesWeb Data Mining and AnalysisMobile and Web Applications

WebArena: A Realistic Web Environment for Building Autonomous Agents

Open access
Zhou, Shuyan, Xu, Frank F., Zhu, Hao, Zhou, Xuhui, Lo, Robert, Sridhar, Abishek, Cheng, Xianyi, Ou, Tianyue +4 more·2023·arXiv
DOI 10.48550/arxiv.2307.13854arXiv 2307.13854
Open this paper in CoChat — free accountOpen PDF

Abstract

With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios. In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%. These results highlight the need for further development of robust agents, that current state-of-the-art large language models are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.13854

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current agentic benchmarks evaluate LLM performance in settings that involve no more than a few dozen rounds of interaction.

    “Current agentic benchmarks evaluate LLM performance in settings that involve no more than a few dozen rounds of interaction between a model and an environment”

    From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Zhou et al., 2023

    The paper’s Additional information section says its WebArena experiments capped execution at 30 state transitions, consistent with a few dozen interactions for this benchmark. Its comparison of existing benchmarks in §5.1 gives no interaction-round limits, so the broader claim about current agentic benchmarks is not established by this source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • In GUI and OS automation, agents synthesize and execute interface commands grounded in DOM trees, accessibility APIs, and executable evaluators.

    “In GUI and OS automation, agents synthesize and execute interface commands grounded in DOM trees, accessibility APIs, and executable evaluators [59, 60].”

    From Code as Agent Harness (2026) · cites this paper as [60]

    Section 2.4 (page unavailable) describes agents issuing browser actions using element IDs derived from DOM or accessibility trees. Section 3.2 (page unavailable) describes programmatic evaluators that assess task outcomes, not inputs that ground agent commands. The paper covers web automation, but does not establish the broader claim about OS automation or accessibility APIs. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Existing benchmarks for language agents do not test interaction with human users or ability to follow domain-specific rules, and instead feature simplified instruction-following setups without human-in-the-loop interaction or domain-specific guidelines.

    “Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications.”

    From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [29]

    The Introduction says existing agent environments often oversimplify real-world situations, and Table 4 in §5.1 compares benchmarks on realism and dynamic interaction. Neither passage establishes that existing benchmarks fail to test interaction with human users or adherence to domain-specific rules; dynamic interaction with an environment is not the same as interaction with a user. The paper does describe rules given to its own baseline agents (§4), but does not make the claimed comparison on that criterion. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • WebArena's URL/string assertions, OSWorld's per-task Python checkers, AndroidWorld's adb-based state inspection, and Spider2-V's enterprise-tool checks all share the pattern that an evaluator is itself a piece of code that interrogates the program world after the agent has finished.

    “WebArena's URL/string assertions, OSWorld's per-task Python checkers operating over OS file I/O and application state, AndroidWorld's adb -based state inspection, and Spider2-V's enterprise-tool checks all share the same pattern, an evaluator is itself a piece of code that interrogates the program world after the agent has finished [60, 396, 391, 413].”

    From Code as Agent Harness (2026) · cites this paper as [60]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • WebArena, BrowserGym, and TheAgentCompany expose Playwright-style code actions whose execution is the ground truth of progress.

    “WebArena, BrowserGym, and TheAgentCompany similarly expose Playwright-style code actions whose execution is the ground truth of progress [60, 398, 404].”

    From Code as Agent Harness (2026) · cites this paper as [60]

    Section 2.4 (Action Space; page unavailable) describes WebArena actions such as clicking, typing, and navigating, but presents them as a defined action space rather than Playwright-style code actions. The Introduction and Section 3 say task progress is evaluated by whether execution achieves the intended outcome, not by execution alone. This verdict concerns WebArena only; the cited paper does not establish the claim about BrowserGym or TheAgentCompany. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 46

  1. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpret- ing visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18- 22, 2018, pp. 3674-3683. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00387. URL http://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_ Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.html. · DOI · Paper page
  2. Yonatan Bisk, Jan Buys, Karl Pichotta, and Yejin Choi. Benchmarking hierarchical script knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4077-4085, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1412. URL https://aclanthology.org/N19-1412. · DOI
  3. S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 82-90, Suntec, Singapore, 2009. Association for Computational Linguistics. URL https://aclanthology.org/P09-1010.
  4. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. URL https://arxiv.org/abs/1606.01540. · arXiv
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107. 03374. · arXiv
  6. Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum? id=rc8o_j8I8PX.
  7. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023.
  8. Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: visual question answering in interactive environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4089-4098. IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00430. URL http://openaccess.thecvf.com/content_cvpr_2018/html/ Gordon_IQA_Visual_Question_CVPR_2018_paper.html. · DOI
  9. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023. · arXiv
  10. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9147. PMLR, 2022. URL https://proceedings.mlr.press/v162/huang22a.html.
  11. Yacine Jernite, Kavya Srinet, Jonathan Gray, and Arthur Szlam. CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant. ArXiv preprint, abs/1905.01978, 2019. URL https: //arxiv.org/abs/1905.01978. · arXiv
  12. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. ArXiv preprint, abs/2303.17491, 2023. URL https://arxiv.org/abs/2303.17491. · arXiv
  13. Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
  14. Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 569ff987c643b4bedf504efda8f786c2-Abstract.html.
  15. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026. · DOI · Paper page
  16. Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022.
  17. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893-18912. PMLR, 2023.
  18. Xinze Li, Yixin Cao, Muhao Chen, and Aixin Sun. Take a break in the middle: Investigating subgoals towards hierarchical script generation. ArXiv preprint, abs/2305.10907, 2023. URL https://arxiv.org/abs/2305.10907. · arXiv
  19. Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural lan- guage instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 8198-8210, Online, 2020. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.729. URL https: //aclanthology.org/2020.acl-main.729. · DOI
  20. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
  21. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id= ryTp3f-0-.
  22. Jieyi Long. Large language model guided tree-of-thought. ArXiv preprint, abs/2305.08291, 2023. URL https://arxiv.org/abs/2305.08291. · arXiv
  23. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384-1403, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https://aclanthology.org/ 2022.emnlp-main.90. Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281-300, 2016.
  24. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
  25. OpenAI. Chatgpt: Optimizing language models for dialogue. 2022. OpenAI. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
  27. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8494-8502. IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00886. URL http://openaccess.thecvf.com/content_cvpr_2018/html/ Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html. · DOI · Paper page
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/D16-1264. · DOI · Paper page
  29. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL https://aclanthology. org/P18-2124. · DOI · Paper page
  30. Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306.00245, 2023. · arXiv
  31. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3135- 3144. PMLR, 2017. URL http://proceedings.mlr.press/v70/shi17a.html.
  32. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv preprint, abs/2303.11366, 2023. URL https://arxiv. org/abs/2303.11366. · arXiv
  33. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10737-10746. IEEE, 2020. doi: 10. 1109/CVPR42600.2020.01075. URL https://doi.org/10.1109/CVPR42600.2020. 01075. · DOI · Paper page
  34. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id= 0IOX0YcCdTn.
  35. Abishek Sridhar, Robert Lo, Frank F Xu, Hao Zhu, and Shuyan Zhou. Hierarchical prompting assists large language model on web navigation. arXiv preprint arXiv:2305.14257, 2023. · arXiv
  36. Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. ArXiv preprint, abs/2105.13231, 2021. URL https://arxiv.org/abs/2105. 13231. · arXiv
  37. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv preprint, abs/2305.16291, 2023. URL https://arxiv.org/abs/2305.16291. · arXiv
  38. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. ArXiv preprint, abs/2212.10481, 2022. URL https://arxiv. org/abs/2212.10481. · arXiv
  39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
  40. Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1022-1032, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.80. URL https://aclanthology.org/ 2021.naacl-main.80. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259. · DOI
  41. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. volume abs/2207.01206, 2022a. URL https://arxiv.org/abs/2207.01206. · arXiv
  42. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022b. URL https://arxiv.org/abs/2210.03629. · arXiv
  43. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, abs/2305.10601, 2023. URL https://arxiv.org/abs/2305.10601. · arXiv
  44. Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arxiv 2017. ArXiv preprint, abs/1709.00103, 2017. URL https://arxiv.org/abs/1709.00103. · arXiv
  45. Shuyan Zhou, Pengcheng Yin, and Graham Neubig. Hierarchical control of situated agents through natural language. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pp. 67-84, Seattle, USA, 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.suki-1.8. URL https://aclanthology.org/2022.suki-1. · DOI
  46. Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, and Graham Neubig. Show me more details: Discovering hierarchies of procedures from semi-structured web data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2998-3012, Dublin, Ireland, 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.214. URL https://aclanthology.org/ 2022.acl-long.214. · DOI

Cited by

Sorted by citations

Authors
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., & Neubig, G.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.13854
arXiv
2307.13854
OpenAlex
W4385327683
Cited by
31
References
0
Citations by year
2026: 10 · 2025: 14 · 2024: 7
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • Zenodo (CERN European Organization for Nuclear Research)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
Keywords
WebArena, benchmarking, self-hosting, OpenStreetMap, web application testing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review