VisualWebArena: evaluating multimodal agents on realistic visual web tasks
Open accessAbstract
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, Daniel Fried. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
WebArena and VisualWebArena are open-source academic implementations operating as desktop applications that launch browser instances for task execution.
“WebArena [8] and VisualWebArena [9], open-source academic implementations operating as desktop applications, also launch browser instances for task execution.”
From Investigating the Impact of Dark Patterns on LLM-Based Web Agents (2026) · cites this paper as [9]
Section 3.1 describes VisualWebArena as a web-browsing environment with URLs and open tabs, and Table 1 lists actions for opening tabs and navigating to URLs. Section 3 says its websites use open-source infrastructure and are standalone self-hosted web applications. The paper does not establish that VisualWebArena is a desktop application, that it launches browser instances for task execution, or that the benchmark itself is open-source. (LLM, conf=0.84)
Experimental — accuracy not yet measuredExisting agent logging frameworks often rely on specific agent implementations that hard-code crucial aspects of agent behavior, defining fixed prompts, predetermined methods for environmental observations, and limited action space, which constrains evaluation of diverse agent implementations.
“Existing approaches [8], [9] often rely on specific agent implementations that hard-code crucial aspects of agent behavior.”
From Investigating the Impact of Dark Patterns on LLM-Based Web Agents (2026) · cites this paper as [9]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredLanguage model agents that interact with a real or virtual world have proliferated as the default setting for web navigation, computer control, and code generation tasks.
“LM agents [17, 42, 47, 54] that interact with a real/virtual world have proliferated as the default setting for web navigation [24, 33, 36, 41, 45, 61, 62, 71], computer control [35, 53, 57], and code generation tasks [16, 50, 63].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [24]
Computer use is another popular area of benchmark research including WebArena, Visual Web Arena, AppWorld, and OS World.
“Computer use is another popular area of benchmark research including WebArena (Zhou et al., 2024), Visual Web Arena (Koh et al., 2024), AppWorld (Trivedi et al., 2024), and OS World (Xie et al., 2024).”
From Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026) · cites this paper as Koh et al., 2024
Benchmark suites such as DeathStarBench provide reproducible microservice workloads that can support end-to-end evaluation of diagnosis and mitigation policies under stress.
“Benchmark suites such as DeathStarBench provide reproducible microservice workloads that can support end-to-end evaluation of diagnosis and mitigation policies under stress [166].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [166]
The Introduction (page unavailable) describes VisualWebArena’s reproducible environments for visually grounded web-agent tasks, not microservice workloads. Searches for DeathStarBench, microservices, and diagnosis or mitigation under stress found no passage addressing the claim in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- NeurIPS.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS.
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. JMLR.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. ICLR.
- NeurIPS.
- Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. ICML.
- Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tian- lin Shi, and Percy Liang. 2018. Reinforcement learn- ing on web interfaces using workflow-guided explo- ration. ICLR.
- Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. ICML.
- Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Her- nandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In ICML.
- Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. ICLR.
- Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Mul- timodal few-shot learning with frozen language mod- els. NeurIPS.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS.
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. 2023c. Self-consistency im- proves chain of thought reasoning in language mod- els. ICLR.
- NeurIPS.
- NeurIPS.
- Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2024. Webarena: A realistic web environment for building autonomous agents. ICLR. ˋˋˋclick [idˋˋˋ: This action clicks on an element with a specific id on the webpage.
Cited by
Sorted by citations
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
Ning, L., Liang, Z., Jiang, Z., Qu, H., Ding, Y., Fan, W., Wei, X.-Y., Lin, S., Liu, H., Yu, P. S., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736555 · Cited by 41 · see citing works
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
Li, K., Meng, Z., Lin, H., Luo, Z., Tian, Y., Ma, J., Huang, Z., & Chua, T.-S. (2025) · Proceedings of the 33rd ACM International Conference on Multimedia · doi:10.1145/3746027.3755688 · Cited by 27 · see citing works
Constraint-Consistent Skill Composition for Reliable Zero-Shot Task Generalization in LLM Agents
Meng, R., Huang, S., Zhang, X., Yang, Y., Zeng, K., & Huang, Z. (2026) · International Symposium on Big Data and Applied Statistics (ISBDAS) · doi:10.1109/isbdas69350.2026.11484408 · Cited by 21 · see citing works
GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots
Singh, S., Fore, M., & Stamoulis, D. S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00063 · Cited by 21 · see citing works
LLM-Powered AI Agent Systems and Their Applications in Industry
Liang, G., & Tong, Q. (2025) · IEEE World AI IoT Congress (AIIoT) · doi:10.1109/aiiot65859.2025.11105299 · Cited by 32 · see citing works
Multi-Agent Geospatial Copilots for Remote Sensing Workflows
Lee, C., Paramanayakam, V., Καρατζάς, Α., Jian, Y., Fore, M., Liao, H., Yu, F., Li, R., Anagnostopoulos, I., & Stamoulis, D. S. (2025) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss55030.2025.11243915 · Cited by 24 · see citing works
WebWalker: Benchmarking LLMs in Web Traversal
Wu, J., Yin, W., Jiang, Y., Wang, Z., Xi, Z., Fang, R., Zhang, L., He, Y., Zhou, D., Xie, P., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.508 · Cited by 18 · see citing works
Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
Deng, A., Cao, T., Chen, Z., & Hooi, B. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00366 · Cited by 19 · see citing works
Kapoor, R., Butala, Y. P., Russak, M., Koh, J. Y., Kamble, K., AlShikh, W., & Salakhutdinov, R. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73113-6_10 · Cited by 12 · see citing works
Self-citationAutoPresent: Designing Structured Visuals from Scratch
Ge, J., Wang, Z. Z., Zhou, X., Peng, Y.-H., Subramanian, S., Tan, Q., Sap, M., Suhr, A., Fried, D., Neubig, G., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00276 · Cited by 10 · see citing works
Self-citationWang, Z. Z., Mao, J., Fried, D., & Neubig, G. (2024) · arXiv · doi:10.48550/arxiv.2409.07429 · Cited by 5 · see citing works
Self-citationLess is More: Optimizing Function Calling for LLM Execution on Edge Devices
Paramanayakam, V., Καρατζάς, Α., Anagnostopoulos, I., & Stamoulis, D. S. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992798 · Cited by 11 · see citing works
Stamoulis, D. S., & Marculescu, D. (2025) · ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3715335.3735494 · Cited by 9 · see citing works
Lifelong Learning of Large Language Model Based Agents: A Roadmap
Zheng, J., Shi, C., Cai, X., Li, Q., Zhang, D., Li, C., Yu, D., & Ma, Q. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3650546 · Cited by 7 · see citing works
Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.05080 · Cited by 7 · see citing works
Fang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6 · see citing works
Multimodal Deep Learning Framework for Automated Usability Evaluation of Fashion E-Commerce Sites
Alowidi, N. (2025) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer20040343 · Cited by 3 · see citing works
Visual Test-Time Scaling for GUI Agent Grounding
Luo, T. S., Logeswaran, L., Johnson, J., & Lee, H. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01859 · Cited by 7 · see citing works
Difficulty-Aware Hybrid Routing for Cost-Efficient Long-Context LLM Inference
Gu, Z., Xiao, Z., & Lin, O. (2026) · International Conference on Intelligent Communications and Computing (ICICC) · doi:10.1109/icicc71012.2026.11637729 · Cited by 3 · see citing works
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers
Green, T., Gubri, M., Puerto, H., Yun, S., & Oh, S. J. (2025) · arXiv · doi:10.48550/arxiv.2506.15674 · Cited by 3
ToolVQA: A Dataset for Multi-Step Reasoning VQA with External Tools
Yin, S., Lei, T., & Liu, Y. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00421 · Cited by 18 · see citing works
Ying, T., Wang, L., Hu, P., & Ding, T. (2026) · International Conference on Electronic Technology and Information Science (ICETIS) · doi:10.1109/icetis70504.2026.11633437 · Cited by 5 · see citing works
Precise Image Editing with Multimodal Agents
Fu, B., Zhang, C., Yin, F., Pei, C., & Huang, Z. (2024) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai62207.2024.10826725 · Cited by 2 · see citing works
SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing
Tu, R.-C., Sun, W., Jin, Z., Liao, J., Huang, J., & Tao, D. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3673949 · Cited by 5 · see citing works
ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
Chen, Z., Kang, M., & Li, B. (2025) · arXiv · doi:10.48550/arxiv.2503.22738 · Cited by 2 · see citing works
AI Web Agents Can Effectively Guide Lesson Design and Predict Student Outcomes
Wang, S., Mitchell, J. C., & Piech, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_21 · Cited by 1 · see citing works
GraphVideoAgent: Enhancing Long-form Video Understanding with Entity Relation Graphs
Chu, M., Li, Y., & Chua, T. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755537 · Cited by 3 · see citing works
GenFlow: Interactive Modular System for Image Generation
Nguyen, D., Huynh, H.-P., Tran, M.-T., et al. (2025) · International Conference on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi66578.2025.11339318 · Cited by 1 · see citing works
AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping
Kim, S., Heo, R., Seo, Y., Yeo, J., & Lee, D. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792724 · Cited by 1 · see citing works
Gómez, R., Miranda, C. E., Romero-González, J.-A., Esparza, D. M. C., Alfonso-Francia, G., Chávez‐Urbiola, E. A., Ramírez-Pedraza, A., & Terven, J. R. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8060141 · Cited by 1 · see citing works
Macklon, F., Boucher, C., & Bezemer, C. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10906-3 · Cited by 1 · see citing works
Zhuhadar, L. (2026) · Future Internet · doi:10.3390/fi18070376 · Cited by 1 · see citing works
"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent Collaboration
Kim, J., Cho, S., Nguyen, D. M., Jeon, J., Shin, M., & Ko, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791536
Investigating the Impact of Dark Patterns on LLM-Based Web Agents
Ersoy, D., Lee, B., Shreekumar, A., Arunasalam, A., Ibrahim, M., Bianchi, A., & Berkay Celik, Z. (2026) · 2026 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp63933.2026.00042 · Cited by 1 · see citing works
Dissecting Adversarial Robustness of Multimodal LM Agents
Wu, C. H., Shah, R., Koh, J. Y., Salakhutdinov, R., Fried, D., & Raghunathan, A. (2024) · arXiv · doi:10.48550/arxiv.2406.12814 · Cited by 1 · see citing works
Self-citationLarge language models for aspect-based sentiment analysis in e-commerce: A review
Huang, H., Li, Y., Dong, Q., Asemi, A., Mustafa, M. B., & Asemi, A. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116083 · Cited by 1 · see citing works
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., Zheng, S., Choe, K. W., Sun, J., Wang, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12670 · Cited by 1 · see citing works
MapQaTor: A System for Efficient Annotation of Map Query DatasetsPreprint
Dihan, M. L., Ali, M. E., & Parvez, M. R. (2025) · Qeios · doi:10.32388/mi3wh4
On the Robustness of GUI Grounding Models Against Image Attacks
Zhao, H., Chen, T., & Wang, Z. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00149 · Cited by 1 · see citing works
FlowJD: Your Imagination Can Help You Jailbreak in Visual Language Models
Zou, X., Chen, Y., Han, Q., & Li, K. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209893
Tan, W., Jiang, C., Duan, Y., Lei, M., Li, J., Hong, Y., Wang, X., & An, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_14 · Cited by 1 · see citing works
Who’s to Blame? Rethinking the Brittleness of Automated Web GUI Testing from a Pragmatic Perspective
Zhang, H., Yao, K., Ding, Z., Liao, L., & Shang, W. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00251
Rochet, A.-L., Despujols, M., Appourchaux, L., et al. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402585
UINavBench: A Framework for Comprehensive Evaluation of Interactive Digital Agents
Agrawal, H., Schoop, E., Pan, X., Mahajan, A., Seff, A., Feng, D., Cheng, R., y Terán, A. R. M., Gomez, E., Sundararajan, A., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02168
KBase Research Agent: Automated Multi-Agent Workflow Construction for Reproducible Genome Analysis *Preprint
Gupta, P., Riehl, W. J., Cashman, M., Chivian, D. C., Neely, C. J., Canon, R. S., Cottingham, R., Henry, C., Arkin, A. P., & Dehal, P. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.01.729336
KBase Research Agent: Automated Multi-Agent Workflow Construction for Reproducible Genome Analysis
Gupta, P., Riehl, W. J., Cashman, M., Chivian, D. C., Neely, C. J., Canon, R. S., Cottingham, R. W., Henry, C. S., Arkin, A. P., & Dehal, P. (2026) · Platform for Advanced Scientific Computing Conference (PASC) · doi:10.1145/3815572.3815744
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Dong, W., Fu, T., Yu, Z., Wang, H., Su, A., Fang, Z., Chen, Y., Wang, S., Wu, M., Jiang, P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37035-8_28
Task-Adaptive Retrieval over Agentic Multi-Modal Web Histories via Learned Graph Memory
Forouzandeh, S., Berahmand, K., & Jalili, M. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809978
WebPilot : An Agent Assisted Browser Extension
Rawat, A., Chaudhary, S., Singh, S., et al. (2026) · doi:10.1109/icicgr68236.2026.11600083
Leveraging Agentic AI for IoT Security Operations
Mosciatti, G., Meier, R., Himmelmann, L., & Lenders, V. (2026) · doi:10.23919/cycon70911.2026.11609433
DRIVE: Organizing Web-Agent Experience into Reasoning and Interaction SkillsPreprint
Liu, X., Zhou, S., Hou, Y., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7206474
Self-citationSelf-evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning
He, T., Chen, Y., Jiang, K., Lee, K. Y., Zhou, K., Shao, K., & Wang, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37556-8_35
Guiding the Blind: Generalizing GUI Agents to Unseen Websites via Multimodal Tutorials
Long, X., Tian, K., Xu, P., Gao, W., Shao, Y., Jia, G., Geng, H., Yang, S., Li, J., Hu, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37490-5_21
WebRLED: A Deep Reinforcement Learning Approach for Automated Web Application Exploration
Gu, Z., Wu, G., Zhang, Y., Yang, C., Chen, W., & Wei, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3843235
A survey on AI agent security: Reasoning, Acting, and Self-Evolving
Liu, D., Luo, T., Huang, S., Yang, J., Liang, S., Jia, X., Lau, C. P., Dong, J., Liu, Y., Bai, Y., et al. (2026) · AI Plus · doi:10.59717/ipj.aiplus.2026.100004
Self-citationLops, A., Ferrara, A., Narducci, F., Ragone, A., & Trizio, M. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834507
Movin, M., Hauff, C., Henriksson, A., & Papapetrou, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41086-3_23
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.11868
Self-citationWho Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents
Wang, Z., Li, Y., Wu, Y., Chen, K., Liu, Z., Wai, F. K., Chen, P.-Y., Thing, V. L. L., Li, B., Tao, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.13385
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075
- Authors
- Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 881-905
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402667024
- Cited by
- 50
- References
- 0
- Citations by year
- 2026: 28 · 2025: 19 · 2024: 3
- FWCI
- 10.32
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- multimodal agents, agent evaluation, multimodal language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
