SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z. (2024) · arXiv · doi:10.48550/arxiv.2404.05719 · Cited by 40 · see citing works

A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models

Ning, L., Liang, Z., Jiang, Z., Qu, H., Ding, Y., Fan, W., Wei, X.-Y., Lin, S., Liu, H., Yu, P. S., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736555 · Cited by 41 · see citing works

ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

Li, K., Meng, Z., Lin, H., Luo, Z., Tian, Y., Ma, J., Huang, Z., & Chua, T.-S. (2025) · Proceedings of the 33rd ACM International Conference on Multimedia · doi:10.1145/3746027.3755688 · Cited by 27 · see citing works

AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation

Wen, H., Tian, S., Pavlov, B., Du, W., Li, Y., Chang, G., Zhao, S., Liu, J., Liu, Y., Zhang, Y.-Q., et al. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729134 · Cited by 19 · see citing works

Lifelong Learning of Large Language Model Based Agents: A Roadmap

Zheng, J., Shi, C., Cai, X., Li, Q., Zhang, D., Li, C., Yu, D., & Ma, Q. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3650546 · Cited by 7 · see citing works

Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

Pahuja, V., Lu, Y., Rosset, C., Gou, B., Mitra, A., Whitehead, S., Su, Y., & Awadallah, A. (2025) · arXiv · doi:10.48550/arxiv.2502.11357 · Cited by 7

Dual-View Visual Contextualization for Web Navigation

Kil, J., Song, C. H., Zheng, B., Deng, X., Su, Y., & Chao, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01369 · Cited by 5 · see citing works

ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow DataPreprint

Shen, J.-H., Jain, A., Xiao, Z., Amlekar, I., Hadji, M., Podolny, A., & Talwalkar, A. (2024) · Qeios · doi:10.32388/8vog0o · Cited by 13 · see citing works

Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks

Chen, Y., Hu, X., Yin, K., Li, J., & Zhang, S. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755646 · Cited by 6 · see citing works

Visual Test-Time Scaling for GUI Agent Grounding

Luo, T. S., Logeswaran, L., Johnson, J., & Lee, H. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01859 · Cited by 7 · see citing works

Toward an automated cross-multimodal verification of mobile app bug fixes integrating user feedback, developer responses, changelogs, and UI visual analysis

Massenon, R., Gambo, I., & Khan, J. A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107996 · Cited by 2 · see citing works

Characterizing Unintended Consequences of GUI Agents For Web Browsing

Zhang, S.-N., Chen, J.-R., Gao, Z., Gao, J., Yi, X., & Li, H.-W. (2026) · International Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790696 · Cited by 8 · see citing works

MT-Agent: Constructing a GUI Agent via Modality Enhancement and Text-Guided Fusion

Dong, J., Jin, L., Zhang, Z., Tang, W., Zhang, R., Xu, L., & Xing, J. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3600573 · Cited by 1 · see citing works

GenFlow: Interactive Modular System for Image Generation

Nguyen, D., Huynh, H.-P., Tran, M.-T., et al. (2025) · International Conference on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi66578.2025.11339318 · Cited by 1 · see citing works

Element-Aware Fine-Tuning of Vision-Language Models for Cost-Efficient GUI Testing in an Industrial Setting

Wu, M., Guo, Y., Cao, Y., Lu, H., Zhang, H., Zeng, X., Yao, L., Deng, Y., Ran, D., Yang, W., et al. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00296 · Cited by 1 · see citing works

An Empirical Study on UI Overlap in OpenHarmony Applications

Liu, F., Zhou, M., & Li, L. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00274 · Cited by 1 · see citing works

MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI

Shakeel, R., Ali, A. R. M., Mushtaq, M., Saleem, T. J., & Ashraf, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37718-0_15 · Cited by 1 · see citing works

Tuning Qwen2.5-VL to Improve Its Web Interaction Skills

Yakovleva, A., Pärssinen, H., Harri Valpola, B., et al. (2026) · ACM Web Conference (WWW) · doi:10.1145/3774904.3792932 · Cited by 1 · see citing works

Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot TaskPreprint

Wu, Z., Cheng, P., Wu, Z., Ju, T., Zhang, Z., & Liu, G. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5770750 · Cited by 1 · see citing works

"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent Collaboration

Kim, J., Cho, S., Nguyen, D. M., Jeon, J., Shin, M., & Ko, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791536

Enhancing UI Tests Robustness With Graph Convolutional Networks

Ayli, M., Bakouny, Y., Seifeddine, H., et al. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93598-5_27

Trishul: Towards Region Identification and Screen Hierarchy Understanding for Large VLM Based GUI Agents

Singh, K. V., Singh, S., & Khanna, M. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00022

On the Robustness of GUI Grounding Models Against Image Attacks

Zhao, H., Chen, T., & Wang, Z. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00149 · Cited by 1 · see citing works

A Domain-Agnostic Framework for Visual Element Detection From High-Level Descriptions

Ayli, M., Bakouny, Y., Jalloul, N., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3613058

G-TADS: GUI Task-Ability Decoupling Strategy for High-Adaptability Multimodal Intelligent Agents

Xia, Z., Zhang, X., Li, Y., Liu, Y., Shi, R., & Xu, J. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209200

CCAgent: Coordinating Collaborative Data Scaling for Operating System Agents via Web3

Chen, L., Zhao, H., Huang, Y., Luo, Y., Lin, T.-H., Xie, W., Wu, R., Wang, P., Xu, R., Wu, M., et al. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761392

Practical Escape of Exploration Tarpits for Mini-Game Testing in an Industrial Setting

Cao, Y., Ran, D., Lu, H., Guo, C., Hao, X., Chen, Z., Xiong, T., Deng, Y., & Xie, T. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00258

Single-shot vs. Zero-shot: A comparative study on GUI component recognition for automated Android testing

López, Y. P., de Oliveira, M. D., Albuquerque, L. D., et al. (2025) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi67495.2025.11358584

Vela: Vision-Enhanced Language Modeling for Scalable and Efficient GUI Operation

Ren, W., Wang, L., Wu, M., Zhou, J., Shi, X., Li, C., Hou, J., & Xiong, Q. (2025) · International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC) · doi:10.1109/icairc68035.2025.11385066

GG-R1: A Fine-Grained RFT Approach for GUI Grounding

Ye, J., Zhao, X., Feng, X., Gao, X., Zhang, X., Li, L., & Hong, W. (2025) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc67200.2025.11390507

Speech-visual integration multimodal multi-agents framework for mobile operation

Wang, Y., Li, J., & Xing, J. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00584-y

CU-1: A Class-Agnostic Detection Transformer Bridging Large Language Models and Big Data Interface Understanding

Rochet, A.-L., Despujols, M., Appourchaux, L., et al. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402585

GUIDE:Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation

Xie, R., Gao, Z., Shi, C., Shang, Z., Chen, L., & Li, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37026-6_32 · Cited by 1 · see citing works

Are GUI Agents Focused Enough? Automated Distraction via Semantic-Level UI Element Injection

Yang, W., Jin, C., Zhu, H., Luo, W., Yuen, D., Shao, K., Duan, J., Huang, H., Cao, J., & He, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37281-9_7

Specialized Approaches for Visual Instruction Grounding in GUI Automation

Ettifouri, E. H., Alassan, M. S. Y., Bouhandi, M., & Dahhane, W. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18716-1_6

AITG: Automating Intent-Oriented Task Generation for Mobile GUI-Agent

Zhang, C., Huang, Y., & Tu, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463965

ClueUp: Resolving Intent Ambiguity in Personalized Web Agents with Profile-Driven Clarification

Hu, D., Ge, J., Tang, W., Kong, H., Li, H., Wu, B., & Li, L. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464552

GUIGroup: Enabling Functional Layout Grouping With Multimodal Large Language Models

Zhang, Y., Yang, Y., Zhang, Z., Li, X., Liu, K., Xing, Z., Xia, X., & Bao, L. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3688725

UINavBench: A Framework for Comprehensive Evaluation of Interactive Digital Agents

Agrawal, H., Schoop, E., Pan, X., Mahajan, A., Seff, A., Feng, D., Cheng, R., y Terán, A. R. M., Gomez, E., Sundararajan, A., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02168

Pixel-to-Action: Reinforcement Learning-Based Agents for Computer Control Without Human Demonstration

Şimon, D.-O., & Bolboacă, R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-17033-0_3

ZonUI-3B: Competitive GUI Grounding with a 3B VLM Trained on a Single Consumer GPU

Hsieh, Z., Yang, S. F., & Wei, T.-J. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00100

MAS-DPD: Multi-Agent System for Dark Pattern Detection Using Large Vision-Language Models

Wang, X. (2026) · doi:10.1145/3807246.3807265

Enhancing the Power of GUI Agents by Reinforcement Learning

Liu, G. H., Ding, R., Jia, Z., Peng, X., Huang, K., Huang, F., Zhai, G., & Min, X. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-9507-5_7

Mechatronics-AI Collaboration: A Flexible Automated Testing Framework for Smart Home Devices Based on Multimodal Large Language Models

Yu, B., & Zhang, C. (2026) · International Conference on Mechatronics Technology and Intelligent Manufacturing (ICMTIM) · doi:10.1109/icmtim69588.2026.11526615

Bridging the Semantic Void: A Dual-branch RAG Framework to Enhance GUI Component Description for BVI Users

Wu, Y., Zhou, S., Wang, Z., Li, L., & Bu, J. (2026) · International Web for All Conference (W4A) · doi:10.1145/3800424.3800448

A UIA-Vision Dual-Engine Automated Testing Method with Domain-Aware Prompting for Medical Imaging Desktop Application

Shiqi, W. (2026) · International Conference on Multi-scale Artificial Intelligence (MAI) · doi:10.1109/mai69289.2026.11543776

Automated Oversight Diagnostic Self-Correction and Failure-Aware Reasoning for GUI Agents(ChinaMM)Preprint

Lu, X., Zhang, B., Qi, Z., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6991012

VisCritic: Visual State Comparison as Process Reward for GUI Agents

Qian, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37595-7_36

Win-Agent: A Multi-Agent Framework for Windows GUI Automation with Hybrid Lightweight Vision-Enhanced Perception

Liu, Y., Zhu, W., & Fan, R. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581472

Domain Adaptation of MLLM-Based GUI Agents in Documentation-Rich Environments with Standard Operating Procedures

Dong, L., Zhou, Z., Cheng, P., Wu, Z., Wu, Z., Liu, G., & Zhang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2856-0_34

AstraGame: Effective and Efficient VLM Agent Serving for Large-Scale Game Testing in an Industry Setting

Guo, Y., Lu, H., Wu, M. S., Xiong, T., Deng, Y., Ran, D., Yang, W. M., & Xie, T. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805252

ReflexAct: Closing the Loop for Web Agents via Visual Verification and Adaptive Grounding

Huang, J., Wu, J., & Li, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3403-5_14

Beyond Trajectory: Explicit Case-Based Memory for Decision-Making in Multimodal GUI Agents

Zhang, Y., & Zheng, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_21

A Platform-Aware Survey of Execution, Verification, and Risk of Graphical User Interface Agents

Wu, Y., & Yang, Y. (2026) · ENGINEERING Information Technology & Electronic Engineering · doi:10.1631/eng.itee.2026.0103

Reinforcement Learning for Multimodal Foundation Models: A Survey

Wu, W., Gao, C., Chen, J., Lin, Q., Meng, Q., Zhang, Y., Qiu, Y., Zhou, H., & Shou, M. Z. (2026) · ACM Computing Surveys · doi:10.1145/3837760 · Cited by 4 · see citing works

Conversational AI for Public Administration Form Filling: Evaluating Agentic Architectures for Citizen–Government Interaction

Diniz, P., Reis, L. P., & Cardoso, H. L. (2026) · IEEE Access · doi:10.1109/access.2026.3727913

DeCoPatch: Revealing Causal Latent Subspaces in Vision-Language Models for GUI Grounding

Zhang, Y., Kang, L., Wang, Z., Wu, D., & Wang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37359-5_2

: Open Reinforcement Learning Recipes for Visual Reasoning

Sarch, G. H., Cai, L., Wang, Q., Wu, H., Chen, D., & Liu, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36969-7_32

Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

Liu, Y., Tang, F., Yan, Y., Lu, Z., Nong, S., Jiang, T., Xu, W., Zhang, W., Lü, W., Xiao, J., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37029-7_14

Guiding the Blind: Generalizing GUI Agents to Unseen Websites via Multimodal Tutorials

Long, X., Tian, K., Xu, P., Gao, W., Shao, Y., Jia, G., Geng, H., Yang, S., Li, J., Hu, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37490-5_21

AI-augmented ACT-R modeling in real interfaces: automating visual perception, knowledge acquisition, and motor misexecution

Bagherzadeh, A., & Tehranchi, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1913953

DOM-YOLO: A Dual-Modal Vision-Structure Fusion Detector for Precise Web UI Element Localization in RPA

Chen, K. (2026) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11689139

A survey on AI agent security: Reasoning, Acting, and Self-Evolving

Liu, D., Luo, T., Huang, S., Yang, J., Liang, S., Jia, X., Lau, C. P., Dong, J., Liu, Y., Bai, Y., et al. (2026) · AI Plus · doi:10.59717/ipj.aiplus.2026.100004

Characterizing and Repairing Obsolete Android GUI Tests under UI Evolution

Song, S., Xiong, Y., Guo, W., Sun, M., Kong, J., & Xie, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832300

WanderVision: Scenario-Based GUI Testing for Smart TVs via Vision-Language Models

Ahn, S., Yoon, J., Kim, T., Song, E., & Yoo, S. (2026) · doi:10.1145/3843777.3844529

From Accessibility to Reachability: LLM-Driven Contrastive Accessibility Testing for Android Apps

Hu, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3844578

Evolutionary Multi-Agent Collaboration for Real-World Video Face Restoration

Tang, B., Wang, T., Zhang, M., Yu, X., Chen, J., Li, B., & Zhang, K. (2026) · doi:10.1109/cvprf71909.2026.00785

AndroidLens: Long-Latency Evaluation with Nested Sub-Targets for Android GUI Agents

Cao, Y., Wang, Y., Bu, P., Xing, J., Jiang, W., Zhu, Z., Ma, J., Zhou, S., Lu, T., Song, J., et al. (2026) · doi:10.1109/cvprf71909.2026.00187 · Cited by 1 · see citing works

UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding

Lian, S., Wu, Y., Ma, J., Ding, Y., Song, Z., Chen, B., Zheng, X., Li, H., & Ji, R. (2026) · doi:10.1109/cvprf71909.2026.00823

Anticipatory Planning for Multimodal AI Agents

Liang, Y., Zhou, S., Gu, Y., Tan, H., Wu, G., Dernoncourt, F., Kil, J., Rossi, R. A., & Zhang, R. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00569

Model Context Protocol Based AI Agent for PC Control

Manoranjitham, R., Anbarasan, S., Bharath, S., Punitha, S. C., Sathish, M., & Niranjan, S. (2026) · International Conference on Smart Systems for Electrical, Electronics, Communication and Computer Engineering (ICSSEECC) · doi:10.1109/icsseecc69062.2026.11722066

Same Outcomes, Different Journeys: Discovering Behavioral Divergence Between Humans and GUI Agents in Search Applications

Movin, M., Hauff, C., Henriksson, A., & Papapetrou, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41086-3_23

SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills

Zheng, B., Fatemi, M. Y., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y., Gu, Y., Srinivasa, J., Liu, G., Neubig, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.07079

OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

Li, Z., Wu, Z., Zhao, Y., Yang, B., Xie, J., Liu, Z., Liu, Z., Jin, K., Liang, J., Li, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.19191

MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents

Zhou, Z., Qu, A., Wu, Z., Kim, S., Prakash, A., Rus, D., Zhao, J., Low, B. K. H., & Liang, P. P. (2025) · arXiv · doi:10.48550/arxiv.2506.15841

ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents

Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., Mashkif, N., & Shlomov, S. (2024) · arXiv · doi:10.48550/arxiv.2410.06703 · Cited by 1 · see citing works

ИСПОЛЬЗОВAНИЕ ПОТЕНЦИAЛA СОЦИAЛЬНЫХ ПAРТНЕРОВ В ПОДГОТОВКЕ БУДУЩИХ ПЕДAГОГОВ

Атабекова, Б. Б. (2024) · Habaršy. Pedagogika ġylymdary seriâsy · doi:10.51889/2959-5762.2023.80.4.019

Susquehanna Chorale Spring Concert "Roots and Wings"

Hawkins, D. (2017)

Investigation on the Spot for Grounding Systems in Buildings

Gil, H.-J., Kim, D.-W., Kim, D.-O., et al. (2010) · Journal of the Korean Institute of Illuminating and Electrical Installation Engineers · doi:10.5207/jieie.2010.24.3.039 · Cited by 5 · see citing works

Using DataGrid Control to Realize DataBase of Querying in VB6.0

Hu, S. (2000) · Modern Computer

Study on Neutral Point Grounding Modes in Medium-Voltage Distribution Network

Fu, Z., Wang, N., Huang, L., et al. (2014) · International Symposium on Computer, Consumer and Control · doi:10.1109/is3c.2014.51 · Cited by 11 · see citing works