UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Open accessAbstract
This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions (e.g., keyboard and mouse operations). Unlike prevailing agent frameworks that depend on heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts and workflows, UI-TARS is an end-to-end model that outperforms these sophisticated frameworks. Experiments demonstrate its superior performance: UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating perception, grounding, and GUI task execution. Notably, in the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld, UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of GUI screenshots for context-aware understanding of UI elements and precise captioning; (2) Unified Action Modeling, which standardizes actions into a unified space across platforms and achieves precise grounding and interaction through large-scale action traces; (3) System-2 Reasoning, which incorporates deliberate reasoning into multi-step decision making, involving multiple reasoning patterns such as task decomposition, reflection thinking, milestone recognition, etc. (4) Iterative Training with Reflective Online Traces, which addresses the data bottleneck by automatically collecting, filtering, and reflectively refining new interaction traces on hundreds of virtual machines. Through iterative training and reflection tuning, UI-TARS continuously learns from its mistakes and adapts to unforeseen situations with minimal human intervention. We also analyze the evolution path of GUI agents to guide the further development of this domain.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For GUI agents, procedural memory stores interface-specific action sequences, including menu traversal, widget manipulation, and error-recovery strategies under desktop or mobile environment constraints.
“For GUI agents, such procedural memory stores interface-specific action sequences, including menu traversal, widget manipulation, and error-recovery strategies under the constraints of desktop or mobile environments (Qin et al., 2025; Wang et al., 2025e).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Qin et al., 2025
Section 3.2 describes GUI-agent memory as retaining previous interactions and execution paths from prior tasks, and §4.3 (p. 16) discusses learning sequences of GUI actions across devices. The paper does not characterize this as procedural memory or say that memory stores the specified menu-traversal, widget-manipulation, and error-recovery strategies. (LLM, conf=0.85)
Experimental — accuracy not yet measuredNative GUI agents demonstrate proficiency in digital navigation but suffer from poor error recovery in stochastic environments, often failing when deviating from expert trajectories.
“they suffer from poor error recovery in stochastic environments, often failing when deviating from expert trajectories”
From OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards (2026) · cites this paper as Qin et al., 2025
Section 4.5 (p. 20) says agents can get stuck without error-correction capabilities and that training on idealized, error-free trajectories leaves them unable to learn recovery. It does not specifically establish frequent failure upon deviation from expert trajectories in stochastic environments; it also describes reflection tuning to improve UI-TARS’s recovery. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
Reasoning beyond limits: Advances and open problems for LLMs
Ferrag, M. A., Tihanyi, N., & Debbah, M. A. (2025) · ICT Express · doi:10.1016/j.icte.2025.09.003 · Cited by 33 · see citing works
Are Autonomous Web Agents Good Testers?
Chevrot, A., Vernotte, A., Falleri, J., Blanc, X., Legeard, B., & Cretin, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728879 · Cited by 11 · see citing works
Allouah, A., Besbes, O., Figueroa, J., Kanoria, Y., & Kumar, A. (2025) · doi:10.2139/ssrn.5381574 · Cited by 8 · see citing works
Attacking Vision-Language Computer Agents via Pop-ups
Zhang, Y., Yu, T., & Yang, D. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.411 · Cited by 4 · see citing works
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3 · see citing works
Self-citationQiu, J., Qi, X., Zhang, T., Juan, X., Guo, J., Lu, Y., Wang, Y., Yao, Z., Ren, Q., Jiang, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.20286 · Cited by 1 · see citing works
Self-citationSmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
“LLM-based agents [3, 19, 71, 74] have emerged as a dominant paradigm for solving complex, multi-step tasks, and are widely adopted in both research and production systems [4, 6, 13, 23, 55, 64, 73].”
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
Self-citationOS-Themis: A Scalable Critic Framework for Generalist GUI Rewards
Li, Z., Wu, Z., Zhao, Y., Yang, B., Xie, J., Liu, Z., Liu, Z., Jin, K., Liang, J., Li, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.19191
Self-citationAllouah, A., Besbes, O., Figueroa, J. D., Kanoria, Y., & Kumar, A. (2025) · arXiv
Memp: Exploring Agent Procedural Memory
Fang, R., Liang, Y., Wang, X., Wu, J., Qiao, S., Xie, P., Huang, F., Chen, H., & Zhang, N. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.866
Self-citationTestFlow: Advancing Mobile UI Testing through Multi-Step Reinforcement Learning
Tang, X., Chen, X., Chen, D., Zhou, S., Jiang, W., & Li, Y. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3732930
Self-citation"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent Collaboration
Kim, J., Cho, S., Nguyen, D. M., Jeon, J., Shin, M., & Ko, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791536
Xiong, Y., Su, T., Sun, J., Wang, J., Li, Q., Pu, G., & Su, Z. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832108 · Cited by 1 · see citing works
PropGen: Automated Property Generation for Property-Based Testing of Mobile Apps
Xiong, Y., Song, S., Ma, B., Su, T., & Xie, X. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837536
- Authors
- Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., Zhong, W., Li, K., Yang, J., Miao, Y., Lin, W., Liu, L., Jiang, X., Ma, Q., Li, J., Xiao, X., Cai, K., Li, C., Zheng, Y., Jin, C., Li, C., Zhou, X., Wang, M., Chen, H., Li, Z., Yang, H., Liu, H., Lin, F., Peng, T., Liu, X., & Shi, G.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2501.12326
- OpenAlex
- W4406755521
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 3 · 2025: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Engineering › Industrial and Manufacturing Engineering › Robotic Process Automation Applications
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- GUI agents, human-computer interaction, multi-step reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
