AppAgent: Multimodal Agents as Smartphone Users
Open accessAbstract
Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multimodal GenAI agents can operate smartphone applications, generating digital trace data while executing user requests.
“Preliminary work on multimodal GenAI agents [95] (preprint) has shown these agents can operate smartphone applications, generating digital trace data while executing user requests.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [95]
The full capabilities of LLMs as comprehensive autonomous trading systems remain underexplored, particularly in their ability to interpret multimodal data and utilize diverse tools.
“Although FinMEM [56] is an LLM trading agent with a humanaligned memory mechanism and character design, the full capabilities of LLMs as comprehensive autonomous trading systems remain underexplored, particularly in their ability to interpret multimodal data and utilize diverse tools.”
From A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist (2024) · cites this paper as [56]
The Introduction and §2.2 discuss multimodal LLM agents, visual interpretation, and tool use in smartphone and other non-trading tasks. Our copy does not address whether comprehensive autonomous trading systems remain underexplored; searches for trading, financial markets, and that specific research gap found no relevant passage. (LLM, conf=0.76)
Experimental — accuracy not yet measuredFinMEM is an LLM trading agent with a human-aligned memory mechanism and character design.
“Although FinMEM [56] is an LLM trading agent with a humanaligned memory mechanism and character design, the full capabilities of LLMs as comprehensive autonomous trading systems remain underexplored, particularly in their ability to interpret multimodal data and utilize diverse tools.”
From A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist (2024) · cites this paper as [56]
Section 2.2 discusses LLM agents generally, while Section 3.2 describes an agent that operates smartphone apps. Searches for FinMEM, trading, and the claimed memory mechanism and character design found no passage addressing the claim in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“Among open-source efforts from the industry and the research community, the LLaMA series [33, 34] are the most popular equivalents and have been fine-tuned to acquire conversational abilities, employing a decoder-only architecture similar to ChatGPT [32, 51].”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2× · see citing works
“Unlike earlier LLMs, these new models [32–34, 49, 51] enable multi-round conversations and have the impressive ability to follow complex instructions.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 2× · see citing works
“Koh et al. [19] introduce a best-first tree search for LLMs, enhancing their performance on complex tasks by improving success rates and reducing steps in interactive environments.”
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · see citing works
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1× · see citing works
“…like AutoGPT [45], HuggingGPT [30], and MetaGPT [15] illustrate this trend, and many projects demonstrate impressive capabilities, moving beyond basic language tasks to engaging in activities requiring higher cognitive functions, such as software development [6, 27] and gaming [10, 26, 43].”
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)Preprint
Yang, Z.-Y., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., & Wang, L.-J. (2023) · arXiv.org · doi:10.48550/arxiv.2309.17421 · Cited by 1,036 · cited 1× · see citing works
“Recent studies [44, 47] have shown that GPT-4V can understand various types of images, including simple user interfaces (UIs) in popular smartphone apps.”
Self-citationThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 1× · see citing works
“This includes performing tasks that require understanding context, making decisions, and learning from interactions [17, 40].”
Robot Programming by Demonstration
Denasi, A., Verhaar, B., Kostić, D., Bruijnen, D., Nijmeijer, H., & Warmerdam, T. (2010) · Simulation, Modeling, and Programming for Autonomous Robots · doi:10.1007/978-3-642-17319-6_28 · Cited by 945 · cited 1×
“PbD has been widely applied in robotics, enabling robots to perform complex tasks by replicating demonstrated actions [2, 5].”
RT-1: Robotics Transformer for Real-World Control at Scale
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.06817 · Cited by 727 · cited 2× · see citing works
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624 · see citing works
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1× · see citing works
“The related work of Wu et al. [39] introduces the Never-ending UI Learner, an autonomous app crawler that learns UI interaction tasks by programmatically tapping UI elements and observing the effects, avoiding the need for costly human labeling.”
Android in the Wild: A Large-Scale Dataset for Android Device ControlPreprint
Rawles, C., Li, A., Rodríguez, D., Riva, O., & Lillicrap, T. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2307.10088 · Cited by 415 · cited 1× · see citing works
“The Android in the Wild (AITW) [28] dataset contains 715,000 operation instances across 30,000 tasks.”
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual PerceptionPreprint
Wang, J.-Y., Xu, H., Ye, J., Yan, M.-S., Shen, W.-Z., Zhang, J., Huang, F., & Sang, J.-T. (2024) · arXiv.org · doi:10.48550/arxiv.2401.16158 · Cited by 372 · cited 1× · see citing works
“To gain deeper insights into the vision capabilities of our agent, we conducted an extensive case study using Adobe Lightroom, an image-editing application.”
GLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281 · see citing works
Self-citationRT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · cited 2× · see citing works
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. (2022) · Science · doi:10.1126/science.ade9097 · Cited by 241 · see citing works
You Only Look at Screens: Multimodal Chain-of-Action Agents
Zhang, Z., & Zhang, A. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2309.11436 · Cited by 222 · cited 1×
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
CogAgent: A Visual Language Model for GUI Agents
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01354 · Cited by 202 · cited 1× · see citing works
“For example, CogAgent [16] uses millions of instances from high-resolution GUI data and VQA datasets.”
Multimodal Web Navigation with Instruction-Finetuned Foundation ModelsPreprint
Furuta, H., Lee, K.-H., Nachum, O., Matsuo, Y., Faust, A., Gu, S., & Gur, I. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2305.11854 · Cited by 189 · cited 1× · see citing works
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
OpenAgents: An Open Platform for Language Agents in the WildPreprint
Xie, T., Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T., Zhao, J., Liu, Q., Liu, C., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.10634 · Cited by 175 · cited 1× · see citing works
“The related work of Wu et al. [39] introduces the Never-ending UI Learner, an autonomous app crawler that learns UI interaction tasks by programmatically tapping UI elements and observing the effects, avoiding the need for costly human labeling.”
Zeugmann, T., Poupart, P., Kennedy, J. G., Jin, X., Han, J.-W., Saitta, L., Sebag, M., Peters, J., Bagnell, J., Daelemans, W., et al. (2010) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_674 · Cited by 174 · cited 1×
“Our work is also related to Programming by Demonstration (PbD), a method where agents learn tasks by observing user demonstrations rather than explicit programming [7, 13].”
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI NavigationPreprint
Yan, A., Yang, Z.-Y., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y.-W., McAuley, J. J., Gao, J.-F., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.07562 · Cited by 168 · cited 1× · see citing works
“This includes performing tasks that require understanding context, making decisions, and learning from interactions [17, 40].”
Self-citationMind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134 · cited 1× · see citing works
“Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
Empowering LLM to use Smartphone for Intelligent Task AutomationPreprint
Wen, H., Li, Y.-C., Liu, G., Zhao, S.-H., Yu, T., Li, T. J.-J., Jiang, S.-Q., Liu, Y., Zhang, Y., & Liu, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2308.15272 · Cited by 87 · cited 1× · see citing works
“LLM-based agents are designed to utilize the advanced language and reasoning skills of LLMs to interact with and manipulate their environment [12, 23, 38, 42].”
Self-citationChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.07924 · Cited by 81 · see citing works
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · see citing works
Self-citationOSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.07972 · Cited by 64 · cited 1× · see citing works
“Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.06175 · Cited by 63 · see citing works
Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and PlanningPreprint
Hu, Z., & Shu, T.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2312.05230 · Cited by 58 · cited 1× · see citing works
“The related work of Wu et al. [39] introduces the Never-ending UI Learner, an autonomous app crawler that learns UI interaction tasks by programmatically tapping UI elements and observing the effects, avoiding the need for costly human labeling.”
Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
Yang, H., Yue, S., & He, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.02224 · Cited by 44 · cited 1× · see citing works
“Initiatives like AutoGPT [45], HuggingGPT [30], and MetaGPT [15] illustrate this trend, and many projects demonstrate impressive capabilities, moving beyond basic language tasks to engaging in activities requiring higher cognitive functions, such as software development [6, 27] and gaming [10, 26,…”
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue DataPreprint
Li, Y., Zhang, C., Yu, G., Wang, Z.-B., Fu, B., Lin, G., Shen, C.-H., Chen, L., & Wei, Y.-C. (2023) · arXiv.org · doi:10.48550/arxiv.2308.10253 · Cited by 37 · cited 1× · see citing works
“Building upon LLaMA, many multimodal LLMs, such as LLaVA [21, 22], ChartLlama [14], and StableLLaVA [20], also demonstrate vision understanding capabilities akin to those of GPT-4V.”
Self-citation<b>JARVIS</b> -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511593 · Cited by 36 · cited 1× · see citing works
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf
Xu, Y., Wang, S., Li, P., Luo, F., Wang, X., Liu, W., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2309.04658 · Cited by 32 · cited 1× · see citing works
“In our approach, the agent learns from human demonstrations or its own exploration, aligning with the objective of PbD by enabling task learning through examples, thus simplifying task automation.”
Never-ending Learning of User Interfaces
Wu, J., Krosnick, R., Schoop, E., Swearngin, A., Bigham, J., & Nichols, J. (2023) · ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606824 · Cited by 32 · see citing works
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31 · cited 1× · see citing works
“Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., & Wang, Y.-X. (2023) · arXiv · doi:10.48550/arxiv.2310.04406 · Cited by 18 · cited 1× · see citing works
“Language Agent Tree Search [52] combines Monte Carlo Tree Search with LLMs to improve multi-step reasoning and planning for autonomous agents, achieving state-of-the-art results in tasks like programming and web navigation.”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Gur, I., Furuta, H., Huang, A., Safdari, M., Matsuo, Y., Eck, D., & Faust, A. (2023) · arXiv · doi:10.48550/arxiv.2307.12856 · Cited by 17 · cited 1× · see citing works
“The related work of Wu et al. [39] introduces the Never-ending UI Learner, an autonomous app crawler that learns UI interaction tasks by programmatically tapping UI elements and observing the effects, avoiding the need for costly human labeling.”
3D-GPT: Procedural 3D Modeling with Large Language ModelsPreprint
Sun, C., Han, J., Deng, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12945 · Cited by 12 · see citing works
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Del Verme, M., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07718 · Cited by 3 · cited 1× · see citing works
“Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
Tree Search for Language Model Agents
Koh, J. Y., McAleer, S., Fried, D., & Salakhutdinov, R. (2024) · arXiv · doi:10.48550/arxiv.2407.01476 · Cited by 2 · cited 1× · see citing works
“Koh et al. [19] introduce a best-first tree search for LLMs, enhancing their performance on complex tasks by improving success rates and reducing steps in interactive environments.”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1 · cited 1× · see citing works
“Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
Flagged first, then by citations
11 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [17]17. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems.
- [29]29. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023c. The dawn of lmms: Preliminary explo- rations with gpt-4v(ision). arXiv preprint arXiv: 2309.17421. · arXiv
Cited by
Sorted by citations
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
“5 The applications (left) and evaluation strategies (right) of LLM-based agents. but are highly related to this area.”
Self-citationA survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893 · see citing works
“The image is from [16].”
Promises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499 · see citing works
“Diffusion models have shown promise in generating high-quality, realistic data and are used in applications such as image synthesis, text generation and other creative tasks.”
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., & Chen, T. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02496 · Cited by 92 · see citing works
“During this LLM carnival, researchers are also seeking generalized LLM solutions for various vision language tasks [37, 50, 62].”
BackgroundSelf-citationA Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671801 · Cited by 74 · see citing works
“Notably, FinAgent is the first advanced multimodal foundation agent designed for financial trading tasks.”
LLM-Based Framework for Administrative Task Automation in Healthcare
Gebreab, S. A., Salah, K., Jayaraman, R., Rehman, M. H. U., & Ellaham, S. (2024) · International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs60797.2024.10527275 · Cited by 56 · see citing works
“Other agents can also utilize the advanced language and reasoning skills of LLMs to interact with and manipulate their environment, including graphical user interfaces [20], [21].”
BackgroundVisualWebArena: evaluating multimodal agents on realistic visual web tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.50 · Cited by 50 · see citing works
Wang, R., Gao, Z., Zhang, L., Yue, S., & Gao, Z. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100755 · Cited by 51 · see citing works
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
Li, K., Meng, Z., Lin, H., Luo, Z., Tian, Y., Ma, J., Huang, Z., & Chua, T.-S. (2025) · Proceedings of the 33rd ACM International Conference on Multimedia · doi:10.1145/3746027.3755688 · Cited by 27 · see citing works
Self-citationAgentBoard: An analytical evaluation board of multi-turn LLM agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2365 · Cited by 26 · see citing works
LLMDroid: Enhancing Automated Mobile App GUI Testing Coverage with Large Language Model Guidance
Wang, C., Liu, T., Zhao, Y., Yang, M., & Wang, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715763 · Cited by 22 · see citing works
“Industry leaders such as Tencent with AppAgent [49] and MM-Navigator [47] with Microsoft’s participation have already made pioneering attempts in applying multimodal large models for app testing.”
Yang, M., Tao, Y., Cai, H., & Hu, B. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3397403 · Cited by 17 · see citing works
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · arXiv · doi:10.48550/arxiv.2401.13178 · Cited by 26 · see citing works
“(Zheng et al., 2024; Yang et al., 2023).”
Human-Centered LLM-Agent User Interface: A Position Paper
Chin, D., Wang, Y., & Xia, G. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-97254-6_18 · Cited by 5 · see citing works
An Evaluation Mechanism of LLM-based Agents on Manipulating APIs
Liu, B., Jianxiang, Z., Meng, D., & Lu, H. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.267 · Cited by 4 · see citing works
“We divide these tools into two categories: API-based tools and UI-based APPs (e.g. websites, desktop software and mobile APPs), as per the different challenges they pose to the LLMs.”
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4 · see citing works
Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
Fan, Z., Wang, Y., Polisetty, S., & Fung, Y. R. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1037 · Cited by 4 · see citing works
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., & Balasubramanian, N. (2024) · arXiv · doi:10.48550/arxiv.2407.18901 · Cited by 2 · see citing works
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1 · see citing works
“Several recent work have also explored the use of strong VLMs for building visual agents for mobile platforms (Zhan & Zhang, 2023; Chu et al., 2023; Yang et al., 2023b) as well as for the web (Gur et al., 2023; Hong et al., 2023).”
BackgroundThe AI Hippocampus: How Far are We From Human Memory?
Jia, Z., Li, J., Kang, Y., Wang, Y., Wu, T., Wang, Q., Wang, X., Zhang, S., Shen, J., Li, Q., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.09113
UI-Hawk: Unleashing the Screen Stream Understanding for GUI AgentsPreprint
Zhang, J., Yu, Y., Liao, M., Li, W., Wu, J., & Wei, Z. (2024) · Preprints.org · doi:10.20944/preprints202408.2137.v1
Mobile App Analysis in the New Era: Challenges and the Road Ahead
Wu, J., Liu, T., Zhao, Y., & Wang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3819579
“Modern development practices have introduced new architectural patterns (e.g., cross-platform frameworks such as React Native [26, 70] and Flutter [12]) and execution paradigms (e.g., super-app ecosystems [184, 199] and agent-driven interactions [204, 235]) that create novel forms of complexity.”
- Authors
- Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2312.13771
- OpenAlex
- W4390136788
- Cited by
- 11
- References
- 53
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- multimodal agents, LLM agents, human demonstration learning, knowledge base, image editing, social media applications, shopping apps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
