OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Open accessThe DOI matches its DataCite record; 62 of the 62 references it cites were found in a registry or an index; 19 of the 26 sources its claims cite were checked for support.
Abstract
Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.
Overview
CoChat Summary
TL;DR
OSWorld introduces a scalable, real computer environment and benchmark for multimodal agents, with 369 Ubuntu tasks (plus 43 Windows tasks) spanning web, desktop, OS file I/O, and multi-app workflows. It shows that current LLM/VLM agents are far from human performance, with the best model reaching 12.24% success versus 72.36% for humans.
Key findings
- The benchmark contains 369 Ubuntu tasks: 268 single-app (72.6%), 101 multi-app workflow (27.4%), 30 infeasible (8.1%), and 84 integrated from other benchmarks (22.8%), with 302 initial states and 134 unique execution-based evaluation functions.
- Humans complete 72.36% of tasks overall, while the best model achieves only 12.24%; for workflow tasks involving multiple apps, the highest baseline performance is 6.57%.
- Human median completion time on OSWorld is 111.94 seconds versus 35.38 seconds on WebArena, and human accuracy is 72.36% versus 88% on WebArena.
- Agents struggle primarily with GUI grounding and operational knowledge; among 550 failed examples, more than 75% involved mouse click inaccuracies, and higher screenshot resolution generally improves performance while more a11y-tree history helps SoM but screenshot-only history does not.
Methods
- OSWorld is a VM-based executable environment supporting Ubuntu, Windows, and macOS, with config-driven initial state setup, execution-based evaluation, observations including screenshots, accessibility trees, and terminal output, and a pyautogui action space plus WAIT, FAIL, and DONE.
- The benchmark was built from real-world computer-use sources and annotated by 9 computer science students over 3 months, consuming approximately 1800 man-hours, with each task including a natural-language instruction, setup configuration, and custom evaluation script.
- Evaluation included human performance studies and LLM/VLM baselines (GPT-4V, GPT-4, GPT-4o, Gemini, Claude-3 Opus, Qwen-Max, Mixtral, Llama-3, CogAgent) under four input settings: accessibility tree, screenshot, screenshot + accessibility tree, and Set-of-Mark, with a max step limit of 15 and temperature 1.0, top-p 0.9.
Limitations
- The paper states that further red teaming could reduce false positives and negatives, that a11y tree quality varies across applications and may be missing or noisy, and that there is no reliable metric to assess agent safety or latent side effects; evaluation focuses on task correctness rather than unnecessary damaging actions.
- The paper states that Windows tasks require further activation by the user due to copyright issues, and that macOS benchmark development was avoided due to copyright concerns, leaving the main benchmark on Ubuntu with only 43 Windows tasks for analysis.
- Evident from the text: baseline evaluation is capped at 15 steps and 30 minutes, which may constrain long-horizon tasks, and the Windows set is small and analytic rather than a full benchmark.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
4 Valid · 7 Not valid · 2 Not checked · 147 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“which contradicts the widely shown effectiveness of SoM in classic image understanding tasks [56], as well as in application areas like web agents [65, 16].”
What the sources say
- Supportedcites [56] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. source
The Introduction reports that SoM significantly improves GPT-4V on a wide range of visual tasks requiring grounding, and the Conclusion describes results across fine-grained vision tasks. The cited paper supports the image-understanding part attached to this citation; the decline is the citing authors’ finding, and the web-agent part has separate citations. (LLM, conf=0.91)
- Disputedcites [65] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024. sourceExperimental — accuracy not yet measured
The Abstract says that set-of-mark prompting is not effective for web agents, contrary to the claim that this source shows its effectiveness in that application. The paper distinguishes web agents from object- or scene-centric image tasks, where it says similar visual marking has been effective. (LLM, conf=0.95)
Checked by a reasoning model — may be wrong.
- Supportedcites [16] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024. source
Our copy of WebVoyager is exactly such an application: the Introduction states it is 'Inspired by Set-of-Mark Prompting (Yang et al., 2023a), we mark interactive web elements on screenshots ... to facilitate decision-making', and §3.3 (Observation Space) says the overlaid bounding boxes 'better guide the agent's action prediction' and that 'The numerical labels assigned to each element are also essential for the model to identify the elements requiring interaction, thereby facilitating accurate action determination' (with the design observation that a single black colour 'yields higher success rates than using multiple colors'). The paper's GPT-4V-backed agent reaches 57.1%/57.9% overall success versus 40.1% for the text-only baseline (Tables 1/3, §5.2), so the copy treats SoM-style marking as effective in a web agent. Caveat: WebVoyager adapts SoM without an object-detection module (GPT-4V-Act) and never isolates the marks against an unmarked-screenshot condition, so the support is an exemplifying application rather than a controlled SoM ablation. (LLM, conf=0.80)
boostedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Our benchmark has a total of 134 unique evaluation functions, which are orders of magnitude larger than prior work [66], showcasing the complexity, diversity, and evaluation challenges of tasks in our benchmark.”
What the sources say
- Could not verifycites [66] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. sourceExperimental — accuracy not yet measured
Section 3.2 names several WebArena scoring functions, and the section on site-navigation and content tasks describes task-dependent reward functions. Our copy does not give a total number of unique evaluation functions for WebArena, so it cannot establish the claimed orders-of-magnitude comparison with OSWorld’s 134 (searched: unique evaluation functions, 134, programmatic evaluation). (LLM, conf=0.71)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“we also integrate 84 examples from other benchmarks focusing on single-application or domain-specific environments such as NL2Bash [29], Mind2Web [9], SheetCopilot [25], PPTC [14], and GAIA [36].”
What the sources say
- Supportedcites [29] Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. arXiv preprint arXiv:1802.08979, 2018. source
The Introduction identifies NL2Bash as a dataset of natural-language descriptions paired with Bash commands for controlling the operating system, supporting its use as an example of a domain-focused benchmark. The 84 examples are described as the citing authors’ own integration, not as a count reported by NL2Bash. (LLM, conf=0.92)
- Partly supportedcites [9] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023. sourceExperimental — accuracy not yet measured
The Introduction identifies Mind2Web as a benchmark with over 2,000 tasks across 137 websites and 31 domains, so the citation supports identifying it as a benchmark. It does not support describing Mind2Web as focused on a single application or domain-specific environment; the paper emphasizes its diversity. The 84 examples are the citing authors’ own integration, not a number established by this source. (LLM, conf=0.86)
- Supportedcites [25] Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. Sheetcopilot: Bring- ing software productivity to the next level through large language models. arXiv preprint arXiv:2305.19308, 2023.
The citation is used to identify SheetCopilot as one of the benchmarks targeting a single-application / domain-specific environment, and our copy is exactly that: the Abstract (p. 1) and Sec. 3 (Dataset and Evaluation, p. 3) present a curated benchmark of 221 spreadsheet-control tasks with a fully automated evaluation pipeline, and Sec. 5.1 (Benchmark Protocol, p. 7) details its metrics (Exec@1, Pass@1, A50/A90). Sec. 2 (Related Works, p. 2) likewise frames the work as LLM control of one application (spreadsheets), alongside e.g. PowerPoint. The '84 examples' figure belongs to the citing paper's own pooling and is not attributed to this source, so nothing here overreaches. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous benchmarks provide datasets of demonstrations without executable environments [9, 40, 21].”
What the sources say
- Partly supportedcites [9] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023. sourceExperimental — accuracy not yet measured
The Introduction describes MIND2WEB as a dataset of manually annotated action sequences, supporting the demonstrations part of the claim. But the Human-Agent Interaction section says it evaluates agents in cached offline environments and includes network traffic that could enable some replay and exploration. The paper therefore supports a lack of fully interactive live evaluation, not the unqualified assertion that it has no executable environment. (LLM, conf=0.82)
- Partly supportedcites [40] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088, 2023. sourceExperimental — accuracy not yet measured
The Abstract and §2.1 describe AITW and earlier device-control datasets as collections of demonstrations, and §5.2 describes an offline evaluation method. The paper does not establish that these benchmarks lack executable environments; §3 says AITW demonstrations were collected using AndroidEnv and an Android emulator. No page numbers are available in our copy. (LLM, conf=0.79)
- Partly supportedcites [21] Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553, 2024. sourceExperimental — accuracy not yet measured
Section 3 describes OmniACT as a dataset of tasks, screenshots, and ground-truth action scripts, supporting the demonstration-dataset part of the claim. Sections 4.1–4.2 describe scoring predicted actions against those scripts, but our copy does not establish that the benchmark lacks an executable environment. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance of these experiments ranges from 0.99% to 12.24%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 160 claims and to verify the 4 references still open.
- It may state two separate claims as one. 59
- Probably not a substantial claim. 44
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Recent advances [266-268] use LLMs as the core reasoning component within autonomous agents-systems that perceive environments, make decisions, and execute tasks through reasoning and interaction.”
From A Survey of Large Language Models (2026) · cites this paper as [267]
The cited source says what the claim attributes to it.
In the citing paper
“Benchmarks like SWE-bench [332], which tests the resolution of real-world software issues, and OSWorld [267], which evaluates performance within operating system environments, are representative of this direction.”
From A Survey of Large Language Models (2026) · cites this paper as [267]
The cited source says what the claim attributes to it.
In the citing paper
“OSWorld [49] broadens computer-use evaluation to desktop and operating-system tasks under VM-style infrastructure with GUI grounding”
From MemGym: a Long-Horizon Memory Environment for LLM Agents (2026) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“AI agent benchmarks have also proliferated (Zhou et al., 2024; Xie et al., 2024; Abhyankar et al., 2025; Deng et al., 2023; Zhang et al., 2024; Huang et al., 2024; Zhu et al., 2025b).”
From Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (2025) · cites this paper as Xie et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“In contrast, professional applications remain largely unexplored, with only a few works featuring specialized tasks such as coding in VSCode (Xie et al., 2024).”
From ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use (2025) · cites this paper as Xie et al., 2024
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Exec.-based
“Infeasible 8.13% 16.67% Feasible 91.87% 13.34%”
§ Exec.-based
“GPT-4V 8.33% 8.55% 22.84% 14.28% 6.57% 11.77%”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Exec.-based
“The first use removes the subtitles embedded in the original video, and the second use saves the extracted subtitles locally.”
§ Exec.-based
“Step 4: pyautogui.typewrite('ffmpeg -i video.mp4 -map 0:s:0 subtitles.srt', interval=0.5) Step 5: pyautogui.typewrite('ffmpeg -i video.mp4 -c copy -sn no_subtitles_video.mp4', interval=0.5)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Excel Word PPT Workflow Overall Examples 11 9 7 16 43 Avg. Inst. Tokens 19.45 21.44 21.86 47.57 32.48”
Reported 32.48 · recomputed 30.722093
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“for workflow tasks that involve cooperation from multiple apps, the highest performance of the baseline agent is only 6.57%.”
§ Exec.-based
“GPT-4o 20.83% 6.99% 16.81% 16.33% 7.56% 11.36%”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Exec.-based
“CogAgent 4.17% 0.85% 2.71% 0.62% 0.09% 1.32%”
§ Focus the Terminal window and type the command
“Screen CogAgent 4.17 2.17 0.00 4.35 6.53 0.00 2.17 0.00 0.00 0.10”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 2
- [44]Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017.
- [67]Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. https://doi.org/10.48550/arXiv.2310.10634 https://doi.org/10.48550/arXiv.2310.10634 · DOI · Paper page
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
AppAgent: Multimodal Agents as Smartphone Users
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713600 · Cited by 652
“Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Wu, Z., Wu, Z., Xu, F., Wang, Y., Sun, Q., Jia, C., Cheng, K., Ding, Z., Chen, L., Liang, P. P., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.23218 · Cited by 410
SWE-smith: Scaling Data for Software Engineering Agents
Yang, J., Lieret, K., Jimenez, C., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., & Yang, D. (2025) · neural information processing systems · doi:10.52202/085713-3239 · Cited by 430
Exploring Large Language Model‐Based Intelligent Agents: Definitions, Methods, and Prospects
Cheng, Y., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Wang, Z., Wang, Z., Yin, F., Zhao, J., et al. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70111 · Cited by 339
Metadata
- Authors
- Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 52040-52094
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2404.07972
- OpenAlex
- W4415797693
- Semantic Scholar
- ff3e4f7c2481fb6df539f02be5945235101cbc19
- Cited by
- 64
- References
- 0
- Citations by year
- 2026: 53 · 2025: 8 · 2024: 3
- FWCI
- 13.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- multimodal agents, benchmarking, open-ended tasks, agent evaluation, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. https://app.cochat.ai/papers/10.48550/arxiv.2404.07972
- The paper
- Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv. https://doi.org/10.48550/arxiv.2404.07972
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.