CoChat

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Open access

The DOI matches its DataCite record; 62 of the 62 references it cites were found in a registry or an index; 19 of the 26 sources its claims cite were checked for support.

Abstract

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.

Overview

CoChat Summary

TL;DR

OSWorld introduces a scalable, real computer environment and benchmark for multimodal agents, with 369 Ubuntu tasks (plus 43 Windows tasks) spanning web, desktop, OS file I/O, and multi-app workflows. It shows that current LLM/VLM agents are far from human performance, with the best model reaching 12.24% success versus 72.36% for humans.

Key findings

  • The benchmark contains 369 Ubuntu tasks: 268 single-app (72.6%), 101 multi-app workflow (27.4%), 30 infeasible (8.1%), and 84 integrated from other benchmarks (22.8%), with 302 initial states and 134 unique execution-based evaluation functions.
  • Humans complete 72.36% of tasks overall, while the best model achieves only 12.24%; for workflow tasks involving multiple apps, the highest baseline performance is 6.57%.
  • Human median completion time on OSWorld is 111.94 seconds versus 35.38 seconds on WebArena, and human accuracy is 72.36% versus 88% on WebArena.
  • Agents struggle primarily with GUI grounding and operational knowledge; among 550 failed examples, more than 75% involved mouse click inaccuracies, and higher screenshot resolution generally improves performance while more a11y-tree history helps SoM but screenshot-only history does not.

Methods

  • OSWorld is a VM-based executable environment supporting Ubuntu, Windows, and macOS, with config-driven initial state setup, execution-based evaluation, observations including screenshots, accessibility trees, and terminal output, and a pyautogui action space plus WAIT, FAIL, and DONE.
  • The benchmark was built from real-world computer-use sources and annotated by 9 computer science students over 3 months, consuming approximately 1800 man-hours, with each task including a natural-language instruction, setup configuration, and custom evaluation script.
  • Evaluation included human performance studies and LLM/VLM baselines (GPT-4V, GPT-4, GPT-4o, Gemini, Claude-3 Opus, Qwen-Max, Mixtral, Llama-3, CogAgent) under four input settings: accessibility tree, screenshot, screenshot + accessibility tree, and Set-of-Mark, with a max step limit of 15 and temperature 1.0, top-p 0.9.

Limitations

  • The paper states that further red teaming could reduce false positives and negatives, that a11y tree quality varies across applications and may be missing or noisy, and that there is no reliable metric to assess agent safety or latent side effects; evaluation focuses on task correctness rather than unnecessary damaging actions.
  • The paper states that Windows tasks require further activation by the user due to copyright issues, and that macOS benchmark development was avoided due to copyright concerns, leaving the main benchmark on Ubuntu with only 43 Windows tasks for analysis.
  • Evident from the text: baseline evaluation is capped at 15 steps and 30 minutes, which may constrain long-horizon tasks, and the Windows set is small and analytic rather than a full benchmark.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.07972

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 11 : Figure 11: Raw XML formatted accessibility tree visualization.
Figure 11 : Figure 11: Raw XML formatted accessibility tree visualization.From the PDF
Figure 12 : Figure12: Use Accerciser to visualize components in accessibility tree. This example shows the corresponding relationship between the "Print" button (which is a printer icon on the bar of the LibreOffice app as shown on the left) and the "Print" tree node (which is the "Print" of the Accerciser Accessibility Explorer as shown on the right).
Figure 12 : Figure12: Use Accerciser to visualize components in accessibility tree. This example shows the corresponding relationship between the "Print" button (which is a printer icon on the bar of the LibreOffice app as shown on the left) and the "Print" tree node (which is the "Print" of the Accerciser Accessibility Explorer as shown on the right).From the PDF
Figure 13 : Figure 13: Comparison of instructions distribution. All datasets are sampled to 300 to make a fair comparison. The hyper-parameters of t-SNE are randomly sampled for each plot.
Figure 13 : Figure 13: Comparison of instructions distribution. All datasets are sampled to 300 to make a fair comparison. The hyper-parameters of t-SNE are randomly sampled for each plot.From the PDF
Figure 14 : Figure 14: Showcase of example screenshots marked by SoM across different app GUI.
Figure 14 : Figure 14: Showcase of example screenshots marked by SoM across different app GUI.From the PDF
Figure 16 : Figure 16: The figure shows examples of usage errors in three types of software. The first row is for GIMP, the second row is for LibreOffice Calc, and the third row is for Google Chrome. Detailed code execution processes are provided below.
Figure 16 : Figure 16: The figure shows examples of usage errors in three types of software. The first row is for GIMP, the second row is for LibreOffice Calc, and the third row is for Google Chrome. Detailed code execution processes are provided below.From the PDF
Figure 17 : Figure17: The error shown in the figure is due to mouse click inaccuracies. The agent was supposed to click on the product category images of the Nike official store and follow the instructions to search for women's jerseys priced over 60 dollars. However, due to a coordinate calculation error, it mistakenly clicked on the favorite button, creating a significant discrepancy between the current state and the target state. Under these circumstances, the agent is unable to backtrack to the previous state and start over.
Figure 17 : Figure17: The error shown in the figure is due to mouse click inaccuracies. The agent was supposed to click on the product category images of the Nike official store and follow the instructions to search for women's jerseys priced over 60 dollars. However, due to a coordinate calculation error, it mistakenly clicked on the favorite button, creating a significant discrepancy between the current state and the target state. Under these circumstances, the agent is unable to backtrack to the previous state and start over.From the PDF
Figure 19 : Figure19: Supplementary examples of tasks performed by humans and agents.
Figure 19 : Figure19: Supplementary examples of tasks performed by humans and agents.From the PDF
Figure 20 : Figure 20: Screenshots of error example analysis of Claude-3 Opus.
Figure 20 : Figure 20: Screenshots of error example analysis of Claude-3 Opus.From the PDF
(Figure 21 : Figure 21: A task showcase where the SoM-equipped agent tried to find interactive settings, while the screen-a11ytree-equipped agents directly modified the value in the setting.json file.
(Figure 21 : Figure 21: A task showcase where the SoM-equipped agent tried to find interactive settings, while the screen-a11ytree-equipped agents directly modified the value in the setting.json file.From the PDF

Verification

Claims the paper makes

4 Valid · 7 Not valid · 2 Not checked · 147 Own results and interpretations

65 more claims

Create a free account to see all 160 claims and to verify the 4 references still open.

See all claims — free account
  • It may state two separate claims as one. 59
  • Probably not a substantial claim. 44

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

6 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. [44]Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017.
  2. [67]Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. https://doi.org/10.48550/arXiv.2310.10634 https://doi.org/10.48550/arXiv.2310.10634 · DOI · Paper page

Cited by

See all 696

Metadata

Authors
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T.
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
52040-52094
ISSN
1049-5258 (ISSN-L)
OpenAlex
W4415797693
Cited by
64
References
0
Citations by year
2026: 53 · 2025: 8 · 2024: 3
FWCI
13.21
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
multimodal agents, benchmarking, open-ended tasks, agent evaluation, human-AI interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. https://app.cochat.ai/papers/10.48550/arxiv.2404.07972
The paper
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv. https://doi.org/10.48550/arxiv.2404.07972

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.