SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
Open accessAbstract
Can large language model agents develop industry-level mobile applications? We introduce SWE-Bench Mobile, a benchmark for evaluating coding agents on realistic software engineering tasks derived from a production iOS codebase. Unlike existing benchmarks that focus on isolated problems or bug fixes, SWE-Bench Mobile captures the full complexity of industrial development: multi-modal inputs (PRDs and Figma designs), a large-scale mixed Swift/Objective-C codebase, and comprehensive test suites. We evaluate 22 agent-model configurations across four coding agents: three commercial (Cursor, Codex, Claude Code) and one open-source (OpenCode), and find that even the best configurations achieve only 12% task success rate. Our analysis reveals that (1) agent design matters as much as model capability: the same model shows up to 6x performance gap across agents, (2) commercial agents consistently outperform open-source alternatives, and (3) simple ''Defensive Programming'' prompts raise the test pass rate by 7.4 points over the baseline, while overly complex prompts hurt task success. These findings highlight a significant gap between current agent capabilities and industrial requirements, while providing actionable insights for practitioners and researchers. We release SWE-Bench Mobile as a hosted benchmark challenge to prevent data contamination and ensure fair evaluation. The public leaderboard is available at https://swebenchmobile.com.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Changing the harness around a fixed large language model can produce a 6× performance gap on the same benchmark.
“Changing the harness around a fixed large language model (LLM) can produce a 6 × performance gap on the same benchmark [47].”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [47]
The harness often matters as much as the model itself.
“The harness -the code that determines what to store, retrieve, and show to the model-often matters as much as the model itself.”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [47]
Cites
Zhang, K., Li, J., Li, G., Shi, X., & Jin, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.737 · Cited by 138 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Meta-Harness: End-to-End Optimization of Model Harnesses
Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026) · arXiv · doi:10.48550/arxiv.2603.28052 · Cited by 3 · see citing works
“Changing the harness around a fixed large language model (LLM) can produce a 6 × performance gap on the same benchmark [47].”
- Authors
- Tian, M., Wang, Z., Tang, Z., Yang, B., Zhu, K., Dong, H., Li, H., Xie, X., Wang, G., & You, J.
- Affiliations
- University of Toronto (CA)
- Xiaohongshu (Little Red Book) (China) (CN)
- Coolwei AI Lab, Beijing, China
- University of Illinois Urbana-Champaign (US)
- University of California, Berkeley (US)
- ORCID
- Venue
- Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
- Publisher
- ACM
- Published
- 2026-08-08
- Pages
- 8077-8087
- OpenAlex
- W7196930851
- Cited by
- 1
- References
- 1
- Citations by year
- 2026: 1
- FWCI
- 10.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- LLM agents, mobile application development, prompt engineering, multimodal input, requirements specification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
