The DOI matches its DataCite record; 20 of the 20 references it cites were found in a registry or an index; 1 of the 17 sources its claims cite were checked for support.
Remote Labor Index: Measuring AI Automation of Remote Work
Open accessAbstract
AIs have made rapid progress on research-oriented benchmarks of knowledge and reasoning, but it remains unclear how these gains translate into economic value and automation. To measure this, we introduce the Remote Labor Index (RLI), a broadly multi-sector benchmark comprising real-world, economically valuable projects designed to evaluate end-to-end agent performance in practical settings. AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%. These results help ground discussions of AI automation in empirical evidence, setting a common basis for tracking AI impacts and enabling stakeholders to proactively navigate AI-driven labor automation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The Remote Labor Index (RLI) is a 240-project benchmark of real remote freelance work, each with a brief, input files, and a human gold-standard deliverable. Across six frontier AI agents, the best automation rate was 2.5%, showing current agents are far from autonomously completing economically valuable remote projects.
Key findings
- Best agent Manus achieved a 2.5% automation rate; other agents scored 2.1% (Grok 4, Sonnet 4.5), 1.7% (GPT-5 CLI), 1.3% (ChatGPT agent), and 0.8% (Gemini 2.5 Pro). All fell below the human baseline Elo of 1,000, with Manus highest at 509.9.
- Economic impact was near the floor: Manus earned $1,720 of a possible $143,991; average AI generation cost was $2.34, and models stopped before exceeding $30 in API costs.
- Failure modes among AI deliverables were poor quality (45.6%), incomplete work (35.7%), corrupted files (17.6%), and inconsistencies (14.8%); categories were not mutually exclusive.
- The dataset spans 240 projects across 23 Upwork subcategories, with mean human completion time 28.9 hours (median 11.5) and average cost $632.6 (median $200); total work exceeded 6,000 hours valued at over $140,000. Automation-rate inter-annotator agreement was 94.4%; Elo ternary agreement was 56.9% versus 33.0% chance.
Methods
- RLI comprises 240 end-to-end remote freelance projects sourced mainly from Upwork freelancers (358 recruited, 550 initial projects) plus long-tail and online work; each project includes a brief, input files, a human deliverable, and recorded cost/time.
- Six frontier agents were evaluated—ChatGPT agent, GPT-5, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, and Manus—using computer-use or OpenHands CLI scaffolds, with agents receiving the brief and inputs to produce deliverables.
- Evaluation was manual and holistic by trained evaluators on a web platform: automation rate was the percentage of projects rated 2 or 3 (accepted as commissioned work), and Elo came from pairwise comparisons with majority voting; human baseline Elo was fixed at 1,000.
Limitations
- The paper states RLI excludes work requiring client interaction, teamwork, physical labor, waiting to evaluate, or non-renderable deliverables, so it does not represent all remote work; an AI with 100% RLI automation may still underperform on unevaluated types.
- The paper states project costs are not inflation-adjusted and likely underestimate current economic value.
- Evident from the text: evaluation is manual and time-limited (20 minutes for automation rate, 30 for Elo; actual averages 11.4 and 17.4 minutes), Elo inter-annotator agreement is 56.9% (above chance but lower than automation rate), and the private test set is 230 projects with only 10 public.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 of 17 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Findingreasoning§ Example Projects from RLI · p. 2
RLI is grounded in actual economic transactions, encompassing the original work brief and the gold-standard deliverable produced by a human freelancer.
“Crucially, by sourcing the majority of projects from freelancing platforms, RLI is grounded in actual economic transactions, encompassing the original work brief and the gold-standard deliverable produced by a human freelancer.”
Quote in the paperSubstantial claim- Rests on an assumption: The human deliverables in RLI successfully complete their projects and are gold-standard.
- Findingown data§ Example Projects from RLI · p. 2
Models are steadily approaching higher automation rates across RLI projects.
“While all models fall well short of the aggregate human baseline, we observe that models are steadily approaching higher automation rates across projects.”
Quote in the paperNot a substantial claim - Findingown data§ 4.3 Qualitative Findings · p. 11
Across a small subset of projects, AI deliverables were judged comparable or better than human output.
“Across a small subset of projects, AI deliverables were judged comparable or better than human output.”
Quote in the paperNot a substantial claim- Rests on an assumption: The human reference deliverables represent the quality and level of work accepted by the original clients.
- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ A.3 Effect of Agent Scaffolds · p. 16
GPT-5 demonstrated superior performance when using a CLI-based agent compared to the Computer-Use Agent (CUA) setup.
“For instance, GPT-5 demonstrated superior performance when using a CLI-based agent compared to the Computer-Use Agent (CUA) setup.”
Quote in the paperNot a substantial claim - Findingown data§ 4.3 Qualitative Findings · p. 11
AI performed well on generating code for interactive data visualization.
“and on generating code for interactive data visualization.”
Quote in the paperNot a substantial claim- Rests on an assumption: The human reference deliverables represent the quality and level of work accepted by the original clients.
- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 3.2 Dataset Collection · p. 5
The study recruited 358 freelancers with verified Upwork accounts and specialization in the target categories.
“We recruited 358 freelancers with verified Upwork accounts and specialization in the target categories.”
Quote in the paperSubstantial claim - Findingreasoning§ 3.2 Dataset Collection · p. 5
Filtering the Upwork taxonomy from 64 categories resulted in 43 eligible categories.
“This filtering resulted in 43 eligible categories.”
Quote in the paperSubstantial claim - Findingown data§ 3.4 Evaluation · p. 9
Current AI agents solve less than 3% of tasks in RLI.
“Current AI agents perform near the floor on RLI, solving less than 3% of tasks in the benchmark.”
Quote in the paperSubstantial claim- Rests on an assumption: The human deliverables in RLI successfully complete their projects and are gold-standard.
- Rests on an assumption: A reasonable client standard is appropriate for judging whether AI deliverables complete commissioned work.
- Findingreasoning§ 4.2 Quantitative Results · p. 10
The metrics tracking economic impact of automation (Dollars Earned and Autoflation) are close to the floor.
“Correspondingly, the metrics tracking the economic impact of automation (Dollars Earned and Autoflation) are also close to the floor.”
Quote in the paperSubstantial claim - Findingown data§ 3.1 Dataset Description · p. 4
RLI covers 23 categories of work out of Upwork's 64.
“Following the collection and review process detailed in Section 3.2, our final dataset covers 23 categories of work out of Upwork's 64 .”
Quote in the paperNot a substantial claim - Findingown data§ 4.3 Qualitative Findings · p. 11
35.7% of AI deliverables were incomplete.
“| Incomplete | 35 . 7 |”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 3.1 Dataset Description · p. 4
RLI input files and deliverables span a wide variety of file types, substantially more than previous comparable benchmarks.
“In addition, the input files and deliverables in RLI span a wide variety of file types (Figure 14), substantially more than previous comparable benchmarks.”
Quote in the paperNot a substantial claim - Findingown data§ 3 Remote Labor Index · p. 4
The final RLI dataset comprises 240 projects.
“The final dataset comprises 240 projects.”
- Also stated in: § 3.2 Dataset Collection · p. 6 · plain
“The final RLI dataset contains 240 projects.”
Quote in the paperNot a substantial claim - Also stated in: § 3.2 Dataset Collection · p. 6 · plain
- Findingown data§ 4.2 Quantitative Results · p. 10
Manus achieved the highest automation rate on RLI at 2.5%.
“with the highest Automation Rate achieved being only 2 . 5% (Manus).”
Quote in the paperNot a substantial claim- Rests on an assumption: The human deliverables in RLI successfully complete their projects and are gold-standard.
- Rests on an assumption: A reasonable client standard is appropriate for judging whether AI deliverables complete commissioned work.
- Findingown data§ 4.2 Quantitative Results · p. 10
Newer frontier models generally achieve higher performance than older ones on RLI.
“The Elo rankings (Figure 8) indicate that models are steadily improving relative to each other, and the rankings generally reflect that newer frontier models achieve higher performance than older ones.”
Quote in the paperSubstantial claim - Findingown data§ 4.3 Qualitative Findings · p. 11
Across all models tested, performance matched or exceeded human baselines on several audio editing, mixing and production tasks.
“Specifically, across all models we tested, performance matched or exceeded human baselines on several audio editing, mixing and production tasks (e.g., creating bespoke sounds effects for a retro video game, separating vocals from accompaniment in a single track, merging voice-overs with intro and outro music)”
Quote in the paperNot a substantial claim- Rests on an assumption: The human reference deliverables represent the quality and level of work accepted by the original clients.
- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 3.1 Dataset Description · p. 4
Real freelance remote labor is far less concentrated in software/research/writing tasks than prior agent benchmarks.
“As Figure 6 shows, however, real freelance remote labor is far less concentrated in these activities.”
Quote in the paperSubstantial claim - Findingown data§ 4.3 Qualitative Findings · p. 11
AI performed well on report writing.
“AI also performed well on report writing”
Quote in the paperNot a substantial claim- Rests on an assumption: The human reference deliverables represent the quality and level of work accepted by the original clients.
- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 4.2 Quantitative Results · p. 10
Models are steadily improving relative to each other on RLI, as measured by Elo rankings.
“The Elo rankings (Figure 8) indicate that models are steadily improving relative to each other, and the rankings generally reflect that newer frontier models achieve higher performance than older ones.”
Quote in the paperSubstantial claim - Findingown data§ 6 Conclusion · p. 12
Frontier AI agents perform near the floor on RLI, achieving an automation rate of less than 3%.
“Frontier AI agents perform near the floor on RLI, achieving an automation rate of less than 3% , revealing a stark gap between progress on computer use evaluations and the ability to perform real and economically valuable work.”
Quote in the paperSubstantial claim- Rests on an assumption: The RLI project set is representative of the remote labor economy in general.
- Findingown data§ Example of Successful Project Completion · p. 12
In most cases where the project completion date is known, the projects were completed in the past five years.
“In most cases where we know the project completion date, the projects were completed in the past five years”
Quote in the paperSubstantial claim- Rests on an assumption: The reported project costs accurately reflect the economic value of the work at the time of completion.
- Findingown data§ B.5 Evaluation verification · p. 19
This gives a false negative rate of ≤5.8% with 95% confidence.
“This gives us ≤ 5 . 8% false negative rate with 95% confidence.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 4.3 Qualitative Findings · p. 11
14.8% of AI deliverables had inconsistencies.
“| Inconsistencies | 14 . 8 |”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 4.3 Qualitative Findings · p. 11
17.6% of AI deliverables exhibited corrupted files.
“| Corrupted files | 17 . 6 |”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingnone given§ Example Projects from RLI · p. 2
RLI is composed of entire projects sourced directly from online freelance platforms.
“RLI is composed of entire projects sourced directly from online freelance platforms, reflecting the diverse demands of the remote labor market.”
Quote in the paperSubstantial claim - Findingown data§ 3.1 Dataset Description · p. 4
The completion time of RLI projects matches the completion time of a random sample of jobs on Upwork.
“This matches the completion time of a random sample of jobs on Upwork, demonstrating how RLI comes closer than previous benchmarks to capturing the true complexity of remote labor markets.”
Quote in the paperSubstantial claim - Findingreasoning§ 4.3 Qualitative Findings · p. 11
Many failures stem from AI agents being unable to verify the correctness of their work and fix mistakes, especially in projects requiring complex and interactive audiovisual verification, such as architecture, game development, and web development.
“For example, many failures stem from AI agents being unable to verify the correctness of their work and fix mistakes, especially in projects requiring complex and interactive audiovisual verification, such as architecture, game development, and web development.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingnone given§ 3.1 Dataset Description · p. 4
RLI includes substantial coverage of design, operations, marketing, administration, data/BI, audio-video production, and other categories.
“RLI is designed for this broader reality: it includes substantial coverage of design, operations, marketing, administration, data/BI, audio-video production, and other categories, sampling across task complexity and deliverable types to reflect end-to-end freelance remote labor.”
Quote in the paperSubstantial claim - Findingown data§ B.5 Evaluation verification · p. 19
We found no false negatives in a manual evaluation of 50 random projects.
“We found no false negatives (cases where a annotators incorrectly labeled the human deliverable to be preferable).”
Quote in the paperSubstantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 4.3 Qualitative Findings · p. 11
45.6% of AI deliverables had poor quality.
“| Poor quality | 45 . 6 |”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Findingown data§ 4.2 Quantitative Results · p. 10
Current AI agents demonstrate minimal capability to perform the economically valuable projects in RLI.
“The central finding of our evaluation is that current AI agents demonstrate minimal capability to perform the economically valuable projects in RLI.”
Quote in the paperSubstantial claim- Rests on an assumption: The human deliverables in RLI successfully complete their projects and are gold-standard.
- Rests on an assumption: A reasonable client standard is appropriate for judging whether AI deliverables complete commissioned work.
- Findingown data§ A.1 Full Results · p. 16
Current AI agents earn a small fraction of the total cost of projects in the dataset.
“Current AI agents earn a small fraction of the total cost of projects in the dataset.”
Quote in the paperNot a substantial claim- Rests on an assumption: The reported project costs accurately reflect the economic value of the work at the time of completion.
- Findingown data§ 3.2 Dataset Collection · p. 6
RLI maintains a private test set of 230 projects used for quantitative evaluation.
“To protect PII and prevent benchmark contamination, we maintain a private test set of 230 projects used for quantitative evaluation.”
Quote in the paperSubstantial claim - Findingown data§ Examples of Unsuccessful Project Completion · p. 29
AI agents fail to successfully complete the vast majority of RLI projects.
“Agents fail to successfully complete the vast majority of RLI projects.”
Quote in the paperNot a substantial claim - Findingown data§ 4.3 Qualitative Findings · p. 11
Rejections in RLI evaluations predominantly cluster around technical/file integrity issues, incomplete/malformed deliverables, quality issues, and inconsistencies.
“Our qualitative analysis across roughly 400 evaluations shows that rejections predominantly cluster around the following primary categories of failure:”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Study design and conductreasoning§ C.5 Project cost and completion time. · p. 27
Self-reported costs provide an accurate measure of the cost of the project.
“Since these often represent the actual amount of money paid by a client, they provide an accurate measure of the cost of the project.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.2 Quantitative Results · p. 10
RLI is sensitive enough to detect ongoing progress in AI capabilities.
“This demonstrates that RLI is sensitive enough to detect ongoing progress in AI capabilities.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Example of Successful Project Completion · p. 12
The reported costs likely underestimate the current economic value of this work when accounting for inflation.
“consequently, the reported costs likely underestimate the current economic value of this work when accounting for inflation.”
Quote in the paperNot a substantial claim- Rests on an assumption: The reported project costs accurately reflect the economic value of the work at the time of completion.
- Interpretationreasoning§ 1 Introduction · p. 1
Performance on existing AI agent benchmarks offers limited insight into the trajectory of human labor automation.
“Consequently, performance on these benchmarks offers limited insight into the trajectory of human labor automation.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 1 Introduction · p. 1
Existing AI agent benchmarks often do not capture the vast diversity and complexity inherent in the broader landscape of remote work.
“These provide valuable signals of capabilities in isolation, yet they often do not capture the vast diversity and complexity inherent in the broader landscape of remote work.”
Quote in the paperSubstantial claim - Study design and conductnone given§ 3.4 Evaluation · p. 8
Inter-annotator agreement is above 94%.
“Inter-annotator agreement is above 94% .”
Quote in the paperNot a substantial claim - Interpretationown data§ Example Projects from RLI · p. 2
Contemporary AI systems are far from capable of autonomously performing the diverse demands of remote labor.
“This demonstrates that despite rapid progress on knowledge and reasoning benchmarks, contemporary AI systems are far from capable of autonomously performing the diverse demands of remote labor.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Example of Successful Project Completion · p. 12
AIs may prove qualitatively different from prior automation technologies.
“In this way, AIs may prove qualitatively different from prior automation technologies.”
Quote in the paperNot a substantial claim - Study design and conductown data§ 3.4 Evaluation · p. 9
The evaluation process demonstrates high reliability, with an inter-annotator agreement of 94.4% for the automation rate metric.
“The evaluation process demonstrates high reliability, with an inter-annotator agreement of 94 . 4% for the automation rate metric.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.3 Qualitative Findings · p. 11
Many successes of AI models lie in domains where current AI models' skills are more developed, such as projects where the complexity is primarily in text processing or image creation.
“Analogously, many of the successes of AI models lie in domains where current AI models' skills are more developed, such as projects where the complexity is primarily in text processing or image creation.”
Quote in the paperSubstantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Interpretationown data§ Example Projects from RLI · p. 2
The best-performing current AI agents fail to complete most RLI projects at a level that would be accepted as commissioned work in a realistic freelancing environment.
“The best-performing current AI agents achieve an automation rate of 2 . 5% , failing to complete most projects at a level that would be accepted as commissioned work in a realistic freelancing environment.”
Quote in the paperSubstantial claim- Rests on an assumption: A reasonable client standard is appropriate for judging whether AI deliverables complete commissioned work.
- Interpretationreasoning§ C.1 Categorization · p. 24
O*NET's limitation for RLI stems from its structure at both the activity and occupational levels.
“This limitation stems from O*NET's structure at both the activity and occupational levels.”
Quote in the paperSubstantial claim - Interpretationcitation§ 4.3 Qualitative Findings · p. 11
Many failures exhibited by AI agents stem from deficits in cognitive skills such as broad world knowledge, memory, and audiovisual abilities.
“We observe that many of the failures exhibited by AI agents stem from deficits in these skills.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
- Interpretationreasoning§ C.1 Categorization · p. 24
Even the most granular taxonomy in O*NET, Detailed Work Activities (DWAs), does not provide meaningful granularity for measuring task breadth.
“Even the most granular taxonomy in O*NET, Detailed Work Activities (DWAs), does not provide meaningful granularity for measuring task breadth.”
Quote in the paperSubstantial claim - Interpretationown data§ A.3 Effect of Agent Scaffolds · p. 16
Current models are not yet able to take full advantage of computer-use environments.
“Our results suggest that current models are not yet able to take full advantage of computer-use environments.”
Quote in the paperSubstantial claim - Study design and conductreasoning§ Example of Successful Project Completion · p. 12
RLI does not fully represent every part of the remote labor economy.
“While RLI does not fully represent every part of the remote labor economy, it is a substantial step towards measuring the ability of AI to automate the remote economy in general, rather than just current tasks.”
Quote in the paperSubstantial claim- Rests on an assumption: The RLI project set is representative of the remote labor economy in general.
- Study design and conductreasoning§ Example of Successful Project Completion · p. 12
RLI does not represent several types of remote work due to these constraints.
“While RLI is the broadest benchmark of its kind, it does not represent several types of remote work due to these constraints.”
Quote in the paperSubstantial claim - Interpretationreasoning§ C.1 Categorization · p. 24
O*NET is not tailored to end-to-end freelance labor markets like Upwork, making it unsuitable for classifying RLI projects and estimating coverage.
“While valuable for capturing activities performed in long-term occupations, it is not tailored to end-to-end freelance labor markets like Upwork, making it unsuitable for classifying RLI projects and estimating coverage.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Example of Successful Project Completion · p. 12
An AI obtaining 100% automation rate on RLI may still underperform humans on types of work that we do not evaluate.
“Thus, an AI obtaining 100% automation rate on RLI may still underperform humans on types of work that we do not evaluate.”
Quote in the paperSubstantial claim- Rests on an assumption: The RLI project set is representative of the remote labor economy in general.
- Interpretationown data§ 6 Conclusion · p. 12
There is a stark gap between progress on computer use evaluations and the ability to perform real and economically valuable work.
“Frontier AI agents perform near the floor on RLI, achieving an automation rate of less than 3% , revealing a stark gap between progress on computer use evaluations and the ability to perform real and economically valuable work.”
Quote in the paperSubstantial claim- Rests on an assumption: The RLI project set is representative of the remote labor economy in general.
- Interpretationreasoning§ C.1 Categorization · p. 24
O*NET's occupational-level structure does not align with the delivery of specific, self-contained freelance projects.
“This structure does not align with the delivery of specific, self-contained freelance projects.”
Quote in the paperSubstantial claim - Study design and conductown data§ 3.4 Evaluation · p. 9
The probability of hard disagreements in Elo evaluations is 5.9%.
“The probability of hard disagreements (one vote for AD1 and one vote for AD2 ) is 5 . 9% , indicating that evaluators are directionally nearly always in agreement.”
Quote in the paperNot a substantial claim - Premisecitation§ C.1 Categorization · p. 24
At the occupational level, O*NET classifications are designed to describe the broad, ongoing responsibilities of long-term workers.
“At the occupational level, O*NET classifications are designed to describe the broad, ongoing responsibilities of long-term workers.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [20] [20] National Center for O*NET Development. O*NET 30.0 database. https://www.onetcenter. org/database.html , 2025. Licensed CC BY 4.0. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation§ C.1 Categorization · p. 24
O*NET relies on an abstract hierarchy of Work Activities to categorize a broad range of work.
“To categorize a broad range of work, O*NET relies on an abstract hierarchy of Work Activities.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [20] [20] National Center for O*NET Development. O*NET 30.0 database. https://www.onetcenter. org/database.html , 2025. Licensed CC BY 4.0. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 49
- The quote only partly states this claim. 4
It may state two separate claims as one.
Premisecitation§ 1 Introduction · p. 1Many existing AI agent benchmarks measure performance on specialized skills such as software engineering and basic computer use, while some focus on simple tasks shared across several professions.
“Many existing AI agent benchmarks measure performance on specialized skills such as software engineering [13, 18, 26] and basic computer use [34, 7, 14, 17, 32], while some focus on simple tasks shared across several professions [23].”
Quote in the paperSubstantial claim1 of 9 sources checked- cites [13] [13] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [18] [18] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115 , 2025. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [26] [26] David Rein, Joel Becker, Amy Deng, Seraphina Nix, Chris Canal, Daniel O'Connel, Pip Arnott, Ryan Bloom, Thomas Broadley, Katharyn Garcia, et al. Hcast: Human-calibrated autonomy software tasks. arXiv preprint arXiv:2503.17354 , 2025. - source
HCAST includes simple general-reasoning tasks: the ‘questions’ task (p. 24) requires no experience, and the qualification procedure (Appendix E.2, p. 23) allows workers qualified in any domain to complete general-reasoning tasks. But HCAST also contains specialized software engineering, machine learning, and cybersecurity tasks (Task Desiderata, p. 4), so describing the benchmark as focused on simple tasks shared across professions overstates that part of its scope. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites [34] [34] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 , 2023. Table 3: Full automation rate and Elo results. In Appendix A.3, we describe our comparison of two agent scaffolds for GPT-5, a command-line interface (CLI) scaffold and computer-use (CUA) scaffold. In the main paper, we show GPT-5 with the CLI scaffold. Table 4: Current models earn a small fraction of the total cost of projects in the dataset.
- cites [7] [7] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems , 36:28091-28114, 2023. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [14] [14] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 , 2024. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [17] [17] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations , 2023. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [32] [32] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems , 37:52040-52094, 2024. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [23] [23] Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. arXiv preprint arXiv:2510.04374 , 2025. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ B.4 Evaluation Instructions · p. 19The final version of the instructions achieved 94.4% inter-annotator agreement.
“Our final version of the instructions achieved 94 . 4% inter-annotator agreement.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ C.1 Categorization · p. 24Music composition projects make up roughly 6% of the benchmark.
“Music composition projects make up roughly 6% of the benchmark.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ C.5 Project cost and completion time. · p. 27Project cost and completion time are roughly log-normal distributed.
“Both variables are roughly log-normal distributed, with project cost and completion time reaching up to $22 , 500 and 450 hours.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingnone given§ Example of Successful Project Completion · p. 28AI agents leverage image generation tools to solve some marketing projects in RLI.
“AI agents leverage image generation tools to solve some marketing projects in RLI.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 3.2 Dataset Collection · p. 6RLI releases a public set of 10 projects along with open-sourced evaluation platform code for qualitative evaluation.
“We release a public set of 10 projects along with the open-sourced code for the evaluation platform to enable qualitative evaluation.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 3.1 Dataset Description · p. 4The completion time for RLI projects exceeds previous benchmarks by more than 2×, with a mean of 28.9 hours and median of 11.5 hours.
“As shown in Figure 6, the completion time for RLI projects exceeds previous benchmarks by more than 2 × , with a mean of 28 . 9 hours and median of 11 . 5 hours.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ 3.2 Dataset Collection · p. 5Freelance Platform Sourcing yielded 207 projects.
“In total, this yielded 207 projects.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ C.5 Project cost and completion time. · p. 27Completion times are available for 84% of projects.
“Completion times are available for 84% of projects.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone given§ B.3 Evaluation and Generation Budgets · p. 18AI deliverables often have glaring errors that are easy to spot.
“For current AI agents, evaluations are possible to complete relatively quickly, because AI deliverables often have glaring errors that are easy to spot.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
It may state two separate claims as one.
Findingown data§ C.2 Filtering & Cleaning Criteria · p. 26After cleaning and filtering, the dataset contains 240 projects from 23 Upwork subcategories.
“After the cleaning and filtering process, the dataset contains 240 projects from the following 23 Upwork subcategories:”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
Findingown data§ Project Cost vs Completion Time · p. 27Project cost and completion time are highly correlated on a log-log scale, with a Pearson correlation of 0.785.
“Project cost and completion time are highly correlated on a log-log scale.”
- Also stated in: § C.5 Project cost and completion time. · p. 27 · plain
“We observe a Pearson correlation of 0 . 785 .”
Quote in the paperNot a substantial claim- Also stated in: § C.5 Project cost and completion time. · p. 27 · plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Findingown data§ B.3 Evaluation and Generation Budgets · p. 18Most evaluations finished in less than the allotted time, with a small number exceeding it.
“As shown in Figure 11, most evaluations finished in less than this amount of time, with a small number exceeding it.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ 3.2 Dataset Collection · p. 5Expanding beyond Upwork yielded an additional 33 projects.
“yielding an additional 33 projects.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingnone given§ C.2 Filtering & Cleaning Criteria · p. 25The majority of projects in RLI were paid for by clients.
“Note: The majority of projects in RLI were paid for by clients, so this is often guaranteed by default.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ 3.2 Dataset Collection · p. 5From these freelancers, the study collected 550 initial projects.
“From these freelancers, we collected 550 initial projects.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 3.1 Dataset Description · p. 5RLI spans a broad range of difficulty, with project costs reaching over $10,000 and completion times for human professionals reaching over 100 hours.
“RLI spans a broad range of difficulty, with project costs reaching over $10 , 000 and completion times for human professionals reaching over 100 hours.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ Abstract · p. 1AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%.
“AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2 . 5% .”
- Also stated in: § Example Projects from RLI · p. 2 · hedged
“The results indicate that performance on the benchmark is currently near the floor. The best-performing current AI agents achieve an automation rate of 2 . 5% , failing to complete most projects at a level that would be accepted as commissioned work in a realistic freelancing environment.”
- Also stated in: § 4.2 Quantitative Results · p. 10 · plain
“Across all models evaluated, absolute performance is near the floor, with the highest Automation Rate achieved being only 2 . 5% (Manus).”
Quote in the paperSubstantial claim- Rests on an assumption: The human deliverables in RLI successfully complete their projects and are gold-standard.
- Rests on an assumption: A reasonable client standard is appropriate for judging whether AI deliverables complete commissioned work.
- Also stated in: § Example Projects from RLI · p. 2 · hedged
The quote only partly states this claim.
Findingown data§ 4.3 Qualitative Findings · p. 11Across all models tested, performance matched or exceeded human baselines on image-generation tasks.
“and on image-generation tasks (e.g., ad and logo creation).”
Quote in the paperNot a substantial claim- Rests on an assumption: The human reference deliverables represent the quality and level of work accepted by the original clients.
- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
It may state two separate claims as one.
Findingown data§ 3.1 Dataset Description · p. 5The projects in RLI represent over 6,000 hours of real work valued at over $140,000.
“In total, the projects in RLI represent over 6 , 000 hours of real work valued at over $140 , 000 .”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingnone given§ · Data & Interactive : · p. 22Less than 20 projects have evaluator notes.
“Less than 20 projects have evaluator notes.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ C.5 Project cost and completion time. · p. 27Costs are available for 95% of projects.
“Costs are available for 95% of projects.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingreasoning§ C.2 Filtering & Cleaning Criteria · p. 25This left us with 45 total Upwork categories to source projects from.
“This left us with 45 total Upwork categories to source projects from.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown data§ C.1 Categorization · p. 23The final RLI dataset includes projects from 9 major Upwork categories and 23 subcategories.
“Our final dataset includes projects from 9 major categories and 23 subcategories.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 3.2 Dataset Collection · p. 5The recruited freelancers had on average 2,341 hours worked, 89 prior jobs, and $23,364 in total earnings on Upwork.
“on average, they had 2 , 341 hours worked, 89 prior jobs, and $23 , 364 in total earnings on Upwork.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ 4.1 Experimental Setup · p. 9For GPT-5, the CLI scaffold outperformed the CUA scaffold.
“For GPT-5, we evaluated both the CUA and CLI scaffolds and report the CLI scaffold in the main tables, as this outperformed the CUA scaffold for this model.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 3.1 Dataset Description · p. 4The average cost of projects in RLI is $632.6 with a median of $200.
“The average cost of projects in RLI is $632 . 6 with a median of $200 .”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ C.5 Project cost and completion time. · p. 275% of projects have neither cost nor completion time data.
“5% of projects have neither cost nor completion time data, but were kept in the dataset due to being high-quality.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 4.3 Qualitative Findings · p. 11Successful AI deliverables were predominantly creative projects, especially audio and image related work, along with writing and data retrieval/web scraping.
“These were predominantly creative projects, especially audio and image related work, along with writing and data retrieval/web scraping.”
Quote in the paperNot a substantial claim- Rests on an assumption: The human reference deliverables represent the quality and level of work accepted by the original clients.
- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ 3.2 Dataset Collection · p. 5Long-Tail Sourcing yielded 7 projects.
“In total, this yielded 7 projects.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ A.3 Effect of Agent Scaffolds · p. 16GPT-5's superior performance with CLI over CUA holds for both Elo scores (CLI: 436.7; CUA: 431.6) and automation rates (CLI: 1.7%; CUA: 0.8%).
“This holds for both the Elo scores (CLI: 436 . 7 ; CUA: 431 . 6 ) and the automation rates (CLI: 1 . 7% ; CUA: 0 . 8% ).”
Quote in the paperSubstantial claimProbably not a substantial claim.
Findingown data§ C.2 Filtering & Cleaning Criteria · p. 25Based on sourcing criteria, we entirely excluded projects from certain Upwork categories during initial project collection.
“Based on these criteria, we entirely excluded projects from the following categories on Upwork during our initial project collection:”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingnone given§ C.2 Filtering & Cleaning Criteria · p. 25Most legal categories were excluded due to PII concerns.
“Most legal categories were excluded due to PII concerns.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone given§ Example Projects from RLI · p. 2RLI projects exhibit significantly higher complexity than tasks found in existing agent benchmarks.
“These projects exhibit significantly higher complexity than tasks found in existing agent benchmarks.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown data§ 3.2 Dataset Collection · p. 5Freelancers were paid between $15 and $200 per project (average $41) to sell existing work samples.
“Freelancers were paid between $15 and $200 per project (average $41 ) to sell us existing work samples.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Findingown data§ C.5 Project cost and completion time. · p. 27Project cost and completion time reach up to $22,500 and 450 hours.
“Both variables are roughly log-normal distributed, with project cost and completion time reaching up to $22 , 500 and 450 hours.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Premisenone given§ Example of Successful Project Completion · p. 12Humans have general cognitive skills that calculators do not.
“This is because humans have general cognitive skills that calculators do not.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductnone given§ B.2 Elo Computation · p. 17We use 100 bootstrap samples to compute 95% confidence intervals in Figure 8.
“We use 100 bootstrap samples to compute 95% confidence intervals in Figure 8.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductnone given§ Project Cost vs Completion Time · p. 27For long-tail project collection, we either purchased the work or received permission from the original author to link to it.
“For long-tail project collection, we either purchased the work or received permission from the original author of the work to link to it in our study.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Interpretationnone given§ C.2 Filtering & Cleaning Criteria · p. 25Many projects in the Content Writing category can already be solved by AIs and would not provide much information to include.
“Many projects in the Content Writing category can already be solved by AIs and would not provide much information to include.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Study design and conductown data§ B.2 Elo Computation · p. 17Stratified sampling across models ensures each model pair is compared on at least 10 projects (median 25).
“We use stratified sampling across models to ensure each model pair is compared on at least 10 projects (median 25 ).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Study design and conductnone given§ 6 Conclusion · p. 12RLI establishes an economically grounded measure of AI automation capacity, with 240 projects spanning 23 domains of digital freelance work, each anchored in demonstrated market value.
“RLI establishes an economically grounded measure of AI automation capacity, with 240 projects spanning 23 domains of digital freelance work, each anchored in demonstrated market value.”
Quote in the paperSubstantial claim- Rests on an assumption: The RLI project set is representative of the remote labor economy in general.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductnone given§ B.2 Elo Computation · p. 17Majority voting uses two independent evaluations with a third to break ties if needed.
“For each project that a model pair is compared on, we perform majority voting, using two independent evaluations with a third to break ties if needed.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Premisereasoning§ 3.4 Evaluation · p. 7Evaluating RLI deliverables is beyond the capabilities of automated evaluation systems, so the paper relies on rigorous manual evaluation.
“As this level of assessment is currently beyond the capabilities of automated evaluation systems, we rely on rigorous manual evaluation.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Study design and conductown data§ C.3 Analysis Details · p. 26We analyzed 275 completed jobs from a random sample of 60 freelancers to determine average completion time and cost for Upwork projects.
“To determine the average completion time and cost for Upwork projects, we analyzed 275 completed jobs from a random sample of 60 freelancers, using the hours worked and dollars earned for each job.”
Quote in the paperSubstantial claim- Rests on an assumption: The random sample of 60 freelancers and 275 completed jobs is representative of Upwork projects.
It may state two separate claims as one.
Interpretationown data§ 3.1 Dataset Description · p. 4RLI is challenging and, in aggregate, more representative of contemporary remote freelance work than previous benchmarks.
“Taken together, these properties yield a benchmark that is challenging and, in aggregate, more representative of contemporary remote freelance work than previous benchmarks.”
Quote in the paperSubstantial claimProbably not a substantial claim. It may be too vague to check against evidence.
Interpretationreasoning§ Example of Successful Project Completion · p. 12RLI is a substantial step towards measuring the ability of AI to automate the remote economy in general, rather than just current tasks.
“While RLI does not fully represent every part of the remote labor economy, it is a substantial step towards measuring the ability of AI to automate the remote economy in general, rather than just current tasks.”
Quote in the paperNot a substantial claim- Rests on an assumption: The RLI project set is representative of the remote labor economy in general.
It may state two separate claims as one.
Study design and conductreasoning§ 3.4 Evaluation · p. 8All evaluations are performed manually by trained workers and subject experts.
“Thus, all evaluations are performed manually by trained workers and subject experts.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationcitation§ Example of Successful Project Completion · p. 12An AI that automates all current remote work without overfitting is likely to have many of the same general cognitive skills as humans, allowing it to automate new jobs as they arise [15].
“An AI that automates all current remote work without overfitting is likely to have many of the same general cognitive skills as humans, allowing it to automate new jobs as they arise [15].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [15] [15] Anton Korinek and Donghyun Suh. Scenarios for the transition to agi. Technical report, National Bureau of Economic Research, 2024. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
Premisecitation§ Example of Successful Project Completion · p. 12Current AIs are not task-specific, but rather have general cognitive skills and are already capturing a substantial fraction of human-level cognitive generality [12].
“Indeed, current AIs are not task-specific, but rather have general cognitive skills and are already capturing a substantial fraction of human-level cognitive generality [12].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12] [12] Dan Hendrycks, Dawn Song, Christian Szegedy, Honglak Lee, Yarin Gal, Sharon Li, Andy Zou, Lionel Levine, Bo Han, Jie Fu, Ziwei Liu, Jinwoo Shin, Kimin Lee, Mantas Mazeika, Long Phan, George Ingebretsen, Adam Khoja, Cihang Xie, Olawale Salaudeen, Matthias Hein, Kevin Zhao, Alex Pan, David Duvenaud, Bo Li, Steve Omohundro, Gabriel Alfour, Max Tegmark, Kevin McGrew, Gary Marcus, Jaan Tallinn, Eric Schmidt, and Yoshua Bengio. A definition of agi, 2025. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation§ C.1 Categorization · p. 24The O*NET database provides a widely used taxonomy of occupational requirements and work activities within the US labor market.
“The O*NET database [20] provides a widely used taxonomy of occupational requirements and work activities within the US labor market.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [20] [20] National Center for O*NET Development. O*NET 30.0 database. https://www.onetcenter. org/database.html , 2025. Licensed CC BY 4.0. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Interpretationown data§ C.3 Analysis Details · p. 26The distribution over Upwork taxonomy categories provides a distribution over types of work performed on Upwork, but not at the job level.
“This provides a distribution over the different types of work performed on Upwork. Note: This is not a distribution at the job-level, which is more skewed toward software tasks.”
Quote in the paperNot a substantial claim- Rests on an assumption: The judge LLM accurately classifies project briefs into the three categories.
It may state two separate claims as one.
Study design and conductown data§ 3.4 Evaluation · p. 9For Elo evaluations, ternary inter-annotator agreement is 56.9%, far above random chance of 33.0%.
“For Elo evaluations, ternary inter-annotator agreement is 56 . 9% , far above random chance of 33 . 0% .”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductnone given§ B.4 Evaluation Instructions · p. 18Annotators were required to review detailed instructional videos and documents covering the evaluation workflow and common pitfalls.
“Before beginning evaluations, annotators were required to review detailed instructional videos and documents covering the evaluation workflow and common pitfalls.”
Quote in the paperNot a substantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductown data§ 3.4 Evaluation · p. 9Evaluations took 11.4 minutes on average for Automation Rate and 17.4 minutes for Elo.
“Evaluations took 11 . 4 minutes on average for Automation Rate and 17 . 4 minutes for Elo.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductnone given§ C.4 Data Collection Details · p. 26We anonymized projects containing PII or copyrighted content by redacting information, sometimes replacing it with synthetic details.
“In cases where the work contains PII or copyrighted content (e.g., logos or company names), we anonymized the project by redacting information. In some cases, redacted information was replaced with synthetic details (e.g., fake company names or logos).”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Study design and conductnone given§ 3.3 Metrics · p. 7The automation rate and Elo metrics are fully compatible, in that automation rate equals the probability of a win or tie against the human baseline under the same standards as the Elo evaluation.
“The automation rate and Elo metrics are fully compatible, in that automation rate equals the probability of a win or tie against the human baseline under the same standards as the Elo evaluation.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Study design and conductnone given§ A.2 Autoflation · p. 16The autoflation metric is sensitive to false positives in annotation.
“Because the metric is sensitive to false positives in annotation, we audit all AI deliverables marked as successful to minimize the false-positive rate.”
Quote in the paperSubstantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
It may state two separate claims as one.
Study design and conductown data§ Example of Successful Project Completion · p. 12RLI excludes some types of work found commonly in the remote labor economy, including projects requiring interaction with the client, jobs that require working on a team, and other types of work that did not meet requirements.
“RLI excludes some types of work found commonly in the remote labor economy, including projects requiring interaction with the client (e.g. tutoring), jobs that require working on a team (e.g., project management), and other types of work that did not meet our requirements (see Appendix C.2 for the full list of requirements).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Study design and conductown data§ B.4 Evaluation Instructions · p. 19We iterated on evaluation instructions and audited annotator quality until achieving an inter-annotator agreement of ≥85% on a random subset of the projects.
“We iterated on these evaluation instructions and audited annotator quality until achieving an inter-annotator agreement of ≥ 85% on a random subset of the projects.”
Quote in the paperSubstantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim.
Study design and conductreasoning§ A.2 Autoflation · p. 16We audit all AI deliverables marked as successful to minimize the false-positive rate.
“Because the metric is sensitive to false positives in annotation, we audit all AI deliverables marked as successful to minimize the false-positive rate.”
Quote in the paperSubstantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one. The quote only partly states this claim.
Study design and conductnone given§ B.7 Evaluation Platform Details · p. 21The evaluation platform natively supports viewing documents, media, design & 3D, and data & interactive file types.
“The evaluation platform is a web-based multimedia viewer and file explorer. It provides native support for viewing the following file types:”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductnone given§ C.4 Data Collection Details · p. 26For projects sourced from freelancers, we only included projects where freelancers explicitly verified that they had the rights to sell us the work.
“For projects sourced from freelancers, we only included projects where freelancers explicitly verified that they had the rights to sell us the work.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductnone given§ B.6 Agent Setup · p. 20The evaluation platform accommodates a comprehensive range of file formats with fallback rendering capabilities for unsupported types.
“which accommodates a comprehensive range of file formats with fallback rendering capabilities for unsupported types.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Study design and conductown data§ B.5 Evaluation verification · p. 19To reduce false positives, we manually audited all annotation cases where the AI deliverable was labeled as good or better than the human deliverable.
“To reduce the rate of false positives, we manually audited all annotation cases where the AI deliverable was labeled as good or better than the human deliverable.”
Quote in the paperSubstantial claim- Rests on an assumption: The evaluators' judgments approximate those of a reasonable client.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Study design and conductnone given§ C.5 Project cost and completion time. · p. 27Self-reported completion times ensure economic accuracy.
“These values were also self-reported to ensure economic accuracy.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationcitation§ C.1 Categorization · p. 24The DWA taxonomy includes many ubiquitous and generic items, and coverage of these DWAs does not indicate meaningful coverage of remote work task types.
“The DWA taxonomy includes many ubiquitous and generic items such as 'Retrieve information from electronic sources,' and 'Read materials to determine needed actions,' [19], and coverage of these DWAs does not indicate meaningful coverage of remote work task types.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [19] [19] National Center for O*NET Development. DWA reference - O*NET 30.0 data dictionary. https://www.onetcenter.org/dictionary/30.0/excel/dwa\_reference.html , 2025. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductreasoning§ 3.2 Dataset Collection · p. 6A blocklist of domains is included to protect against contamination for long-tail data.
“To further protect against contamination in these cases, we include a blocklist of domains.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductnone given§ 3.2 Dataset Collection · p. 6None of the project descriptions in RLI are searchable.
“None of the project descriptions in RLI are searchable.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductcitation§ B.2 Elo Computation · p. 17Elo scores are computed using global Bradley-Terry fitting following the Chatbot Arena methodology [6].
“Following the Chatbot Arena methodology [6], we use global Bradley-Terry fitting on sampled preference edges to compute utility scores, which we refer to as Elo scores for ease of understanding.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [6] [6] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning , 2024. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
Interpretationcitation§ Example of Successful Project Completion · p. 12AI differs qualitatively from other automation technologies; it is not designed merely to automate specific tasks, but is being explicitly developed to automate human intelligence itself.
“AI differs qualitatively from other automation technologies; it is not designed merely to automate specific tasks, but is being explicitly developed to automate human intelligence itself.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Premisecitation§ 4.3 Qualitative Findings · p. 11Hendrycks et al. [12] show that the skills and weaknesses of LLMs can be decomposed into several distinct categories, such as broad world knowledge, memory, and audiovisual abilities.
“Hendrycks et al. [12] show that the skills and weaknesses of LLMs can be decomposed into several distinct categories, such as broad world knowledge, memory, and audiovisual abilities.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12] [12] Dan Hendrycks, Dawn Song, Christian Szegedy, Honglak Lee, Yarin Gal, Sharon Li, Andy Zou, Lionel Levine, Bo Han, Jie Fu, Ziwei Liu, Jinwoo Shin, Kimin Lee, Mantas Mazeika, Long Phan, George Ingebretsen, Adam Khoja, Cihang Xie, Olawale Salaudeen, Matthias Hein, Kevin Zhao, Alex Pan, David Duvenaud, Bo Li, Steve Omohundro, Gabriel Alfour, Max Tegmark, Kevin McGrew, Gary Marcus, Jaan Tallinn, Eric Schmidt, and Yoshua Bengio. A definition of agi, 2025. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
Study design and conductown data§ B.6 Agent Setup · p. 20In the main results, we report GPT-5 (CLI) because it outperformed GPT-5 (CUA).
“In the main results, we report GPT-5 (CLI), as this outperformed GPT-5 (CUA).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Study design and conductnone given§ Example of Successful Project Completion · p. 12The cost of projects reported by human professionals reflects the cost at the time of project completion and is not adjusted for inflation.
“The cost of the projects reported by human professionals reflects the cost at the time of project completion and is not adjusted for inflation.”
Quote in the paperNot a substantial claim- Rests on an assumption: The reported project costs accurately reflect the economic value of the work at the time of completion.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductnone given§ B.2 Elo Computation · p. 17Projects and model pairs are randomly sampled for comparison, using random ordering of model pairs to remove order effects.
“Projects and model pairs are randomly sampled for comparison, using random ordering of model pairs to remove order effects.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
There is a lack of standardized, empirical methods for monitoring the trajectory of AI automation.
“However, we lack standardized, empirical methods for monitoring the trajectory of AI automation.”
The paper relies on this without support.
The remote labor economy consists of a large share of computer-based work.
“spanning the large share of the economy that consists of computer-based work.”
The paper relies on this without support.
The human deliverables in RLI successfully complete their projects and are gold-standard.
“Human deliverable : A gold-standard deliverable that successfully completes the project, produced by a professional”
The paper relies on this without support.
A reasonable client standard is appropriate for judging whether AI deliverables complete commissioned work.
“whether the deliverable would be accepted by a reasonable client as the commissioned work.”
The paper relies on this without support.
The RLI project set is representative of the remote labor economy in general.
“While RLI does not fully represent every part of the remote labor economy, it is a substantial step towards measuring the ability of AI to automate the remote economy in general, rather than just current tasks.”
The paper relies on this without support.
The human reference deliverables represent the quality and level of work accepted by the original clients.
“The human reference deliverable establishes the baseline level and quality of work accepted by the original client.”
The paper relies on this without support.
The reported project costs accurately reflect the economic value of the work at the time of completion.
“The cost of the projects reported by human professionals reflects the cost at the time of project completion and is not adjusted for inflation.”
The paper relies on this without support.
The evaluators' judgments approximate those of a reasonable client.
“We instructed annotators to judge each deliverable holistically from the perspective of a reasonable client commissioning the project.”
The paper relies on this without support.
RLI is the broadest benchmark of its kind.
“While RLI is the broadest benchmark of its kind, it does not represent several types of remote work due to these constraints.”
The paper relies on this without support.
The Upwork taxonomy is designed for categorizing freelance work.
“since this taxonomy is designed for categorizing freelance work.”
The paper relies on this without support.
The judge LLM accurately classifies project briefs into the three categories.
“we computed the distribution over project types for RLI and GDPval by using a judge LLM to classify the project briefs using the following instructions.”
The paper relies on this without support.
The random sample of 60 freelancers and 275 completed jobs is representative of Upwork projects.
“we analyzed 275 completed jobs from a random sample of 60 freelancers, using the hours worked and dollars earned for each job.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Elo evaluations are described as both always triple-rated and conditionally triple-rated
MinorThe evaluation section says all judgments use three independent evaluations. Appendix B.2 instead says Elo comparisons start with two evaluations and receive a third only when needed to break a tie.
“We use majority voting across three independent evaluations to determine the final judgment.”
“For each project that a model pair is compared on, we perform majority voting, using two independent evaluations with a third to break ties if needed.”
Eligible-category count changes from 43 to 45
MinorSection 3.2 says the initial filtering left 43 eligible Upwork categories, while Appendix C.2 says the same sourcing criteria left 45 categories.
“This filtering resulted in 43 eligible categories.”
“This left us with 45 total Upwork categories to source projects from.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- [1][1] Daron Acemoglu. The simple macroeconomics of ai. Economic Policy , 40(121):13-58, 2025. -
- [17][17] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations , 2023. -
- [27][27] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling , 2024. -
Cited by
Sorted by citations
International AI Safety Report 2026Preprint
Bengio, Y., Clare, S., Prunkl, C., Andriushchenko, M., Bucknall, B., Murray, M., Bommasani, R., Casper, S., Davidson, T., Douglas, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.21012 · Cited by 74
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.14249 · Cited by 65
Self-citationThe 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems
Staufer, L., Feng, K. J. K., Wei, K., Bailey, L., Duan, Y., Yang, M., Ozisik, A. P., Casper, S., & Kolt, N. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806728 · Cited by 58
PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
Akyürek, A. F., Gosai, A., Zhang, C. B. C., Gupta, V., Jeong, J., Gunjal, A., Rabbani, T., Mazzone, M., Randolph, D., Meymand, M. M., et al. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1958 · Cited by 44
Self-citationHow Well Does Agent Development Reflect Real-World Work?Preprint
Wang, Z. Z., Vijayvargiya, S., Chen, A., Zhang, H., Arangarajan, V. A., Chen, J., Chen, V., Yang, D., Fried, D., & Neubig, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.01203 · Cited by 13
Chen, K., Wu, Q., Hou, T., Tang, T., Hu, X., Hou, Y., Li, B., Qian, C., Wang, G., Chen, H., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.20613 · Cited by 9
BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research AgentsPreprint
Wang, A., Meinhardt, G., Katz, J., Kim, J. H., Chaudhary, P. K., Blagden, C., & Xu, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03829 · Cited by 9
Self-citationSafePro: Evaluating the Safety of Professional-Level AI AgentsPreprint
Zhou, K., Jangam, S., Nagarajan, A., Polu, T., Oruganti, S., Liu, C., Kuo, C.-C., Zheng, Y., Narayanaraju, S. J., & Wang, X. E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.06663 · Cited by 8
CollabSkill: Evaluating Human-Agent Collaboration On Real-World TasksPreprint
Shao, Y., Wang, Z. Z., Ahuja, N., Wang, Y., Liu, B., & Yang, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.09833 · Cited by 6
Self-citationYi, D., Liu, T., Terzolo, M., Hasson, L., Sinha, A., Mendes, P. N., & Rabinovich, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.12306 · Cited by 5
Measuring Progress Toward AGI: A Cognitive FrameworkPreprint
Burnell, R., Yamamori, Y., Fırat, O., Olszewska, K., Hughes-Fitt, S., Kelly, O., Galatzer‐Levy, I. R., Morris, M. R., Dafoe, A., Snyder, A. M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.28405 · Cited by 5
ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services
Ji, F., Yang, J., Song, Z., Gao, L., Liang, J., Chen, Z., Zhang, J., & Chen, X. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.2014 · Cited by 3
Reliable agent engineering should integrate machine-compatible organizational principlesPreprint
Xian, R. P., Gabison, G. A., Alaa, A., Riedl, C., & Chrysos, G. G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.07665 · Cited by 3
Self-Sovereign AgentPreprint
Qu, W., Zhao, X., Zhang, J., & Song, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.08551 · Cited by 3
Zhou, J., Zhao, Y., Bao, Q. D., Cao, J., Chen, Z., Gao, C., Guo, K., Guo, M., Li, M., Lu, X., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.27155 · Cited by 3
Self-citationAhn, S. H., & Kim, J. (2026) · JMIR Formative Research · doi:10.2196/90273 · Cited by 2
CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI AgentsPreprint
Seong, H., Kim, S., Cho, Y. M., Joe, M., Kim, G., Park, Y., Kim, S., Seong, S., Kim, Y., Choi, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.20216 · Cited by 2
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive EconomiesPreprint
Hu, X., Xia, J., Xu, S., Song, K., Yuan, Y., Zhang, G., Ren, J., Feng, B., Lu, L., Zeng, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.09514 · Cited by 2
Economic Evaluations of Language ModelsPreprint
Wan, A., Hatgis-Kessell, S., Aguirre, T., Liang, P., & Bommasani, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.19375 · Cited by 2
Addressing the Reality Gap: A Three-Tension Framework for Agentic AI Adoption
Fournier, J., & Łodzikowski, K. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_5 · Cited by 1
AnalystBench: Benchmarking professional long-form report generation with web-mined multimodal tasks
Pham, C. M., Wang, Z., Mathur, P., Siu, A. F., Jain, A., Garimella, A., Sai, A. B., Lipka, N., Iyyer, M., & Manjunatha, V. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1197 · Cited by 1
SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet WorkbooksPreprint
Kundurthy, S., Na, C., Handley, M., Kirshner, Z., Zhang, C. B. C., Sharma, M., Strubell, E., & Ling, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.10002 · Cited by 1
Self-citationReflections and New Directions for Human-Centered Large Language ModelsPreprint
Ziems, C., Zhao, D., Wang, R., Jörke, M., Rushdi, A., Deepak, A., Yu, S., Agarwal, A., Agarwal, H., Aránguiz-Dias, G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.06901 · Cited by 1
Self-citationThe 2026 Singapore Consensus on Global AI Safety Research PrioritiesPreprint
Casper, S., Galeev, O., Bengio, Y., Kankanhalli, M., Lee, W. S., Maharaj, T., Meserole, C., Ong, L., Russell, S., Song, D., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.14611 · Cited by 1
Self-citationJobBench: Aligning Agent Work With Human Will
Li, Y., Feng, Y., Xu, Z., Ma, Z., Zheng, K., Jiang, F., Sun, X., Shao, R., Chen, Z., Huang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26329
Biological risk and the AI capability shift
Turner, S. D. (2026) · doi:10.59350/zp9qy-3gg73
Let's Build a Trustworthy Model Context Protocol!Preprint
Swaminathan, A., & Hannemann, A. (2026) · doi:10.36227/techrxiv.177100996.69474432/v1
Andréoletti, J., Maier, A., Domenech, L., & David, T. (2026) · HAL
A Statistical and Practical Study of Deploying Fair, Safe, and Productive AI
Somerstep, S. (2026) · Deep Blue (University of Michigan) · doi:10.7302/dspace/29729
The Open Source Economic Index of AI Adoption and CapabilityPreprint
Somerstep, S., Guha, A., Srivastava, D., & Sun, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.26118
Measuring AI-Induced Disempowerment: A Framework and Proposed Metrics
Chooi, J. Q., Lee, J., & Li, J. X. (2026) · Workshop on Evaluating Evaluations (EvalEval) · doi:10.18653/v1/2026.evaleval-1.36
Walking on the DARKSIDEPreprint
Aldo Gangemi, A., & Bottazzi, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.23370
DAYJOB: A Benchmark for Long-Horizon Professional WorkPreprint
Finley, S., Panavas, L., Mikkelson, T., Hinton, C., Ganss, S., Monton, B., Kendall, E., Spradlin, M., Bye, L., O'Brien, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.01306
How Much Work Can AI Agents Do? Task Value Inside the AI Time-Horizon FrontierPreprint
Harris, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7570818
AI, Bullshit, and the Liberal ArtsPreprint
Mitsch, C. (2026) · PhilSci-Archive (University of Pittsburgh)
MA-Bench: Measuring Operational Planning in Language ModelsPreprint
Lafuente-Mercado, R., Sincero, R., Sage, P., & Xue, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19138
FDE-Gym: The Last Mile of AI DeliveryPreprint
Yang, D., Yan, H., Wang, R., Zhang, B., Zuo, H., Zhao, Y., Chen, Y., Liao, X., Gao, C., Jiang, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.22350
Self-citationJoules-to-GDP: How Much Economic Value Do AI Systems Produce per Unit of Energy?Preprint
Zeinali, A., Saad-Falcon, J., Narayan, A., Abbasi, M., Hartley, J., Hennessy, J., Re, C., & Mirhoseini, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.30858
HERD: Harness-Evolving Recursive Distillation for Continual LearningPreprint
Sun, X., Nguyen, T. T., & Vu, T.-T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31401
SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?Preprint
Li, J., Wang, D., Zhao, X., Bai, Y., Zhu, X., Zhang, Y., Yin, C., Shi, W., Prabhu, G., Rajagopal, K. G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31854
Out of Place in My Own Work: How Generative AI Displaces Software Engineers
Tavantzis, T., Lambiase, S., & Russo, D. (2026) · VBN Forskningsportal (Aalborg Universitet) · doi:10.4230/lipics.esem.2026.9
Work Agents: Systems, Capabilities, and OptimizationPreprint
Xu, Z., Du, E., Gao, M., Yang, C., Shen, J., Chen, Z., Yao, Z., Guo, Y., Lu, Y., Huang, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.35845
WorkFactory: Turning Professional Intents into Executable Work for Scalable Agent TrainingPreprint
Zeng, Z., Chen, T., Deng, J., Wang, Q., Tang, S., Yang, Y., Ren, T., Zhao, W. X., Chang, F., He, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.37161
Harder Than the Task: From Technical to Organizational Difficulty in AI BenchmarkingPreprint
Barres, V., Barton, A., Foss, N. J., & Nielsen, H. M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.03494
Metadata
- Authors
- Mazeika, M., Gatti, A., Menghini, C., Sehwag, U. M., Singhal, S., Orlovskiy, Y., Basart, S., Sharma, M., Peskoff, D., Lau, E., Lim, J., Carroll, L., Blair, A., Sivakumar, V., Basu, S., Kenstler, B., Ma, Y., Michael, J., Li, X., Ingebretsen, O., Mehta, A., Mottola, J., Teichmann, J., Yu, K., Shaik, Z., Khoja, A., Ren, R., Hausenloy, J., Phan, L., Htet, Y., Aich, A., Rabbani, T., Shah, V., Novykov, A., Binder, F., Chugunov, K., Ramirez, L., Geralnik, M., Mesura, H., Lee, D., Cardona, E.-Y. H., Diamond, A., Yue, S., Wang, A., Liu, B., Hernandez, E., & Hendrycks, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2510.26787
- OpenAlex
- W7103153711
- Cited by
- 44
- References
- 0
- Citations by year
- 2026: 41 · 2025: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Innovationsfonden (4354-00006B)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Labor market dynamics and wage inequality
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- AI automation, remote work, large language models, AI agents, labor automation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
