Measuring AI Ability to Complete Long Software Tasks

Classified as probably not a substantial claim. May combine two claims.

In contrast to LAMBADA and GSM8K, which test skills not directly applicable to software engineering, the SWAA set represents actions needed in both software engineering work and the longer tasks studied.

“In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study.”

• cites [44] [44] Denis Paperno, Germ´ an Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern´ andez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1525-1534, 2016. - sourceSupported
• cites [45] [45] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. - sourceSupported

• AssumptionRests on an assumption: Real-world intellectual labor consists in part of measurable, single-step actions shorter than 1 minute.

May combine two claims.

RE-Bench consists of 7 challenging open-ended ML research engineering environments, each intended to take a human expert approximately 8 hours to complete.

“RE-Bench consists of 7 challenging open-ended ML research engineering environments, each of which are intended to take a human expert approximately 8 hours to complete.”

• cites [2] [2] Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114 , 2024. - sourcePartly supported

  The Abstract (p. 1) confirms that RE-Bench has seven challenging, open-ended ML research engineering environments. Section 3.2.1 (p. 8) says they were designed to allow significant progress in eight hours, not completion; the Conclusion (p. 21) says most were not saturated even by top human experts after eight hours. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

For RE-Bench, the paper used baselines from [2], considers task duration of each of 6 tasks as 8 hours, and uses mean score of baseliners who spent between 7 and 9 hours to convert raw score into success threshold.

“For RE-Bench, we used the baselines from [2]. As baseliners were instructed to achieve the best performance for each task, we consider the task duration of each of these 6 tasks as 8 hours, and instead use the mean score achieved by baseliners who spent between 7 and 9 hours to convert raw score into success threshold.”

• cites [2] [2] Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114 , 2024. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both modular-public and triframe/duet incorporate principles from the ReAct framework, which interleaves reasoning traces with actions.

“Both modular-public and triframe/duet incorporate principles from the ReAct framework [46], which interleaves reasoning traces with actions.”

• cites [46] [46] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https:// arxiv.org/abs/2210.03629 . - sourceSupported

Classified as probably not a substantial claim.

We forego the ability to use IRT metrics designed for test design, such as Fisher information functions for individual tasks.

“In exchange, we forego the ability to use IRT metrics designed for test design, such as Fisher information functions for individual tasks (used in e.g. Kipnis et al. [47]).”

• cites Kipnis et al. [47] [47] Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench - a sparse benchmark of reasoning and knowledge in large language models, 2025. URL https://arxiv.org/abs/2407.12844 . - sourceSupported

May combine two claims.

Algorithmic improvements have historically decreased the compute requirements for a fixed performance level and can substitute for compute limitations.

“However, algorithmic improvements, which have historically decreased the compute requirements for a fixed performance level [48] [49], can substitute for compute limitations.”

• cites [48] [48] Ege Erdil and Tamay Besiroglu. Algorithmic progress in computer vision, 2023. URL https: //arxiv.org/abs/2212.05153 . - sourceSupported
• cites [49] [49] Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models, 2024. URL https://arxiv.org/abs/2403.05812 . - sourceSupported

May combine two claims.

Between the release of GPT-2 and today, the compute used to train the most impressive frontier language models has increased by at least a factor of 10,000x, with training compute usage doubling every 6-10 months.

“Between the release of GPT-2 and today, the compute used to train the most impressive frontier language models has increased by at least a factor of 10,000x [40], with training compute usage doubling every 6-10 months [41].”

• cites [40] [40] Epoch AI. Data on notable AI models, 2024. URL https://epoch.ai/data/ notable-ai-models . Accessed: 2025-03-04. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [41] [41] Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2022. - sourcePartly supported

  Table 7 (Training compute, p. 19) reports a large increase in training compute for language models across 2009–2022. It does not establish that compute for frontier language models rose by at least 10,000× specifically between GPT-2’s release and the citing paper’s ‘today.’ The doubling-time clause is attached to a different citation and is not judged here. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

This is a common observation; see e.g. the improvements to software development capabilities from AIDE or recent work on KernelBench.

“This is a common obervation; see e.g. the improvements to software development capabilities from AIDE [50] or recent work on KernelBench: [51]”

• cites [50] [50] Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138 , 2025. - sourceSupported
• cites [51] [51] METR. Measuring automated kernel engineering. https://metr.org/blog/ 2025-02-14-measuring-automated-kernel-engineering/ , 02 2025. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous research has found that techniques such as best-of-k can substantially improve performance on a subset of these tasks.

“Previous research has found that techniques such as best-of-k can substantially improve performance on a subset of these tasks,[2]”

• cites [2] [2] Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114 , 2024. - sourceSupported

May combine two claims.

The 2023-2025 trend without the SWAA dataset retrodicts the trend since 2019, and the trends agree.

“We check whether the 2023-2025 trend without the SWAA dataset retrodicts the trend since 2019, and find that the trends agree.”

May combine two claims.

There is a strong upwards trend over time in model success rates, with recent models completing approximately 50% of all tasks, while earlier models perform substantially worse.

“There is a strong upwards trend over time, with recent models completing approximately 50% of all tasks, while earlier models perform substantially worse.”

May combine two claims.

AI agent performance on internal PR tasks is worse than would be predicted by maintainer time-to-complete as a measure of task length, but is consistent with contractor time-to-complete.

“We also find that AI agent performance is worse than would be predicted by maintainer time-to-complete as a measure of task length, but is consistent with contractor time-to-complete,”

May combine two claims.

The three supplementary external validity experiments find little evidence of performance trends being slower on the somewhat more realistic tasks tested, but do not rule out the possibility that trends are meaningfully slower on the distribution of tasks required to automate software engineer jobs.

“We include three supplementary external validity experiments, which find little evidence of performance trends being slower on the somewhat more realistic tasks we tested, but do not rule out the possibility that trends are meaningfully slower on the distribution of tasks required to automate software engineer jobs.”

May combine two claims.

The doubling time on SWE-bench Verified results is around 70 days, shorter than the 143 days predicted by HCAST + SWAA + RE-bench using 2024 models.

“while the doubling time predicted by HCAST + SWAA + RE-bench using 2024 models is 143 days, the doubling time on SWE-bench Verified results is shorter-around 70 days.”

May combine two claims.

Our tasks are designed to require relatively low amounts of context, but baseliners still spend over 25% of their time reminding themselves of relevant information.

“Our tasks are designed to require relatively low amounts of context but base liners still spend over 25% of their time reminding themselves of relevant information (e.g. how to do port forwarding).”

May combine two claims.

The paper measured how systematic differences might affect AI agent performance by including properties as 'messiness' factors, and found that absolute performance on 'messier' tasks was lower but trends in performance were similar to less messy tasks.

“We attempted to measure how these systematic differences might affect AI agent performance in Section F.2, by including the above properties as 'messiness' factors. We found that though the absolute performance on 'messier' tasks was lower, the trends in performance were similar to less messy tasks.”

May combine two claims.

A substantial fraction of human failures appeared to be due to insufficient expertise, giving up for unclear reasons, or being interrupted or bored.

“A substantial fraction of human failures appeared to fall in this category including humans having insufficient expertise for the task, or giving up on a task for unclear reasons (possibly due to getting interrupted, or getting bored).”

Probably not a substantial claim: may not pass S3 (load-bearing).

HCAST tasks range from easy tasks that take humans a couple of minutes to tasks that take expert humans multiple hours.

“Tasks in this suite range from easy tasks that take humans a couple of minutes (e.g. looking up a basic factual question on Wikipedia) to tasks that take expert humans multiple hours (e.g. writing CUDAkernels, or fixing a subtle bug in PyTorch).”

May combine two claims.

HCAST mainly includes tasks longer than 4 minutes, while SWAA focuses on tasks in the 2-second to 15-second range to measure GPT-2 and GPT-3.

“HCAST mainly includes tasks longer than 4 minutes, while we focused on tasks in the 2-second to 15-second range with SWAA in order to measure GPT-2 and GPT-3.”

May combine two claims.

Over a third of the GPT-4 failures resulted from repeating failed actions, compared to 2 out of 32 for o1.

“We find that over a third of the GPT-4 failures resulted from repeating failed actions, compared to 2 out of 32 for o1”

May combine two claims.

Issue 11 was the simplest task for maintainers; it requires writing simple comments across 10 Python files, and it took maintainers less than five minutes to complete.

“For example, Issue 11 was the simplest task for maintainers. It requires writing simple comments across 10 Python files, and it took maintainers less than five minutes to complete.”

May combine two claims.

Two of the tasks hardest for models in their respective time buckets are 'blackbox' and 'symbolic regression'; both involve guessing a hidden function under the constraint that probing for information is costly.

“For example, two of the tasks hardest for models in their respective time buckets are 'blackbox' and 'symbolic regression'. Both of these tasks involve guessing a hidden function under the constraint that probing for information is costly.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Models were consistently able to add a missing folder creation step to a single stage in a data pipeline.

“For example, models were consistently able to add a missing folder creation step to a single stage in a data pipeline.”

The quote only partly states this claim.

Models did not successfully complete Issue 11 once in 30 runs.

“However, models did not successfully complete the task successfully once in 30 runs.”

May combine two claims.

Different issues had different average times to score, with easier issues generally being much quicker to score, and all issues being faster to score than to complete.

“Different issues had different average times to score, with easier issues generally being much quicker to score, and all issues being faster to score than to complete.”

May combine two claims.

Early models like davinci-002 (GPT-3) struggle with syntactically correct tool use, while later models improved with clear explanations and examples in their prompt.

“Early models like davinci-002 (GPT-3) struggle with syntactically correct tool use, while later models improved with clear explanations and examples in their prompt.”

May combine two claims.

Models have become increasingly robust over time, and especially seem to be able to avoid repeating failed actions.

“Models have become increasingly robust over time, and especially seem to be able to avoid repeating failed actions.”

May combine two claims.

Repo maintainers scored submissions much faster than contractors; contractors took an average of 8 minutes per run, while repo maintainers took an average of 3.5 minutes per run.

“Repo maintainers scored submissions much faster than contractors, although the ratio was less dramatic than time to complete-contractors took an average of 8 minutes per run, while repo maintainers took an average of 3.5 minutes per run.”

May combine two claims.

Repository maintainers scored results much more consistently than contractors; contractor correlations were between 50-60%, and repo maintainer correlations were between 88-91%.

“In practice, repo maintainers scored results much more consistently than contractors. Contractor correlations were between 50-60%. Repo maintainer correlations were between 88-91% .”

May combine two claims.

All models still exhibit deficiencies in understanding their own limitations or proactively seeking out helpful information; instead, models tend to assume that they already know how to complete the tasks and then only reevaluate after they fail to do so.

“We also observe that all models still exhibit deficiencies in understanding their own limitations or proactively seeking out helpful information; instead, models tend to assume that they already know how to complete the tasks and then only reevaluate after they fail to do so.”

May combine two claims.

Agents improved greatly on tasks requiring situational awareness of the AI agent's limitations or defeating an opposing strategy.

“In addition, we noticed that agents improved greatly on tasks requiring situational awareness of the AI agent's limitations or defeating an opposing strategy.”

May combine two claims.

When tasks required editing Python scripts, GPT-3 struggled with formatting even when provided explicit instructions, GPT-4o would sometimes write scripts line-by-line using echo commands in bash, while o1-preview would consistently edit files efficiently.

“For example, we noticed that when tasks required editing Python scripts, GPT-3 struggled with formatting even when provided explicit instructions, and GPT-4o would sometimes write scripts line-by-line using echo commands in bash, while o1-preview would consistently edit files efficiently.”

May combine two claims.

Current AI models consistently demonstrate knowledge of tools and techniques, but appear to struggle with the larger context required for working in a real codebase.

“Current AI models, on the other hand, consistently demonstrate knowledge of tools and techniques, but appear to struggle with the larger context required for working in a real codebase.”

May combine two claims.

Our tasks are designed to require minimal context, which somewhat mitigates the context problem; our internal PRs were not designed this way, and so baseliners took many times longer than employees.

“Our tasks are designed to require minimal context, which somewhat mitigates this problem; our internal PRs (Section C.2) were not designed this way, and so baseliners took many times longer than employees.”

May combine two claims.

Half of the o1 failures resulted from abandoning the task prematurely, while only a quarter of the GPT-4 failures resulted from the same.

“Half of the o1 failures resulted from abandoning the task prematurely, while only a quarter of the GPT-4 failures resulted from the same”

May combine two claims.

In practice, o1 often decides to grep for the term 'AR report' and fails to find it (as the actual report is named 'ARA report'), and then makes a guess.

“In practice, o1 often decides to grep for the term 'AR report' and fails to find it (as the actual report is named 'ARA report'), and then makes a guess.”

• AssumptionRests on an assumption: Real-world intellectual labor often involves messy details that benchmarks usually don't include.

May combine two claims.

Tasks with high messiness and which take humans a long time to complete tend to have low success rates, and vice versa for shorter, lower messiness tasks.

“Tasks with high messiness and which take humans a long time to complete tend to have low success rates, and vice versa for shorter, lower messiness tasks.”

• AssumptionRests on an assumption: The 16 properties used to rate tasks are representative of how real-world tasks might be systematically harder and are relevant to AI agent performance.

May combine two claims.

The doubling time for the 80% time horizon is similar to the 50% plot, but horizons are substantially lower.

“The doubling time is similar to the 50% plot, but horizons are substantially lower.”

May combine two claims.

Messiness is correlated with task length, and there are very few short tasks with a high messiness score.

“We find that messiness is correlated with task length, and that we have very few short tasks with a high messiness score.”

• AssumptionRests on an assumption: The 16 properties used to rate tasks are representative of how real-world tasks might be systematically harder and are relevant to AI agent performance.

May combine two claims.

Linear and hyperbolic curves have poor fits.

“Linear and hyperbolic curves have poor fits (Figure 15).”

May combine two claims.

o1's performance is on average better on the messier variant of the report, and human baseliners performed much worse.

“Surprisingly, o1's performance is on average better on the messier variant of the report, and human baseliners performed much worse.”

• AssumptionRests on an assumption: Real-world intellectual labor often involves messy details that benchmarks usually don't include.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Our baseliners' skills vary significantly, and a wide variety of skills are relevant to these tasks.

“Our baseliners' skills also vary significantly, and a wide variety of skills are relevant to these tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The modality of interaction in these tasks is relatively narrow; no tasks require the use of a mouse.

“For example, the modality of interaction in these tasks is also relatively narrow-for example, no tasks require the use of a mouse.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

No tasks require cooperating or competing with humans or other agents, while real software engineering or ML research involves communicating and coordinating with managers and other engineers or researchers.

“No tasks require cooperating or competing with humans or other agents, 18 while real software engineering or ML research involves communicating and coordinating with managers and other engineers or researchers.”

May combine two claims.

AI agent success rate is imperfectly predicted by human time-to-complete, meaning that other factors also substantially influence the difficulty of tasks.

“Figure 3 shows that AI agent success rate is imperfectly predicted by human time-to-complete, meaning that other factors also substantially influence the difficulty of tasks.”

May combine two claims.

There is no evidence of either much slower performance trends or a plateau specific to the higher messiness subset.

“In particular, we find no evidence of either much slower performance trends, or a plateau, specific to our higher messiness subset.”

• AssumptionRests on an assumption: The 16 properties used to rate tasks are representative of how real-world tasks might be systematically harder and are relevant to AI agent performance.

May combine two claims.

From manual reviews of baseline attempts, we observe that humans sometimes simply give up even when the task seems within their capabilities, and it is unclear what selection effects are produced on the distribution of success times as a result.

“From manual reviews of baseline attempts, we also observe that humans sometimes simply give up even when the task seems within their capabilities, and it is unclear what selection effects are produced on the distribution of success times as a result.”

May combine two claims.

When told to look in one of the resources, o1 performs a more general search, and finds the report it's looking for more often.

“On the other hand, when told to look in one of the resources, o1 performs a more general search, and finds the report it's looking for (more often).”

• AssumptionRests on an assumption: Real-world intellectual labor often involves messy details that benchmarks usually don't include.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The 2024-2025 trend appears faster than the 2019-2025 trend, but due to the small number of models it is unclear if this is noise.

“The 2024-2025 trend appears faster than the 2019-2025 trend, but due to the small number of models it is unclear if this is noise.”

Classified as probably not a substantial claim. May combine two claims.

The baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, with the majority having attended world top-100 universities and an average of about 5 years of relevant experience.

“Our baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, with the majority having attended world top-100 universities. They have an average of about 5 years of relevant experience, with software engineering baseliners having more experience than ML or cybersecurity baseliners.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Time horizon is always measured relative to a domain, task distribution, and human baseliners' level of skill and context, and can be difficult to measure in practice, especially for longer horizons or success rates near 100%.

“we caution that time horizon is always measured relative to a domain, task distribution, and human baseliners' level of skill and context, and can be difficult to measure in practice, especially for longer horizons or success rates near 100%.”

Classified as probably not a substantial claim.

Our tasks do not perfectly represent the average segment of intellectual labor by researchers and software engineers.

“Since our tasks do not perfectly represent the average segment of intellectual labor by researchers and software engineers, this raises the question of external validity (Section 4): whether the exponential trend holds on real-world tasks.”

May combine two claims.

148 of the 169 tasks have human baselines, but researcher estimates are relied on for 21 tasks in HCAST.

“148 of the 169 tasks have human baselines, but we rely on researcher estimates for 21 tasks in HCAST.”

May combine two claims.

Errors in individual model horizon lengths are highly correlated between models because tasks at the same human time rating vary widely in difficulty for models, and sampling easy or hard tasks shifts all models' horizon estimates in the same direction.

“While there are wide error bars on each individual models' horizon lengths, these errors are highly correlated between models. This is because tasks at the same human time rating vary widely in difficulty for models, and sampling easy (or hard) tasks will result in a higher (or lower) horizon estimate for all models.”

May combine two claims.

If future progress follows the 2024-2025 trend, 1-month AI would arrive sooner, with half the probability in 2027 and even late 2026.

“If future progress instead follows the 2024-2025 trend, 1-month AI would arrive sooner, with half the probability in 2027 and even late 2026.”

May combine two claims.

Time horizon estimates for SWE-bench Verified, which use annotator times, likely underestimate the time horizon of less capable models relative to contractor times, shortening doubling times.

“As a result, our time horizon estimates for SWE-bench Verified (which use annotator times) are likely to underestimate the time horizon of less capable models relative to contractor times, in turn shortening doubling times.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If the difficulty ratings are underestimates, the paper may be underestimating the pace of model improvement.

“If this is the case, we may be underestimating the pace of model improvement.”

May combine two claims.

The paper aggregates human baseline times into a task length rating by taking the geometric mean time of successful baseline runs, which is more predictive of model performance than mean time because baseline times are roughly log-normally distributed.

“We aggregate human baseline times into a task length rating by taking the geometric mean time of successful baseline runs, which is more predictive of model performance than mean time because baseline times are roughly log-normally distributed.”

• AssumptionRests on an assumption: Baseline times are roughly log-normally distributed.

Probably not a substantial claim: may not pass S3 (load-bearing).

Carlsmith and Cotra developed the 'bio-anchors' framework, relating the compute involved in training AI models to the 'effective horizon length' of tasks required for AI to have transformative impacts.

“Carlsmith [22] and Cotra [23] developed the 'bio-anchors' framework, in which they related the compute involved in training AI models to the 'effective horizon length' of tasks required for AI to have transformative impacts.”

• cites [22] [22] Joseph Carlsmith. How much computational power does it take to match the human brain?, 2020. URL https://www.openphilanthropy.org/research/ how-much-computational-power-does-it-take-to-match-the-human-brain/ . Open Philanthropy report, August 14, 2020. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [23] [23] Ajeya Cotra. Draft report on ai timelines, 2020. URL https://www.alignmentforum. org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines . Draft report on AI timelines; technical report. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The time horizon of human baseliners is not directly comparable to the time horizon of ML models.

“Note that the time horizon is not directly comparable to the time horizon of ML models (see Section C.1.4)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper excludes human baseline attempts done by in-house staff or using different methodology from its human baseline numbers.

“As many of these attempts were done by in-house staff or were done using different methodology, we exclude these attempts from our human baseline numbers.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Tasks in the suite typically use static environments that do not significantly change unless directly acted upon by the agent, whereas real tasks often occur in changing environments.

“Static environments Tasks in this suite typically use environments that do not significantly change unless directly acted upon by the agent. In contrast, real tasks often occur in the context of a changing environment.”

May combine two claims.

The paper filters successful runs because collecting enough baselines to estimate the time-versus-success curve for each task would be practically difficult, and to exclude cases where baseline failed for reasons not applicable to models.

“We chose to filter successful runs for two main reasons - one practical and one principled. Firstly, collecting enough baselines to estimate the time-versus-success curve for each task would be practically difficult. Secondly, we wanted to exclude cases where the baseline failed for reasons that are not applicable to models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The majority of HCAST tasks do not require visual or multimodal capabilities, and all tasks are solvable by text editing via a bash shell.

“Because modern frontier AI systems are relatively more proficient at text-based tasks, the majority of these tasks do not require visual/multimodal capabilities, and all tasks are solvable by text editing via a bash shell.”

• AssumptionRests on an assumption: Modern frontier AI systems are relatively more proficient at text-based tasks.

Classified as probably not a substantial claim.

The three task suites studied were designed to isolate a specific unit of work that can be reliably accomplished within a time limit.

“As with most benchmarks, the three task suites we study were designed to isolate a specific unit of work that can be reliably accomplished within a time limit.”

Classified as probably not a substantial claim. May combine two claims.

HCAST baseliners work in the same environment as agents, using Vivaria, with screens and audio recorded for manual review, to prevent cheating.

“Baseliners work in the same environment as agents, using Vivaria (an open source platform for language model agent evals) with their screens and audio recorded for manual review, to prevent cheating.”

Probably not a substantial claim: may not pass S4 (based).

The paper expects that most HCAST tasks are solvable by humans with a few years of professional experience in the relevant domain.

“As a result, we expect that most of these tasks are solvable by humans with a few years of professional experience in the relevant domain.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The human time horizon of around 1.5 hours implies that the best models will surpass humans in under 7 months.

“However, in practice it's much lower, at around 1.5 hours (which would imply that the best models will surpass humans in under 7 months).”

Probably not a substantial claim: may not pass S3 (load-bearing).

SWAA is baselined by internal employees with relevant expertise using a custom webapp that enables more accurate timing.

“Unlike HCAST and RE-Bench, which were baselined by external contractors, SWAA is baselined by internal employees with relevant expertise using a custom webapp that enables more accurate timing.”

Classified as probably not a substantial claim. May combine two claims.

For GPT-3 and GPT-3.5, release dates for the closest available models are used because the evaluated models are closed-source and no longer available through API access.

“However, GPT-3 (davinci) and GPT-3.5 (code-davinci-002 and text-davinci-002) are closed-source and no longer available through API access, but we use their release dates for the closest available models, which OpenAI advertises as having equivalent performance: GPT-3's release date for davinci-002, and GPT-3.5's date for gpt3.5-turbo-instruct.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For SWAA decision-based tasks, only one selection is allowed to avoid random guessing; for fill-in-the-blank tasks, baseliners can try multiple times until getting the answer correctly or opting to skip.

“For decision based tasks, only one selection is allowed to avoid random guessing; for fill in the blank tasks, baseliners can try multiple times until getting the answer correctly or opting to skip.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Benchmarks like HCAST and SWE-bench Verified may be insufficient for forecasting real-world AI capabilities, and accurate forecasting will require deliberate construction of more realistic benchmarks without these limitations.

“It may be that benchmarks like HCAST and SWEBench Verified are insufficient for forecasting real-world AI capabilities, and accurate forecasting will require deliberate construction of more realistic benchmarks without these limitations.”

• AssumptionRests on an assumption: Realistic tasks rarely have automatic scoring, and often require dealing with other agents, resource constraints, dynamic environments, and high reliability requirements.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

No tasks involve interacting with other autonomous agents, which likely increases task difficulty in real-world settings.

“No interaction with other agents No tasks involve interacting with other autonomous agents. Coordinating with, or competing with, other agents seems likely to increase task difficulty.”

Probably not a substantial claim: may not pass S4 (based).

Most HCAST tasks are designed to be realistic, such that doing well on the task requires skills expected to be economically useful.

“Instead, most tasks are designed to be realistic, such that doing well on the task requires skills we expect to be economically useful.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The tasks used to benchmark AI capabilities are systematically different from real tasks, and these differences could result in observed trends not generalizing to real-world tasks.

“The tasks we use to benchmark AI capabilities are systematically different from real tasks. These differences could result in the trends we observe on these tasks not generalizing to real world tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Contextualizing AI capabilities using human time horizon was proposed by Ngo [21], and similar concepts were discussed in Carlsmith [22] and Cotra [23].

“Contextualizing AI capabilities using human time horizon was proposed by Ngo [21], and similar concepts were discussed in Carlsmith [22] and Cotra [23].”

• cites Ngo [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Carlsmith [22] [22] Joseph Carlsmith. How much computational power does it take to match the human brain?, 2020. URL https://www.openphilanthropy.org/research/ how-much-computational-power-does-it-take-to-match-the-human-brain/ . Open Philanthropy report, August 14, 2020. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Cotra [23] [23] Ajeya Cotra. Draft report on ai timelines, 2020. URL https://www.alignmentforum. org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines . Draft report on AI timelines; technical report. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

After screening out attempts on tasks other than the subset, failed attempts, and those with issues, the paper includes 286 successful baselines from approximately 460 total attempts.

“After screening out attempts on tasks other than our subset of HCAST, failed attempts, and those with issues (such as using disallowed AI tools), we include 286 successful baselines from approximately 460 total attempts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Ngo proposed using the time horizon for which AI systems outperform most human experts at most tasks to measure general AI capabilities.

“Ngo [21] proposed using the time horizon for which AI systems outperform most human experts at most tasks to measure general AI capabilities.”

• cites [21] [21] [Richard Ngo. Clarifying and predicting AGI. https://www.lesswrong.com/posts/ BoA3agdkAzL6HQtQP/clarifying-and-predicting-agi , 2023. Accessed: 2024-03-21.](https://www.lesswrong.com/posts/BoA3agdkAzL6HQtQP/clarifying-and-predicting-agi) -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

RE-Bench baseliners had access to AI tools, so the human scores on these tasks may be biased upwards.

“RE-Bench baseliners had access to AI tools, so the human scores on these tasks may be biased upwards.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

No SWAA tasks, and few HCAST tasks, saliently involve making efficient use of a limited resource, a common constraint in real-world tasks.

“Lax resource constraints NoSWAAtasks, and few HCAST tasks saliently involve making efficient use of a limited resource-a common constraint in real-world tasks.”

May combine two claims.

All SWAA, HCAST, and RE-Bench tasks are automatically scorable, which imposes constraints on solution format and reduces task open-endedness and the need for sensible value judgments.

“Automatic scoring All tasks we use are automatically scorable, meaning a piece of code running in the task environment determines the final score. This imposes constraints on e.g. the format of solutions, that tend to reduce task open-endedness, and the need for sensible value judgments.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Development of SWAA tasks was blind to AI agent performance; all tasks were written before seeing AI attempts, and elicitation was carried out on a separate development task suite.

“Development of SWAA tasks was blind to AI agent performance; that is, all tasks were written before seeing AI attempts, and elicitation (development of the few-shot prompt used for evaluation) was carried out on a separate development task suite.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Task durations are calculated using the geometric mean of successful baselines, with manual estimates for tasks lacking successful baselines.

“Task durations are calculated using the geometric mean of successful baselines, with manual estimates for tasks lacking successful baselines.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paper baselined each SWAA decision-based task 4 times and each fill-in-the-blank style task 3 times.

“We baselined each decisionbased task 4 times and each fill-in-the-blank style task 3 times.”

Probably not a substantial claim: may not pass S1 (asserted).

Ngo [21] writes that a 1-month AGI would necessarily exceed human performance at economically valuable endeavors like writing large software applications, founding startups, and making novel scientific discoveries.

“First, Ngo [21] writes that a 1-month AGI (defined as an AI that outperforms most knowledgeable humans who are given 1 month of work hours, i.e. 167 hours, to perform the task) would necessarily exceed human performance at economically valuable endeavors like writing large software applications, founding startups, and making novel scientific discoveries.”

• cites Ngo [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Item Response Theory models the relationship between latent traits, such as ability, and observed responses to test items.

“Item Response Theory (IRT) [24], which models the relationship between latent traits (such as ability) and observed responses to test items.”

• cites [24] [24] Frank B Baker. The basics of item response theory . ERIC, 2001. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Very few tasks are punishing of single mistakes, in part to reduce the expected cost of collecting human baselines.

“Unpunishing Similarly, very few tasks are punishing of single mistakes. 14 This is in part to reduce the expected cost of collecting human baselines.”

Probably not a substantial claim: may not pass S4 (based).

Existing agent and capability benchmarks often lack a unified metric that allows for tracking progress over time and comparing models of vastly different capabilities.

“While these benchmarks provide valuable insights into specific capabilities, they often lack a unified metric that allows for tracking progress over time and comparing models of vastly different capabilities.”

May combine two claims.

One month is around the period when new hires at a company begin to complete onboarding and generate economic value, so an AI with a horizon of 1 month could be capable of acquiring context like a human employee.

“Second, one month is around the period when new hires at a company begin to complete onboarding and generate economic value, 9 and so an AI with a horizon of 1 month could be capable of acquiring context like a human employee, allowing it to complete high-context as well as low-context tasks.”

• cites [29] [29] Dave Zielinski. How to optimize onboarding, June 2019. URL https://www.shrm. org/topics-tools/news/hr-magazine/how-to-optimize-onboarding . Accessed on March 17, 2025. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Employees often start generating economic value midway through the onboarding process.

May combine two claims.

This bias is most significant on longer tasks, which often have a baseliner success rate below 50%; therefore, the paper has lower confidence in difficulty ratings of these longer tasks and expects they may be underestimates.

“This bias is most significant on longer tasks, which often have a baseliner success rate below 50%; therefore, we have lower confidence in the difficulty ratings of these longer tasks, and expect they may be underestimates.”

May combine two claims.

HCAST is designed to be diverse, but its shortest tasks are around 1 minute long, limiting representativeness and achievable resolution in measuring AI agent performance on shorter tasks.

“HCAST is designed to be a diverse set of tasks, but the shortest tasks are around 1 minute long, limiting both the representativeness and the achievable resolution in measuring AI agent performance on shorter tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The preliminary set of 71 SWAA tasks was filtered down to the final set of 66 by excluding tasks that more than one baseliner failed.

“These were filtered down to the final set of 66 by excluding tasks that more than one baseliner failed.”

May combine two claims.

The paper thinks the human time horizon of around 1.5 hours is artificially low, given that many human failures seemed to be artifacts of the incentive scheme.

“As discussed above, we think this is artificially low, given that many human failures seemed to be artifacts of our incentive scheme.”

Classified as probably not a substantial claim. May combine two claims.

We used the same agent scaffolds across the evaluation suite, with no task-specific prompting or scaffolding, except for the SWAA tasks, which used a simple prompting scaffold.

“We used the same agent scaffolds across the evaluation suite, with no task-specific prompting or scaffolding, except for the SWAA tasks, which used a simple prompting scaffold.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-2 is incompatible with our scaffolding due to low context length, so we imputed a score of zero for GPT-2 on all tasks in RE-Bench and HCAST.

“GPT-2 is incompatible with our scaffolding due to low context length, so we imputed a score of zero for GPT-2 on all tasks in RE-Bench and HCAST.”

Classified as probably not a substantial claim.

These agents are developed on a held-out dev set of tasks, not included in HCAST, to reduce the likelihood of overfitting to these tasks.

“These agents are developed on a held-out dev set of tasks, not included in HCAST, to reduce the likelihood of overfitting to these tasks.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

If such AGI systems are ever developed, the long-term trend in time horizon will be faster than exponential, with an asymptote at the date of AGI deployment.

“Therefore, if such systems are ever developed, the long-term trend in time horizon will be faster than exponential, with an asymptote at the date of AGI deployment.”

Probably not a substantial claim: may not pass S4 (based).

For models to be cost competitive on real work, their total cost to complete a PR must be less than the cost for a repository maintainer to implement this PR themselves plus ancillary costs.

“For models to be cost competitive on real work, their total cost to complete a PR must be less than the cost for a repository maintainer to implement this PR themselves (plus ancilliary costs, e.g. the time of a code reviewer).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Removing the imputed GPT-2 zero scores has a negligible effect on all subsequent results in this paper.

“Removing these imputed GPT-2 zero scores has a negligible effect on all subsequent results in this paper.”

Probably not a substantial claim: may not pass S4 (based).

To fit equation 1, we weight tasks by diversity, such that a task from a family of size n gets weight 1 / √ n.

“To fit equation 1, we weight tasks by diversity, such that a task from a family of size n gets weight 1 / √ n .”

Probably not a substantial claim: may not pass S3 (load-bearing).

Forecasting AI time horizons far into the future is much more sensitive to changes in doubling rate than to constant factors in time horizon.

“Although extrapolation is always imperfect, forecasting AI time horizons far into the future is much more sensitive to changes in doubling rate than to constant factors in time horizon.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A naive extrapolation based on o1's time horizon of 39 minutes and a past doubling time of 218 days (3.2x/year) predicts that AIs will reach a 1-month 50% time horizon about 4.8 years after the release date of o1.

“For example, a naive extrapolation based on o1's time horizon of 39 minutes and a past doubling time of 218 days (3.2x/year) predicts that AIs will reach a 1-month 50% time horizon (roughly 8 doublings over o1) about 4.8 years after the release date of o1.”

May combine two claims.

This allowed models to more reliably interpret task instructions correctly and use code to find correct answers.

“This allowed models to more reliably interpret task instructions correctly and use code to find correct answers.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The choice γ_task = 0 only affects the time horizon of GPT-2 and GPT-3 and does not meaningfully affect our results.

“This choice only affects the time horizon of GPT-2 and GPT-3 and does not meaningfully affect our results.”

• AssumptionThe authors state this as an assumption: In the 2PL model and our method, γ_task = 0 for most tasks, though some SWAA tasks are multiple-choice with a 25% chance of guessing correctly.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Issue 8 demonstrates how context on the codebase may provide a dramatic speedup in human time to complete between baseliners and repository maintainers.

“Issue 8 demonstrates how context on the codebase may provide a dramatic speedup in human time to complete between baseliners and repository maintainers.”

May combine two claims.

Baseliners and models both struggle compared to repository maintainers, but they struggle in different ways.

“While baseliners and models both struggle compared to repository maintainers, they struggle in different ways.”

May combine two claims.

A 2x increase in doubling time would delay the 1-month point by a further 4.8 years, but a 2x constant factor decrease in time horizon would only delay it by 0.6 years.

“A 2x increase in doubling time would delay the 1 month point by a further 4.8 years, but a 2x constant factor decrease in time horizon would only delay it by 0.6 years.”

May combine two claims.

In our method, each agent's time horizon is computed individually, whereas IRT has an initial calibration/joint optimization stage in which task difficulty is estimated from all agent scores, so the ability score of one agent may depend on other agents' performance.

“First, IRT has an initial calibration/joint optimization stage in which the task difficulty is estimated from all agent scores, so the ability score of one agent may depend on other agents' performance; in our method each agent's time horizon is computed individually.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The human baseliners that determine the length of most tasks have much less context than average employees, potentially increasing measured task length.

“The human baseliners that determine the length of most tasks have much less context than average employees, potentially increasing measured task length.”

Probably not a substantial claim: may not pass S4 (based).

All agents were provided with the same affordances provided to human baseliners.

“All agents were provided with the same affordances provided to human baseliners.”

May combine two claims.

The 2PL model uses one slope parameter per task, whereas we use one slope parameter per agent; in our setting with many fewer agents than tasks, this reduces the overall number of parameters.

“Third, the 2PL model uses one slope parameter per task, whereas we use one slope parameter per agent. In our setting with many fewer agents than tasks, this reduces the overall number of parameters.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If future AI systems are capable of substituting for human research engineers and/or increasing the compute-efficiency of training, the rate of AI progress will increase.

“If future AI systems are capable of substituting for human research engineers and/or increasing the compute-efficiency of training, the rate of AI progress will increase.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

If an AGI is capable of completing all tasks expert humans can with a success rate of at least X%, its X% time horizon will necessarily be infinite.

“If an artificial general intelligence (AGI) is capable of completing all tasks expert humans can with a success rate of at least X%, its X% time horizon will necessarily be infinite.”

The quote only partly states this claim.

We would not expect models to consistently complete Issue 11.

“and so we would not expect models to consistently complete this task.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Measuring time horizon requires a large dataset annotated with human time, and time horizon is always measured relative to a task distribution and baseliners' levels of context and skill.

“Although time horizon is an intuitive measure of AI agent capability, measuring it requires a large dataset annotated with human time, and time horizon is always measured relative to a task distribution and baseliners' levels of context and skill.”

Classified as probably not a substantial claim. May combine two claims.

Resolving the internal issues was real work performed by internal staff, so results on these tasks might better represent performance on real economically valuable tasks than a typical benchmark task.

“Resolving these issues was real work performed by internal staff, so we might expect results on these tasks to better represent performance on real economically valuable tasks than a typical benchmark task.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Highly skilled baseliners can complete tasks far faster than average employees.

“However, highly skilled baseliners can also complete tasks far faster than average employees.”

Classified as probably not a substantial claim. May combine two claims.

We select only successful completions of a task, and encourage baseliners to give up on tasks they may not complete in a reasonable amount of time.

“Most notably, we select only successful completions of a task, and encourage baseliners to give up on tasks they may not complete in a reasonable amount of time.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

There are reasons to believe that our task distribution is systematically different from the distribution of economically valuable work (and perhaps systematically different than the distribution of risk-model relevant tasks).

“There are reasons to believe that our task distribution is systematically different from the distribution of economically valuable work (and perhaps systematically different than the distribution of risk-model relevant tasks).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We lose some information in the multiple stages of estimation-conversion of baseline data to task difficulty ratings, time horizon computation, and linear regression to find the trend-and end-to-end methods could be more data-efficient.

“We also lose some information in the multiple stages of estimationconversion of baseline data to task difficulty ratings, time horizon computation, and linear regression to find the trend-and end-to-end methods could be more data-efficient.”

May combine two claims.

As a result, model performance might be more negatively correlated with the task messiness measure than it ought to be, and predictive adjustments based on messiness might work less well for models released long before or after the time of this analysis.

“As a result, we might expect model performance to be more negatively correlated with our task messiness measure than it ought to be-and for predictive adjustments based on messiness to work less well for models released long before, or after, the time of this analysis.”

• AssumptionRests on an assumption: The 16 properties used to rate tasks are representative of how real-world tasks might be systematically harder and are relevant to AI agent performance.

Probably not a substantial claim: may not pass S3 (load-bearing).

The SWE-bench Verified analysis includes only 6 of the 11 models included in the main result.

“Note that this is only 6 of the 11 models we include in the main result.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper verified annotator time buckets by running seven baselines across 6 SWE-bench Verified tasks.

“We verified annotator time buckets by running seven baselines across 6 SWE-bench Verified tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

When models are measured in different domains of intellectual labor like research mathematics, computational biology, or law, we expect their time horizons to differ.

“When models are measured in different domains of intellectual labor like research mathematics, computational biology, or law, we expect their time horizons to differ.”

Classified as probably not a substantial claim. May combine two claims.

This adversarial selection is due to selecting for perceiving the factor to be relevant to model performance, and all factors being expected to make tasks more difficult.

“This is due to the combination of selecting for perceiving the factor to be relevant to model performance, and all of these factors being expected to make tasks more difficult.”

• AssumptionRests on an assumption: The 16 properties used to rate tasks are representative of how real-world tasks might be systematically harder and are relevant to AI agent performance.

May combine two claims.

We have put a limited amount of effort into eliciting models, so our results are a reasonable lower bound, and some models may have somewhat greater capabilities than we demonstrate.

“We have put a limited amount of effort into eliciting models to get good performance on our tasks, so while our results are a reasonable lower bound, some models may have somewhat greater capabilities than we demonstrate.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Accurately measuring very long time horizons requires a dataset of difficult tasks with long human baseline runs, which can be impractical to construct, especially because success criteria for realistic difficult tasks are often complex enough to require manual grading.

“First, accurately measuring very long time horizons requires a dataset of difficult tasks with long human baseline runs, which can be impractical to construct, especially because success criteria for realistic difficult tasks are often complex enough to require manual grading.”

May combine two claims.

Our per-task human time estimates are likely noisy due to relatively small sample size, and potentially also systematically skewed in various ways.

“Our per-task human time estimates are likely noisy due to relatively small sample size, and potentially also systematically skewed in various ways.”

May combine two claims.

This results in easier tasks' difficulty being underestimated, meaning that the SWE-bench Verified time horizon estimates likely underestimate true model performance for less capable models, which in turn leads to a steeper exponential trend.

“This results in easier tasks' difficulty being underestimated, meaning that the SWE-bench Verified time horizon estimates likely underestimate true model performance for less capable models, which in turn leads to a steeper exponential trend.”

May combine two claims.

Applying fits with more parameters would be more likely to overfit than to give accurate predictions, because there are only 12 frontier models and the exponential fit has high R2 with only two parameters.

“Because there are only 12 frontier models in the time span we studied, and the exponential fit has such high R 2 with only two parameters, we think applying fits with more parameters would be more likely to overfit than to give accurate predictions.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Though we attempt to match baseliners with appropriate tasks, this process is unlikely to be perfect.

“Though we attempt to match baseliners with appropriate tasks, this process is unlikely to be perfect.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Many real-world tasks require very high reliability, and these are underrepresented in our dataset due to the difficulty of measuring models on these tasks.

“Many real-world tasks require very high reliability, and these are underrepresented in our dataset due to the difficulty of measuring models on these tasks.”

Probably not a substantial claim: may not pass S4 (based).

Accurately measuring that the X%-time horizon of an AI agent is about t minutes requires many tasks of human length t that the AI agent completes with a success rate of about X%.

“Accurately measuring that the X%-time horizon of an AI agent is about t minutes requires many tasks of human length t that the AI agent completes with a success rate of about X%.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Our expert baseliners are likely much more skilled than the average software engineer, potentially decreasing our measured task length.

“Our expert baseliners are likely much more skilled than the average software engineer, potentially decreasing our measured task length.”

Classified as probably not a substantial claim.

Factors that make humans worse at some tasks may improve agent performance.

“indeed, factors that make humans worse at some tasks may indeed improve agent performance.”

• AssumptionRests on an assumption: Real-world intellectual labor often involves messy details that benchmarks usually don't include.