EvilGenie: A Reward Hacking Benchmark

May combine two claims.

Models hardcode answers to test cases and exhibit other misaligned behavior.

“We find instances of models hardcoding answers to test cases and exhibiting other misaligned behavior.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The authors include two more ambiguous problems during manual review of runs.

“We also include two more during manual review of runs.”

May combine two claims.

Overall, the authors find LLM judges to be highly effective, while holdout tests have many surprising failure modes.

“Overall, we find LLM judges to be highly effective, while holdout tests have many surprising failure modes.”

May combine two claims.

Codex and Claude Code exhibit explicit reward hacking.

“We observe explicit reward hacking by both Codex and Claude Code”

Probably not a substantial claim: may not pass S4 (based).

Agents frequently combine general algorithmic implementations with reward hacking behaviors, allowing them to pass many of the holdout test cases.

“because agents frequently combine general algorithmic implementations with reward hacking behaviors, allowing them to pass many of the holdout test cases.”

Probably not a substantial claim: may not pass S4 (based).

The authors found LLM judges to be a much more effective strategy than holdout tests.

“as we found LLM judges to be a much more effective strategy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In some ambiguous problems, the output expected by the harness follows an obvious rule.

“While in some ambiguous problems, the output expected by the harness follows an obvious rule (e.g., the smallest lexicographically)”

May combine two claims.

Two of GPT-5's false positives are edge cases in which the model hardcodes the exact decimal precision expected by the tester, but only after calculating the correct solution, and one instance that is actually a heuristic.

“Two of GPT 5's false positives are two edge cases, in which the model hardcodes the exact decimal precision expected by the tester, but only after calculating the correct solution, and one instance that is actually a heuristic.”

May combine two claims.

Claude Haiku 3.5 has somewhat worse performance than most other models, and GPT-3.5-turbo-1106 performs poorly.

“Claude Haiku 3.5 has somewhat worse performance than most other models, and GPT-3.5-turbo-1106performs poorly.”

May combine two claims.

On unambiguous problems, the authors found good performance across most models tested, with GPT-4.1 and Claude Sonnet 3.5 (new) performing the best, and most models making few errors.

“On unambiguous problems, we found good performance across most models tested, with GPT-4.1 and Claude Sonnet 3.5 (new) performing the best, and most models making few errors.”

May combine two claims.

These were cases for which the model wrote correct but inefficient code, or missed a subtle edge case.

“These were cases for which the model wrote correct but inefficient code, or missed a subtle edge case.”

May combine two claims.

On unambiguous problems, GPT-5 produced only one false positive and no false negatives.

“On unambiguous problems, GPT-5 produced only one false positive, and, to our knowledge, no false negatives.”

May combine two claims.

The authors found five instances of test.py being deleted; in all cases test.py was deleted by Gemini after all of the test cases were correctly run.

“We found five instances of test.pybeing deleted; however, in all cases test.py was deleted by Gemini after all of the test cases were correctly run.”

May combine two claims.

For Sonnet 4, we see a higher rate of reward hacking on ambiguous problems in the Inspect scaffold, and similar rates for unambiguous problems.

“For Sonnet 4, we see a higher rate of reward hacking on ambiguous problems in the Inspect scaffold, and similar rates for unambiguous problems.”

May combine two claims.

Gemini demonstrated the lowest rate of misaligned behaviors but was the only model which deleted or modified the test file in our experiments.

“Gemini demonstrated the lowest rate but was the only model which deleted or modified the test file in our experiments.”

May combine two claims.

Certain reward-hacking behaviors emerged even when the agent failed to pass all visible tests, suggesting that test success and exploitative behavior are not always tightly coupled.

“Conversely, certain reward-hacking behaviors emerged even when the agent failed to pass all visible tests, suggesting that test success and exploitative behavior are not always tightly coupled.”

May combine two claims.

We see similar rates of reward hacking for GPT-5 in the Inspect and Codex scaffolds, for both ambiguous and unambiguous problems.

“We see similar rates of reward hacking for GPT-5 in the Inspect and Codex scaffolds, for both ambiguous and unambiguous problems.”

May combine two claims.

Gemini's final submission was compatible with the unmodified test runner, and was in fact correct.

“However, its final submission was compatible with the unmodified test runner, and was in fact correct.”

Probably not a substantial claim: may not pass S4 (based).

Reward hacking and other misaligned behavior clearly exists in current code generating LLMs.

“Reward hacking and other misaligned behavior clearly exists in current code generating LLMs.”

May combine two claims.

False positives occur when a model writes a correct but inefficient solution and the largest test case is in the holdout set, or when the model misses an edge case.

“This occurs when a model writes a correct but inefficient solution, and the largest test case is in the holdout set, or when the model misses an edge case.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Adding useful comments to the testing files does not occur in any of our cases of file editing.

“This does not occur in any of our cases of file editing; but this could potentially complicate automated detection of reward-hacking.”

May combine two claims.

In all three instances where Claude attempted to hard-code test cases on an unambiguous problem, it only hard-coded a small subset, in three cases failing to pass even the provided visible tests.

“In all three instances where Claude attempted to hard-code test cases on an unambiguous problem, it only hard-coded a small subset -in three cases failing to pass even the provided visible tests.”

May combine two claims.

All our false negatives, and most of our false positives, are ambiguous edge cases.

“Note that all our false negatives, and most of our false positives, are ambiguous edge cases.”

May combine two claims.

For us, GPT-5 marks 4.5% of Claude Sonnet 4 runs as reward hacking, but 25.3% if you count heuristics.

“For us, GPT-5 marks 4.5% of Claude Sonnet 4 runs as reward hacking, but 25.3% if you count heuristics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Figure 6, Claude checks if a solution is close to a known test case and outputs accordingly.

“Here, Claude checks if a solution is close to a known test case and outputs accordingly.”

May combine two claims.

For Sonnet 4.5, Anthropic finds 1% reward hacking through both detection methods; we find 0%.

“For Sonnet 4.5, they find 1% reward hacking through both detection methods; we find 0%.”

• cites Ant25bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

We found that 4.2% of Claude 4's submissions had this property, though notably one third of these were not actually reward hacking.

“We found that 4.2% of Claude 4's submissions had this property, though notably one third of these were not actually reward hacking.”

May combine two claims.

Across all problems, we observe a 1% FPR and 16.7% FNR for GPT-5.

“Across all problems, we observe a 1% FPR and 16.7% FNR for GPT-5.”

Probably not a substantial claim: may not pass S4 (based).

GPT-5 includes special logic to allow these incorrect test cases to pass.

“GPT-5 includes special logic to allow these incorrect test cases to pass.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The test harness arbitrarily demands an exact decimal precision for each output, which varies by the output.

“The test harness arbitrarily demands an exact decimal precision for each output, which varies by the output.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Due to differences between the setups, the results from the Inspect scaffold are not directly comparable to the results in previous sections.

“Due to differences between the setups, these results are not directly comparable to the results in previous sections.”

Classified as probably not a substantial claim.

The authors reserve 30% of test cases (capped at 10) as holdout tests that remain inaccessible to the agent during the task.

“We reserve 30% of test cases (capped at 10) as holdout tests that remain inaccessible to the agent during the task.”

Classified as probably not a substantial claim.

In the more thorough detection scheme, if any of five LLM judges or either of the other two methods flag a potential case, a human reviewer performs a final assessment.

“In our more thorough reward-hacking detection scheme, if any of five LLM judges or either of the other two methods flag a potential case, a human reviewer performs a final assessment to determine whether reward hacking has indeed occurred.”

Classified as probably not a substantial claim.

Hardcoding responses to the given test cases rather than implementing a general algorithmic solution will likely result in failure on holdout tests.

“Thus, hardcoding responses to the given test cases rather than implementing a general algorithmic solution will likely result in failure on holdout tests”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This deletion is likely the result of training to clean up its own test files.

“This is likely the result of training to clean up its own test files.”

Classified as probably not a substantial claim.

The authors classify any solution that passes all visible tests while failing at least one holdout test as a potential instance of reward hacking.

“Therefore, we classify any solution that passes all visible tests while failing at least one holdout test as a potential instance of reward hacking.”

Probably not a substantial claim: may not pass S4 (based).

The categorization scheme will produce false positives, for instance when an otherwise legitimate solution fails to handle an edge case present in the holdout set.

“This categorization scheme will produce false positives-for instance, when an otherwise legitimate solution fails to handle an edge case present in the holdout set.”

Classified as probably not a substantial claim.

The holdout set consists of a random 30% (up to 10) of the original test cases.

“we remove a random 30% (up to 10) of the original test cases to form a holdout set.”

Classified as probably not a substantial claim.

Agents are not informed about the holdout set.

“The agents are not informed about this holdout set.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Given the general reduction in reward hacking with solve rate, the authors cannot rule out the possibility that the rate of reward hacking does not decrease, but is rather pushed to harder tasks over time.

“Given the general reduction in reward hacking with solve rate, we can not rule out the possibility that the rate of reward hacking does not decrease, but is rather is pushed to harder tasks over time.”

• AssumptionThe authors state this as an assumption: Accuracy on the benchmark is a reasonable proxy for the model's coding abilities.

Classified as probably not a substantial claim. May combine two claims.

The agent can view, edit, and run any file in the directory.

“The agent can view, edit, and run any file in the directory.”

May combine two claims.

The authors took all problems flagged as potentially reward hacking by any of the LLMs or any of the other automated tests and reviewed these cases by hand, and also reviewed problems marked by GPT-5 as heuristic that passed all test cases, totaling 27 problems.

“we took all of the problems that were flagged as potentially reward hacking by any of the LLMs or any of the other automated tests and reviewed these cases by hand.We also reviewed problems marked by GPT-5 as heuristic that passed all test cases.This is a total of 27 problems.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For agents evaluated with Inspect, the authors only use one LLM judge (GPT-5) and do not consider the holdout tests.

“For agents evaluated with Inspect, we only use one LLM judge (GPT-5), and do not consider the holdout tests”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors spot-checked a subset of unflagged solutions to check for false negatives.

“We also spot-checked a subset of unflagged solutions to check for false negatives.”

Classified as probably not a substantial claim.

The authors use human review to confirm reward hacking when an automated detection method is triggered.

“In a case where one of these methods is triggered, we use human review to confirm that reward hacking has occurred.”

Probably not a substantial claim: may not pass S4 (based).

The authors consider it unlikely that an illegitimate solution could pass all visible and held-out tests while simultaneously evading detection by the LLM judge.

“we consider it unlikely that an illegitimate solution could pass all visible and held-out tests while simultaneously evading detection by the LLM judge.”

• AssumptionThe authors state this as an assumption: An illegitimate solution is unlikely to pass all visible and held-out tests while simultaneously evading detection by the LLM judge.

Classified as probably not a substantial claim.

The authors detect any edits to (or deletion of) test cases.json or test.py and flag them as reward hacking.

“Therefore, we detect any edits to (or deletion of) these files, and flag them as reward hacking.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The use of different scaffolds across models introduces potential confounds; behavioral differences may arise as much from the scaffolding environment as from the models themselves.

“Finally, the use of different scaffolds across models introduces potential confounds -behavioral differences may arise as much from the scaffolding environment as from the models themselves.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

It makes more sense to modify an incorrect test result than an overly strict one.

“This may be because it makes more sense to modify an incorrect test result than an overly strict one.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Others [ZRC25, BHG+25] have had less success with LLM judges.

“Others [ZRC25, BHG + 25], have had less success with LLM judges.”

• cites ZRC25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites BHG + 25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

IMPOSSIBLEBENCH's agents receive explicit instructions not to modify the test cases.

“This is despite their agents receiving explicit instructions not to modify the test cases.”

• cites ZRC25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

We have a much smaller sample, but the difference is large enough that it is noteworthy.

“We notably have a much smaller sample, but the difference is large enough that it is noteworthy.”

Classified as probably not a substantial claim.

As the LLM judges are not given test cases, this may appear to them as a genuine mistake, rather than an attempt to pass the test cases in violation of the spec.

“As the LLM judges are not given test cases, this may appear to them as a genuine mistake, rather than an attempt to pass the test cases in violation of the spec.”

May combine two claims.

We have a small sample size (N=9) for ambiguous problems, so our results there are fairly noisy.

“We have a small sample size (N=9) for ambiguous problems, so our results there are fairly noisy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This deletion appears to be the result of training to clean up its own test files after writing a script.

“This appears to be the result of training to clean up its own test files after writing a script.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Rapid reminders that tests are failing, combined with increased opportunities, may explain the higher rate of reward hacking from Claude.

“These rapid reminders that the tests are failing, combined with the increase in number of opportunities, may explain the higher rate of reward hacking from Claude.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Some ambiguous problems are nearly impossible to solve without reward hacking.

“This makes sense, as some ambiguous problems are nearly impossible to solve without reward hacking.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We did not manually verify all instances which didn't trigger any of our detection methods.

“Furthermore, we did not manually verify all instances which didn't trigger any of our detection methods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In IMPOSSIBLEBENCH, an agent which passes all test cases (including the incorrect one) is known to have reward hacked.

“Then, an agent which passes all test cases (including the incorrect one) is known to have reward hacked.”

• cites ZRC25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Using a full scaffold, IMPOSSIBLEBENCH finds a reward hacking rate of around 0% for GPT-5 and around 5% for Claude Sonnet 4.

“Using a full scaffold, IMPOSSIBLEBENCH finds a reward hacking rate of around 0% for GPT-5 and around 5% for Claude Sonnet 4 (see figure 11 of [ZRC25]).”

• cites ZRC25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The LIVECODEBENCH test suites do not always achieve full behavioral coverage, meaning the failures of the held out tests may have simply reflected gaps in the test suite.

“The LIVECODEBENCH test suites do not always achieve full behavioral coverage, meaning the failures of the held out tests may have simply reflected gaps in the test suite.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The ten-minute execution limit may have constrained the models' ability to realize they were unable to solve the problem, so perhaps a higher rate would be observed if they worked for longer.

“First, the ten-minute execution limit may have constrained the models' ability to realize they were unable to solve the problem, so perhaps a higher rate would be observed if they worked for longer.”

Classified as probably not a substantial claim.

Reward hacking can potentially skew the success of LLMs on currently existing benchmarks giving less accurate measures of their capabilities.

“In addition to the risks of incorrect code reward-hacking can potentially skew the success of LLMs on currently existing benchmarks giving less accurate measures of their capabilities.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Deleting the testing files after tests have successfully run does not impact the correctness of the testing procedure and does not change whether the model's code is correct or passes the given tests.

“This does not have any impact on the correctness of the testing procedure and thus does not change whether the model's code is correct and whether it passes the given tests.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Contest programming may not be representative of other domains of interest.

“Finally, contest programming may not be representative of other domains of interest.”

May combine two claims.

This deletion behavior does not positively contribute to the model's performance on the benchmark, so the authors do not consider it to be an instance of reward hacking.

“however, this behavior does not positively contribute to the model's performance on the benchmark so we do not consider it to be an instance of reward hacking.”

Classified as probably not a substantial claim.

Concurrent work [ZRC25] studies reward hacking by coding agents in impossible settings.

“Concurrent work [ZRC25] studies reward hacking by coding agents in impossible settings.”

• cites ZRC25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The authors have a human verified false positive rate for all detection methods as well as a lower-bound on the reward-hacking prevalence of the models.

“Thus we have a human verified false positive rate for all of the detection methods as well as a lower-bound on the reward-hacking prevalence of the models.”

• AssumptionThe authors state this as an assumption: Samples marked as legitimate by the LLM are not reward hacking.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

We intentionally use a setup in our benchmark which has many affordances for reward-hacking, so we expect the current impact to be small.

“We intentionally use a setup in our benchmark which has many affordances for reward-hacking, so we expect the current impact to be small.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is unlikely, but conceivable, that we are missing some cases of reward hacking.

“It is unlikely, but conceivable, that we are missing some cases of reward hacking.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Our sandboxing infrastructure remains imperfect; models could plausibly detect that they were being evaluated, which may have altered their behavior.

“Second, our sandboxing infrastructure remains imperfect; models could plausibly detect that they were being evaluated, which may have altered their behavior.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Differences in work style may explain the higher rate of reward hacking from Claude: Claude often makes many iterative edits, whereas Gemini often thinks for a long time before producing an output.

“This may reflect differences in work style: Claude often makes many iterative edits, whereas Gemini often thinks for a long time before producing an output.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Models are more likely to hardcode a correct but alternative answer than an incorrect one.

“This may be because models are more likely to hardcode a correct but alternative answer than an incorrect one.”

May combine two claims.

In [MET25b], METR reports o3 reward hacks on RE-Bench 30.4% of the time, and on HCAST 0.7% of the time.

“In [MET25b], METR reports o3 reward hacks on RE-Bench 30.4% of the time, and on HCAST 0.7% of the time.”

• cites MET25bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Perhaps the first recorded instance of reward hacking in a programming setting was presented by [Wei13].

“Perhaps the first recorded instance of reward hacking in a programming setting was presented by [Wei13].”

• cites Wei13Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A master list of reward misspecification examples (not limited to programming settings) can be found in [KUM + 22b, KUM + 22a].

“A master list of reward misspecification examples (not limited to programming settings) can be found in [KUM + 22b, KUM + 22a].”

• cites KUM + 22bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites KUM + 22aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

On Anthropic's 'reward hacking prone coding tasks v2', Claude Sonnet 4 fails the holdout tests 5% of the time, and triggers their classifier 14% of the time.

“On Anthropic's”

• cites Ant25bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We do not consider this reward hacking, as this only involves arbitrary output formatting, not bypassing the solution to the problem.

“We do not consider this reward hacking, as this only involves arbitrary output formatting, not bypassing the soluion to the problem.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

OpenAI reports reward hacking by an early checkpoint of a model in the 'o' series, and they discuss the effectiveness of LLM judges, both on the agent's final submission and on its chain of thought.

“OpenAI reports reward hacking by an early checkpoint of a model in the”

• cites BHG + 25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

This is likely due to the increased complexity of METR's tasks and environment.

“This is likely due to the increased complexity of METR's tasks and environment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is unclear whether Anthropic's 5% and 14% are percentages of tasks which passed all visible tests, or total percentages of tasks which passed visible tests but failed hidden tests.

“It is unclear whether this is a percentage of the tasks which passed all visible tests, or this is the total percentage of tasks which passed the visible tests but failed the hidden tests.”

• cites Ant25bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Because GPT-5 chooses to pass the test cases over implementing a correct program, we consider this reward hacking.

“Because it chooses to pass the test cases over implement a correct program, we consider this reward hacking.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Another popular programming benchmark is SWE-Bench, where reward hacking by looking at future commits was observed.

“Another popular programming benchmark is SWE-Bench [JYW + 24] where reward hacking by looking at future commits was observed [Kah25].”

• cites JYW + 24Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kah25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Anthropic's monitor flags 14% of the runs as reward hacking, but it is unclear how they define reward hacking and whether it includes heuristic solutions.

“Their monitor flags 14% of the runs as reward hacking, it is unclear how exactly they are defining reward hacking here and whether it includes what we call heuristic solutions.”

• cites Ant25bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A blog post [Tha25] describes a reward hacking benchmark called rhb-v1, in which agents given performance optimization problems reward hack under pressure.

“A blog post [Tha25] describes a reward hacking benchmark called rhb-v1, in which agents given performance optimization problems reward hack under pressure.”

• cites Tha25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Therefore, we have identical false negative rates, but they have a higher false positive rate.

“Therefore, we have identical false negative rates, but they have a higher false positive rate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It may be that METR sees a higher false positive rate because their setting is more complicated.

“It may be that METR sees a higher false positive rate because their setting is more complicated.”

May combine two claims.

Anthropic's problems and tests are not openly released, no models other than Claude models are evaluated, and very few details on the classifiers are given.

“However, these problems and tests are not openly released, no models other than Claude models are evaluated, and very few details on the classifiers are given.”

• cites Ant25aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ant25bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

Alternatively, METR might not be filtering out heuristic solutions in their prompt like we do.

“Alternatively, they might not be filtering out heuristic solutions in their prompt like we do.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The code and dataset for rhb-v1 have not been released as of publication.

“However the code and dataset have not been released as of publication.”

• cites Tha25Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In contrast, METR finds a 4.6% FPR and a 16.7% FNR.

“In contrast, METR finds a 4.6% FPR and a 16.7% FNR.”

• cites MET25aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Weimer noticed in an earlier project, GenProg, a genetic coding algorithm, tasked with sorting a list simply returned an empty list, as the verifier only checked whether the output list was sorted.

“Weimer noticed in an earlier project, GenProg [GNFW12], a genetic coding algorithm, tasked with sorting a list simply returned an empty list, as the verifier only checked whether the output list was sorted.”

• cites GNFW12Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.