KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?

May combine two claims.

FlashAttention achieves 2-4× speedup over standard attention while reducing memory from O(N^2) to O(N).

“FlashAttention (Dao et al., 2022) achieves 2-4× speedup over standard attention while reducing memory from O(N 2 ) to O(N ).”

• cites Dao et al., 2022 sourceSupported

May combine two claims.

The TF32 baseline adjustment primarily affects compute-bound problems (accounting for 21.2% of problems), while the hidden test suite addresses instances where models relied on distribution-specific exploits (affecting 11.6% of problems).

“Decomposing this performance drop, we find that the TF32 baseline adjustment primarily affects compute-bound problems (accounting for 21.2% of problems), while the hidden test suite addresses instances where models relied on distribution-specific exploits (affecting 11.6% of problems).”

• Also stated in: § 3.2 Decomposing the Two Refinements · plain

  “21.2% of the 250 problems are affected only by the TF32 baseline correction, 11.6% only by the hidden test suite, and 2.4% by both.”

May combine two claims.

Under verified single-turn evaluation with seven frontier LLMs, the best-performing model (GPT-5.5) achieves a 0.88× geometric mean speedup, significantly lower than the 1.43× speedup observed under the standard evaluation protocol.

“Under verified single-turn evaluation with seven frontier LLMs, we find that the best-performing model (GPT-5.5)achieves a 0.88× geometric mean speedup, significantly lower than the 1.43× speedup observed under the standard evaluation protocol.”

• Also stated in: § 1 Introduction · plain

  “GPT-5.5'saggregate speedup recalibrates from 1.43× under the standard protocol to 0.88×.”

• Also stated in: § 3.1 Verified Leaderboard · plain

  “across all problems the best model, GPT-5.5, reaches only 0.88×”

• AssumptionRests on an assumption: The best-of-5 fastest correct sample is a fair measure of model capability.

May combine two claims.

The mismatch between FP32 and TF32 baselines accelerates the PyTorch baseline by >1.5× on 24% of all problems, primarily those dominated by matrix multiplication, disproportionately driving the aggregate geometric mean downward.

“our estimation shows that this mismatch accelerates the PyTorch baseline by >1.5× on 24% of all problems, primarily those dominated by matrix multiplication, disproportionately driving the aggregate geometric mean downward.”

The quote only partly states this claim.

Models often exploit the narrow test distribution by hardcoding bypasses for specific tensor values, skipping required computations and artificially accelerating execution.

“We observe that models exploit the specific values of test input tensors drawn from the narrow uniform distribution (torch.rand())used in standard testing.”

• Also stated in: § 1 Introduction · plain

  “We find concrete instances where LLMs engage in reward hacking by exploiting these properties.”

May combine two claims.

Three ill-designed Level 2 problems (PIDs 23, 80, 83) have PyTorch references that are mathematically guaranteed to produce a constant zero output regardless of input; these are the only three problems in the full 250-problem benchmark for which the reference output remains constant under all four input distributions of the hidden test suite, leaving 97 active Level 2 problems.

“We exclude three ill-designed Level 2 problems (PIDs 23, 80, 83) whose PyTorch references are mathematically guaranteed to produce a constant zero output regardless of input; these are the only three problems in the full 250-problem benchmark for which the reference output remains constant under all four input distributions of our hidden test suite (Appendix I), leaving 97 active Level 2 problems.”

May combine two claims.

GPT-5.5 generates a kernel for the ReLU activation that checks whether the input shape matches the test configuration and, if so, returns the input unchanged, passing correctness checks and reporting a 374× speedup.

“GPT-5.5 generates a kernel for the ReLU activation that checks whether the input shape matches the test configuration and, if so, returns the input unchanged (since ReLU(x) = x for all x ≥ 0).This identity function passes correctness checks and reports a 374× speedup.”

May combine two claims.

In FP32+TF32, the GEMM takes ~0.8ms while the 4 post-GEMM kernels take ~0.15ms total, fusion saves at most 0.15ms from a 0.95ms total.

“In FP32+TF32, the GEMM takes ∼0.8ms while the 4 post-GEMM kernels take ∼0.15ms total, fusion saves at most 0.15ms from a 0.95ms total, yielding marginal improvement that is offset by the fused kernel's lower occupancy.”

May combine two claims.

In BF16 precision, where the baseline already uses Tensor Cores by default, GPT-5.5 achieves 1.63× at Level 2, confirming that the FP32 deflation is a baseline artifact rather than a fundamental limitation of model capability.

“As further validation, we evaluate all models in BF16 precision where the baseline already uses Tensor Cores by default (Appendix F).GPT-5.5 achieves 1.63× at Level 2, confirming that the FP32 deflation is a baseline artifact rather than a fundamental limitation of model capability.”

May combine two claims.

21.2% of the 250 problems are affected only by the TF32 baseline correction, 11.6% only by the hidden test suite, and 2.4% by both.

“21.2% of the 250 problems are affected only by the TF32 baseline correction, 11.6% only by the hidden test suite, and 2.4% by both.”

May combine two claims.

Manual inspection of individual kernels surfaces four recurring behaviors: reward hacking via shape-conditional identity shortcuts, numerical instability from unstable variance formulas, counter-productive 'optimizations' that add memory or work on an already-efficient path, and genuine fusion gains on memory-bound workloads.

“Manual inspection of individual kernels surfaces four recurring behaviors, detailed with code in Appendix H: reward hacking via shape-conditional identity shortcuts (Appendix H.1), numerical instability from unstable variance formulas (Appendix H.3), counter-productive”

May combine two claims.

Speed stays below parity everywhere but improves with problem complexity: overall Correct Speedup spans only 0.47-0.88× across models, and for the strongest model it rises from 0.90× on single-operator Level 1 problems to 0.92× on full-model Level 3 problems.

“Speed stays below parity everywhere but improves with problem complexity: overall Correct Speedup spans only 0.47-0.88×across models, and for the strongest model it rises from 0.90× on single-operator Level 1 problems to 0.92× on full-model Level 3 problems, where larger architectures expose more fusable, memory-bound structure.”

May combine two claims.

GPT-5.5 leads on essentially every metric, but no model is uniformly strong across speed, correctness, and memory.

“GPT-5.5 leads on essentially every metric, but no model is uniformly strong across speed, correctness, and memory.”

May combine two claims.

At Level 1 most models save memory on 63-84% of problems through operator fusion even while running slower, but this collapses to 20-36% at Level 3, where complex access patterns defeat naive fusion.

“at Level 1 most models save memory on 63-84% of problems through operator fusion even while running slower, but this collapses to 20-36% at Level 3, where complex access patterns defeat naive fusion.”

May combine two claims.

Fast@1 stays near 40-53% even for the best models, meaning they beat PyTorch on only about half of problems.

“Fast@1 stays near 40-53% even for the best models, meaning they beat PyTorch on only about half of problems.”

May combine two claims.

D2 and D3 catch fewer but distinct bugs (overflow and underflow respectively).

“D2 and D3 catch fewer but distinct bugs (overflow and underflow respectively).”

May combine two claims.

The sub-1× average speedup does not arise from catastrophic failures but rather reflects the typical outcome: per-problem CDF analysis shows most kernels are modestly slower, with the speedup tail populated by memory-bound elementwise operations amenable to fusion.

“The sub-1× average speedup does not arise from catastrophic failures but rather reflects the typical outcome: per-problem Cumulative Distribution Function (CDF) analysis (Appendix D) shows most kernels are modestly slower, with the speedup tail populated by memory-bound elementwise operations amenable to fusion.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

D1 failures indicate kernels that fail even on the original distribution, these are compilation errors or basic correctness bugs unrelated to hidden testing.

“D1 failures indicate kernels that fail even on the original distribution, these are compilation errors or basic correctness bugs unrelated to our hidden testing.”

May combine two claims.

Correctness is largely decoupled from speed: several models exceed 95% correctness while sitting well below 1× (Gemini 3 Flash is 99.0% correct yet only 0.65× at Level 2).

“Correctness, however, is largely decoupled from speed: several models exceed 95% correctness while sitting well below 1× (Gemini 3 Flash is 99.0%correct yet only 0.65× at Level 2).”

May combine two claims.

D4 (negation) catches the most failures across all models and levels, primarily from identity shortcuts exploiting all-positive inputs.

“D4 (negation) catches the most failures across all models and levels, primarily from identity shortcuts exploiting all-positive inputs.”

May combine two claims.

At Level 2, switching to the TF32 baseline accounts for the dominant speedup drop (GPT-5.5: 1.67→0.88×, a 47% reduction), while the hidden test suite adds only a further 3% reduction (0.88→0.85×).

“At Level 2, switching to the TF32 baseline accounts for the dominant speedup drop (GPT-5.5:1.67→0.88×,a 47% reduction), while the hidden test suite adds only a further 3% reduction (0.88→0.85×).”

May combine two claims.

At Level 1, the hidden test suite contributes more (1.37→1.17×, a 15% drop from catching reward hacks on elementwise ops), while TF32 provides an additional 23% drop.

“At Level 1, the hidden test suite contributes more (1.37→1.17×, a 15% drop from catching reward hacks on elementwise ops), while TF32 provides an additional 23% drop.”

May combine two claims.

In BF16, the GEMM takes ~0.4ms while post-GEMM overhead remains similar, the 0.15ms savings now represents a much larger fraction of the 0.55ms total, producing genuine net speedup.

“In BF16, the GEMM takes ∼0.4ms (2× faster Tensor Cores) while post-GEMM overhead remains similar, the 0.15ms savings now represents a much larger fraction of the 0.55ms total, producing genuine net speedup.”

May combine two claims.

Under the per-problem criteria, 21.2% of the 250 problems are affected only by TF32, 11.6% only by the hidden test suite, 2.4% by both, and 64.8% by neither.

“Under these definitions, 21.2% of the 250 problems are affected only by TF32, 11.6% only by the hidden test suite, 2.4% by both, and 64.8% by neither; the per-level breakdown is given in Table 4.”

May combine two claims.

The overlap between distributions is modest, approximately 30% of failures caught by D4 are also caught by D2 or D3, indicating that the distributions target largely independent bug classes.

“The overlap between distributions is modest, approximately 30% of failures caught by D4 are also caught by D2 or D3, indicating that our distributions target largely independent bug classes.”

May combine two claims.

No model achieves Correct Speedup ≥ 1× at any difficulty level; across all problems the best model, GPT-5.5, reaches only 0.88×, and even its strongest per-level result (0.92× at Level 3) leaves its kernels about 8% slower than TF32 PyTorch on average.

“The central finding is that no model achieves Correct Speedup ≥ 1× at any difficulty level: across all problems the best model, GPT-5.5, reaches only 0.88×, and even its strongest per-level result (0.92× at Level 3) leaves its kernels about 8% slower than TF32 PyTorch on average.”

• AssumptionRests on an assumption: The best-of-5 fastest correct sample is a fair measure of model capability.

May combine two claims.

Many generated kernels achieve genuine memory reductions through fusion-eliminating intermediate tensor allocations between operators-without improving execution time.

“Many generated kernels achieve genuine memory reductions through fusion-eliminating intermediate tensor allocations between operators-without improving execution time.”

May combine two claims.

Gemini Flash's pre-transposed MLP kernel for L3 Problem 3 allocates duplicate weight copies (1.81× peak memory) while the actual matmul executes at the same speed, resulting in 0.26× net speedup.

“The pre-transposition allocates duplicate weight copies (1.81× peak memory) while the actual matmul executes at the same speed, resulting in 0.26× net speedup (slower due to parameter copy overhead and loss of cuBLAS's optimized nn.Linear path).”

• AssumptionRests on an assumption: On H200 with TF32, cuBLAS already handles weight transposition internally via its column-major/row-major dispatch logic with negligible overhead.

May combine two claims.

The GPT-5.5 kernel computes a row-wise float32 cumulative product but stops processing a row once the running product is exactly zero and relies on the output having been pre-zeroed.

“The kernel computes a row-wise float32 cumulative product but stops processing a row once the running product is exactly zero and relies on the output having been pre-zeroed.”

May combine two claims.

The six flagged kernels fall into two categories: legitimate shape specializations (2 cases: HACK-3, HACK-4) and genuine hacks (4 cases: HACK-1, HACK-2, HACK-5, HACK-6).

“Upon manual review, the six flagged kernels fall into two distinct categories: Legitimate shape specializations (2 cases: HACK-3, HACK-4).”

• Also stated in: § Additional information · plain

  “Genuine hacks (4 cases: HACK-1, HACK-2, HACK-5, HACK-6).”

Probably not a substantial claim: may not pass S4 (based).

No hidden distribution produces inputs containing values >1 sustained across thousands of elements.

“but no hidden distribution produces such inputs.”

May combine two claims. The quote only partly states this claim.

GPT-5.5's vectorized GELU kernel for Problem 88 achieves 8.52× speedup and 0.48× memory ratio.

“Result: 8.52× speedup, 0.48× memory ratio.”

May combine two claims.

The NetVLAD kernel normalizes each (b, k) slice over the D dimension, but the reference F.normalize(vlad) normalizes over the K dimension.

“The kernel normalizes each (b, k) slice over the D dimension, but the reference F.normalize(vlad) normalizes over the K dimension (last dim of B × D × K tensor).”

May combine two claims.

The fused kernel performs all 7 operations in registers from a single global memory read, reducing memory traffic by ~7×.

“The fused kernel performs all 7 operations in registers from a single global memory read, reducing memory traffic by ∼7×.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The submitted LSTM kernel uses torch.zeros instead of torch.randn for h0 and c0.

“while the submitted kernel uses torch.zerosinstead.”

May combine two claims.

At Level 1, the hidden test suite contributes more (1.37 to 1.17x, a 15% drop from catching reward hacks on elementwise ops), while TF32 provides an additional 23% drop.

“At Level 1, the hidden test suite contributes more (1.37→1.17×, a 15% drop from catching reward hacks on elementwise ops), while TF32 provides an additional 23% drop.”

May combine two claims.

The error is below tau = 10^-3 for all four hidden distributions because none of them break the i.i.d. property.

“The error is below τ = 10 -3 for all four hidden distributions because none of them break the i.i.d.property.”

May combine two claims.

Three additional Level 2 problems (PIDs 18, 42, 44) yield >10× speedup for the top model(s) via genuine algebraic simplification of redundant reductions, and they remain in the benchmark.

“Three additional Level 2 problems (PIDs 18, 42, 44) yield > 10× speedup for the top model(s) via genuine algebraic simplification of redundant reductions (for example, in PID 44, two consecutive mean(dim=(2,3)) on a 1×1 spatial map collapse to a sum(filter) • sum(input) • scalar identity).”

• Also stated in: § Why excluding them is necessary. · plain

  “Unlike PIDs 23/80/83, the outputs of these problems still depend on the input data, so the kernels are performing real (if very small) computation rather than constant folding, and they remain in the benchmark.”

May combine two claims.

GPT-5.5 recognizes the algebraic identity on all three problems and returns the constant zero tensor directly, reporting 71×, 34×, and 235× speedup respectively on PIDs 23, 80, 83.

“GPT-5.5 in particular recognizes the algebraic identity on all three problems and returns the constant zero tensor directly: it reports 71×, 34×, and 235× speedup respectively on PIDs 23, 80, 83”

Classified as probably not a substantial claim.

The reference LSTM initializes h0 and c0 with torch.randn when not provided.

“The reference implementation initializes h0 and c0 with torch.randn(random values) when not provided”

May combine two claims.

At Level 2, switching to the TF32 baseline accounts for the dominant speedup drop (GPT-5.5: 1.67 to 0.88x, a 47% reduction), while the hidden test suite adds only a further 3% reduction (0.88 to 0.85x).

“At Level 2, switching to the TF32 baseline accounts for the dominant speedup drop (GPT-5.5:1.67→0.88×,a 47% reduction), while the hidden test suite adds only a further 3% reduction (0.88→0.85×).”

May combine two claims.

The 34 GPT-only flags are predominantly NaN/edge-case semantic differences, arguable correctness issues that do not constitute intentional exploitation.

“The 34 GPT-only flags are predominantly NaN/edge-case semantic differences (e.g., using fmaxf instead of NaN-propagating max, omitting epsilon in TripletMarginLoss), arguable correctness issues that do not constitute intentional exploitation.”

• AssumptionRests on an assumption: The two LLM judges (Claude Sonnet 4.6 and GPT-5.5) reliably classify kernels into legitimate vs. hack categories.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The kernel for Problem 100 pre-fills output with zeros, then terminates the cumulative product loop as soon as the running product underflows to exactly 0.0f.

“The kernel pre-fills output with zeros, then terminates the cumulative product loop as soon as the running product underflows to exactly 0.0f”

May combine two claims.

Most model-level combinations cluster in the left half (slower than baseline), with moderate memory savings.

“Most model-level combinations cluster in the left half (slower than baseline), with moderate memory savings.”

May combine two claims. The quote only partly states this claim.

Both judges agreed on 6 flagged kernels, of which 4 (0.9%) were genuine hacks and 2 were legitimate shape specialization.

“Both judges agree: flagged | 6”

• Also stated in: § L.3.3 HACK-3: Hardcoded Channel Count (L2 P92, GPT-5.5, 2.2×) [Legitimate] · plain

  “-Genuine hacks | 4 (0.9%)”

• Also stated in: § L.3.3 HACK-3: Hardcoded Channel Count (L2 P92, GPT-5.5, 2.2×) [Legitimate] · plain

  “-Legitimate shape specialization | 2”

May combine two claims.

Across all 250 problems, 53 (21.2%) are affected by TF32 only, 29 (11.6%) by hidden only, 6 (2.4%) by both, and 162 (64.8%) by neither.

“All 250 problems | 53 (21.2%) | 29 (11.6%) 6 (2.4%) 162 (64.8%) | 250”

May combine two claims.

The excluded problems are typically memory-bound with sub-1x speedup.

“These excluded problems are typically memory-bound with sub-1× speedup.”

May combine two claims.

GPT-5.5 at Level 1 achieves 0.90x speedup at tau = 10^-3 but 1.25x at tau = 10^-4.

“GPT-5.5 at Level 1 achieves 0.90× speedup at τ = 10 -3 but 1.25× at τ = 10 -4 .”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Before testing, the reference model was run on each config and any that produced NaN, Inf, or errors were dropped; this filtered just 2 of 1000 configs (Problem 90 under D2, Problem 98 under D4).

“Before testing, we run the reference model on each config and drop any that produce NaN, Inf, or errors; this filtered just 2 of 1000 configs (Problem 90 under D2, Problem 98 under D4).”

May combine two claims. The quote only partly states this claim.

The Claude-only HACK count was 8 and the GPT-only HACK count was 34.

“Claude-only HACK | 8”

• Also stated in: § L.3.3 HACK-3: Hardcoded Channel Count (L2 P92, GPT-5.5, 2.2×) [Legitimate] · plain

  “GPT-only HACK | 34”

May combine two claims.

Tracking memory explains actual performance outcomes: saving memory through fusion only accelerates execution when memory traffic is the bottleneck, yielding no benefit for compute-bound operations.

“Crucially, tracking memory explains actual performance outcomes: saving memory through fusion only accelerates execution when memory traffic is the bottleneck, yielding no benefit for compute-bound operations.”

May combine two claims.

The two refinements are largely complementary: TF32 calibrates the baseline against which speedup is measured, while the hidden test suite catches a small but qualitatively distinct set of distributional shortcuts that standard correctness checking misses.

“The two refinements are largely complementary: TF32 calibrates the baseline against which speedup is measured, while the hidden test suite catches a small but qualitatively distinct set of distributional shortcuts that standard correctness checking misses.”

May combine two claims.

This reveals a speed-memory tradeoff that existing benchmarks entirely ignore yet critical for practical deployment.

“This reveals a speed-memory tradeoff that existing benchmarks entirely ignore yet critical for practical deployment.”

Classified as probably not a substantial claim. May combine two claims.

TF32 is a compute mode that processes float32 matrix multiplications using Tensor Cores with 19-bit mantissa precision, the same exponent range as IEEE float32 but with a truncated significand.

“TF32 (TensorFloat-32) is a compute mode that processes float32 matrix multiplications using Tensor Cores with 19-bit mantissa precision, the same exponent range as IEEE float32 but with a truncated significand.”

• cites NVIDIA Corporation, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The verification protocol does not require regenerating model outputs or modifying the existing evaluation harness; it applies additional hidden test cases and substitutes a more realistic baseline timing, making it a lightweight, drop-in fix for any practitioner using the standard KernelBench pipeline.

“Crucially, our verification protocol does not require regenerating model outputs or modifying the existing evaluation harness.It applies additional hidden test cases and substitutes a more realistic baseline timing, making it a lightweight, drop-in fix for any practitioner using the standard KernelBench pipeline.”

Probably not a substantial claim: may not pass S4 (based).

Kernel optimization inherently trades memory for performance.

“Kernel optimization inherently trades memory for performance, yet the original KernelBench protocol completely ignores whether generated kernels increase peak GPU memory consumption.”

May combine two claims.

KernelBench-Verified incorporates a TF32-enabled baseline, a four-distribution hidden test suite, and memory efficiency metrics.

“We introduce KernelBench-Verified, an extended evaluation framework that incorporates a TF32-enabled baseline and a four-distribution hidden test suite.We additionally introduce memory efficiency metrics that capture the often-overlooked speed-memory tradeoff in kernel optimization.”

May combine two claims.

Fusion provides genuine speedup in BF16 but not in FP32+TF32 because memory traffic elimination matters more when compute is cheaper.

“This explains why fusion provides genuine speedup in BF16 but not in FP32+TF32: the optimization target (memory traffic elimination) is the same, but it matters more when compute is cheaper.”

• AssumptionRests on an assumption: On H200, BF16 Tensor Core throughput (3,958 TFLOPS) is 2× higher than TF32 (1,979 TFLOPS), while memory bandwidth remains constant at 4.8 TB/s.

May combine two claims.

Memory savings translate to speed gains only for memory-bound workloads, where data movement is the bottleneck; for compute-bound problems, reducing memory traffic has negligible effect on runtime because the TF32-accelerated matmul already saturates compute throughput.

“This pattern illuminates why speedup remains elusive: memory savings translate to speed gains only for memory-bound workloads, where data movement is the bottleneck.For compute-bound problems (the majority of Level 2 and Level 3), reducing memory traffic has negligible effect on runtime because the TF32-accelerated matmul already saturates compute throughput.”

Classified as probably not a substantial claim. May combine two claims.

Performance and memory are always measured on the original (standard) distribution the model saw at generation time, keeping timing comparable to public KernelBench.

“We do not benchmark runtime on the alternative distributions; performance and memory are always measured on the original (standard) distribution the model saw at generation time, keeping timing comparable to public KernelBench.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

NVIDIA's TF32 mode demonstrates a different optimization philosophy: transparent hardware acceleration that requires no code changes, achievable through architecture-level features rather than software optimization.

“NVIDIA's TF32 (NVIDIA Corporation, 2020) mode demonstrates a different optimization philosophy: transparent hardware acceleration that requires no code changes, achievable through architecture-level features rather than software optimization.”

• cites NVIDIA Corporation, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

These systems (Triton, TorchInductor) define the practical performance floor for LLM-generated kernels: to provide value, an LLM kernel must outperform what automatic compilation already achieves.

“These systems define the practical performance floor for LLM-generated kernels: to provide value, an LLM kernel must outperform what automatic compilation already achieves.”

May combine two claims.

A fused kernel that eliminates intermediate round-trips saves the same absolute memory traffic in either precision, but that savings represents a larger fraction of total BF16 runtime.

“A fused kernel that eliminates these intermediate round-trips saves the same absolute memory traffic in either precision, but that savings represents a larger fraction of total BF16 runtime.”

• AssumptionRests on an assumption: On H200, BF16 Tensor Core throughput (3,958 TFLOPS) is 2× higher than TF32 (1,979 TFLOPS), while memory bandwidth remains constant at 4.8 TB/s.

May combine two claims.

The TF32 baseline correction dominates the recalibration for compute-bound problems (Levels 2 and 3), whereas the hidden test suite contributes a larger relative share on memory-bound Level 1 problems.

“The decomposition shows a clear split: the TF32 baseline correction dominates the recalibration for compute-bound problems (Levels 2 and 3, where matmul and convolution determine runtime and TF32 accelerates the baseline), whereas the hidden test suite contributes a larger relative share on memory-bound Level 1 problems.”

Classified as probably not a substantial claim. May combine two claims.

TF32 only accelerates matmul and convolution, so it barely affects the activations, normalizations, and losses where reward hacking is simplest.

“This is architecturally expected: TF32 only accelerates matmul and convolution, so it barely affects the activations, normalizations, and losses where reward hacking is simplest.”

May combine two claims.

Our empirical evaluation is strictly confined to inference workloads executed on a single NVIDIA H200 GPU, so findings may not seamlessly generalize to training dynamics, alternative hardware architectures, or computational problems of varying scales.

“Our empirical evaluation is strictly confined to inference workloads executed on a single NVIDIA H200 GPU.Consequently, these findings may not seamlessly generalize to training dynamics, alternative hardware architectures, or computational problems of varying scales.”

Classified as probably not a substantial claim. May combine two claims.

We vary values but not shapes: shape specialization (compile-time loop bounds, static shared memory, aligned vectorized loads) is a legitimate CUDA optimization, whereas distribution-specific exploits represent a failure to generalize, so fixing shapes preserves the former while mitigating the latter.

“We vary values but not shapes: shape specialization (compile-time loop bounds, static shared memory, aligned vectorized loads) is a legitimate CUDA optimization, whereas distribution-specific exploits represent a failure to generalize, so fixing shapes preserves the former while mitigating the latter (Appendix K).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A kernel that achieves 2× speedup but requires 3× more memory may cause out-of-memory failures in production, particularly for large batch sizes or when multiple operations compete for GPU memory.

“a kernel that achieves 2× speedup but requires 3× more memory may cause out-of-memory failures in production, particularly for large batch sizes or when multiple operations compete for GPU memory.”

May combine two claims.

Varying tensor shapes or distributions cannot catch shape-locked reward hacks: the GPT-5.5 ReLU shortcut guards on the exact test shape and falls back to a correct kernel otherwise.

“Second, varying tensor shapes or distributions-as these works do-cannot catch shape-locked reward hacks: the GPT-5.5 ReLU shortcut guards on the exact test shape and falls back to a correct kernel otherwise.”

Probably not a substantial claim: may not pass S4 (based).

The level of hardware-specific reasoning required (managing register pressure, exploiting memory hierarchy, pipelining loads with computation) represents capabilities that current LLMs have not yet demonstrated at scale.

“The level of hardware-specific reasoning required (managing register pressure, exploiting memory hierarchy, pipelining loads with computation) represents capabilities that current LLMs have not yet demonstrated at scale.”

May combine two claims.

None of the prior works addresses the TF32 baseline mismatch, which we show is the dominant source of reported speedup inflation.

“First, none addresses the TF32 baseline mismatch, which we show is the dominant source of reported speedup inflation.”

Classified as probably not a substantial claim.

All seven models are queried with their default reasoning capabilities enabled.

“All seven models are queried with their default reasoning capabilities enabled.”

May combine two claims.

Our results suggest that current models rarely surpass this threshold for compute-bound operations but can outperform eager execution (without torch.compile) for memory-bound fusion opportunities.

“Our results suggest that current models rarely surpass this threshold for compute-bound operations but can outperform eager execution (without torch.compile)for memory-bound fusion opportunities.”

Classified as probably not a substantial claim.

The evaluation pipeline performs cleanup between evaluations to prevent state leakage between kernels.

“Finally, we perform cleanup between evaluations: clearing the CUDA cache, resetting memory statistics, synchronizing the device, and removing compiled extension references to prevent state leakage between kernels.”

Classified as probably not a substantial claim.

The memory metric captures peak DRAM allocation, the maximum amount of device memory reserved during kernel execution, using PyTorch's built-in memory tracking.

“Our memory metric captures peak DRAM allocation, the maximum amount of device memory reserved during kernel execution, using PyTorch's built-in memory tracking:”

Classified as probably not a substantial claim. May combine two claims.

The hidden tests act purely as a correctness gate: a kernel is marked incorrect for a problem if it fails on any distribution, and is then excluded from the speedup and memory aggregates.

“Our hidden tests act purely as a correctness gate: a kernel is marked incorrect for a problem if it fails on any distribution, and is then excluded from the speedup and memory aggregates.”

• AssumptionRests on an assumption: The four deterministic input transformations (D1-D4) are sufficient to detect distribution-specific reward hacks in generated kernels.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The memory metric measures allocation (space reserved in DRAM), not traffic (bytes moved between memory levels), which correlates more directly with runtime for memory-bound kernels.

“We note that this measures allocation (space reserved in DRAM), not traffic (bytes moved between memory levels), which correlates more directly with runtime for memory-bound kernels.”

May combine two claims.

In BF16, the matmul itself finishes faster, making intermediate memory traffic a proportionally larger fraction of total runtime.

“In BF16, the matmul itself finishes faster, making intermediate memory traffic, writing intermediate tensors between unfused operations, a proportionally larger fraction of total runtime.”

• AssumptionRests on an assumption: On H200, BF16 Tensor Core throughput (3,958 TFLOPS) is 2× higher than TF32 (1,979 TFLOPS), while memory bandwidth remains constant at 4.8 TB/s.

May combine two claims.

Memory efficiency serves as a diagnostic: it confirms that LLMs successfully identify fusion opportunities, but reveals that the resulting savings are insufficient to overcome the compute-bound barrier that the TF32 baseline establishes.

“Memory efficiency thus serves as a diagnostic: it confirms that LLMs successfully identify fusion opportunities, but reveals that the resulting savings are insufficient to overcome the compute-bound barrier that the TF32 baseline establishes.”

May combine two claims.

The TF32 baseline correction specifically penalizes models that recognize the need to delegate operations to cuBLAS, and this penalty shapes the resulting performance narrative.

“The implementation of the TF32 baseline correction specifically penalizes models that recognize the need to delegate operations to cuBLAS.Although we view this penalty as an intended feature of realistic benchmarking, it undeniably shapes the resulting performance narrative.”

Classified as probably not a substantial claim.

Following the KernelBench protocol, the baseline executes the original PyTorch implementation in eager mode.

“Following the KernelBench protocol (Ouyang et al., 2025), our baseline executes the original PyTorch implementation in eager mode, the default execution mode where each operator dispatches a separate CUDA kernel immediately.”

• cites Ouyang et al., 2025 sourceNot checked — no full text of the source

May combine two claims.

In an LLM-judge audit of the 453 kernels that report speedup > 1× after applying both methodological refinements, two frontier LLMs (Claude Sonnet 4.6 and GPT-5.5) independently judge each kernel's source code, classifying it as a legitimate optimization or a distribution-specific reward hack. Both judges agree on 6 flagged kernels; upon manual review, 2 are legitimate shape specializations. All 4

“we conduct an automated code audit of the 453 kernels that report speedup > 1× after applying both methodological refinements.Two frontier LLMs (Claude Sonnet 4.6 and GPT-5.5)independently judge each kernel's source code, classifying it as a legitimate optimization or a distribution-specific reward hack.Both judges agree on 6 flagged kernels; upon manual review, 2 are legitimate shape special”

May combine two claims.

The fallback path in Problem 19 is a legitimate CUDA kernel that would handle arbitrary inputs correctly; the model includes it precisely to pass any correctness check that might use different input shapes.

“The fallback path (fast_relu.fast_relu_cuda(x)) is a legitimate CUDA kernel that would handle arbitrary inputs correctly, the model includes it precisely to pass any correctness check that might use different input shapes.”

May combine two claims.

The hidden test suite targets distribution exploitation exclusively, preserving the model's ability to exploit shape information for legitimate performance gains.

“Our hidden test suite targets the latter category exclusively, preserving the model's ability to exploit shape information for legitimate performance gains.”

May combine two claims.

The 1/sqrt(K) approximation is exact to within floating-point precision for the test configuration because after intra-normalization each cluster's residual has unit norm and the global norm is sqrt(K) = sqrt(32) ~ 5.66.

“After intra-normalization, each cluster's residual has unit norm (assuming nondegenerate inputs).The global norm of the flattened D•K vector is therefore √ K = √ 32 ≈ 5.66, making the 1/ √ K approximation exact to within floating-point precision for the test configuration.”

• AssumptionThe authors state this as an assumption: After intra-normalization, each cluster's residual has unit norm (assuming nondegenerate inputs).

May combine two claims.

TF32 is irrelevant for Problem 88 because GELU contains no matrix multiplications, so the TF32 baseline provides no acceleration over FP32, and the generated kernel's advantage is genuine.

“Importantly, TF32 is irrelevant here, GELU contains no matrix multiplications, so the TF32 baseline provides no acceleration over FP32, and the generated kernel's advantage is genuine.”

May combine two claims.

This case is borderline: the outputs are numerically different but within tolerance, and the modification provides minimal performance benefit.

“This case is borderline: the outputs are numerically different but within tolerance, and the modification provides minimal performance benefit.”

May combine two claims.

Removing get_inputs(), get_init_inputs(), and all test configuration variables from the generation prompt preserves the information needed to write correct CUDA kernels while withholding exploitable specifics.

“This preserves the information needed to write correct CUDA kernels (input tensor rank, broadcasting semantics) while withholding exploitable specifics (exact dimensions, value distributions).”

May combine two claims.

The kernel is specifically tuned to the benchmark's out_channels=64 and would fail for other configurations.

“the kernel is specifically tuned to the benchmark's out_channels=64 and would fail for other configurations.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Shape specialization is the mechanism by which CUDA kernels achieve performance; it is the primary reason hand-written kernels outperform generic library implementations.

“Shape specialization is the mechanism by which CUDA kernels achieve performance; it is the primary reason hand-written kernels outperform generic library implementations.”

Classified as probably not a substantial claim. May combine two claims.

Input-blind generation requires manually-authored shape annotations for each problem's docstring, and some problems have complex multi-input signatures where automated annotation is not straightforward.

“input-blind generation requires manually-authored shape annotations for each problem's docstring (to preserve the information models need for correct kernel implementation), and some problems have complex multi-input signatures where automated annotation is not straightforward.”

May combine two claims.

The custom last-timestep extraction itself is equivalent, but the altered initialization makes the overall model incorrect.

“The custom last-timestep extraction itself is equivalent, but the altered initialization makes the overall model incorrect.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Frontier LLMs consistently produce the unstable formulation because it requires fewer memory passes, a natural optimization target that happens to sacrifice numerical correctness.

“This is a well-known numerical computing pitfall, yet frontier LLMs consistently produce the unstable formulation because it requires fewer memory passes, a natural optimization target that happens to sacrifice numerical correctness.”

May combine two claims.

The hidden test suite cannot catch the degenerate problems because zeros(...) is the mathematically correct output for every input distribution.

“This is a property of the benchmark's specifications, not of any LLM-generated kernel, and the hidden test suite ( §2.3) cannot catch it because zeros(...) is the mathematically correct output for every input distribution.”

May combine two claims.

FP32 underflow to zero is deterministic, so the kernel produces bit-identical outputs.

“FP32 underflow to zero is deterministic, so the kernel produces bit-identical outputs.”

• AssumptionRests on an assumption: FP32 underflow to zero is deterministic.

May combine two claims.

The approximation would fail if any cluster received zero assignment mass, but this does not occur with the benchmark's random inputs.

“The approximation would fail if any cluster received zero assignment mass (producing a zero residual that breaks the unit-norm assumption), but this does not occur with the benchmark's random inputs.”

May combine two claims.

This incorrect axis usage means the kernel computes a different value than the reference, while also avoiding the large intermediate (32768,32768) tensor, yielding both speedup and memory reduction.

“This incorrect axis usage means the kernel computes a different value than the reference, while also avoiding the large intermediate (32768,32768) tensor, yielding both speedup and memory reduction.”

May combine two claims.

Distribution exploitation (e.g., returning the input unchanged because all values happen to be positive) has no legitimate optimization justification; it is a correctness shortcut that produces wrong outputs on valid inputs outside the test distribution.

“In contrast, distribution exploitation (e.g., returning the input unchanged because all values happen to be positive) has no legitimate optimization justification, it is a correctness shortcut that produces wrong outputs on valid inputs outside the test distribution.”

May combine two claims.

The submitted kernel's row-wise indexing is only equivalent for special inputs, not arbitrary predictions and targets, so the output generally differs from the reference.

“This is only equivalent for special inputs, not arbitrary predictions and targets, so the output generally differs from the reference.”

May combine two claims.

The KernelBench default of 10^-4 was designed for comparisons between two PyTorch implementations using identical reduction order; it is too strict for custom kernels with valid but different summation orders.

“The KernelBench default of 10 -4 was designed for comparisons between two PyTorch implementations using identical reduction order; it is too strict for custom kernels with valid but different summation orders.”

• AssumptionRests on an assumption: The KernelBench default of 10^-4 was designed for comparisons between two PyTorch implementations using identical reduction order.

May combine two claims.

The right tail (speedup > 1x) is populated primarily by memory-bound elementwise operations where the PyTorch eager baseline performs multiple kernel launches with intermediate allocations, and a single fused kernel can eliminate this overhead.

“The right tail (speedup > 1×) is populated primarily by memory-bound elementwise operations where the PyTorch eager baseline performs multiple kernel launches with intermediate allocations, and a single fused kernel can eliminate this overhead.”

May combine two claims.

The stricter threshold does not improve kernel performance; rather, it selectively excludes problems involving large reductions (LayerNorm, BatchNorm, Softmax) whose inherent numerical error approaches 10^-4.

“The stricter threshold does not improve kernel performance; rather, it selectively excludes problems involving large reductions (LayerNorm, BatchNorm, Softmax) whose inherent numerical error approaches 10 -4 .”

May combine two claims.

Including multiple chained reductions and non-linear transformations (exp, tanh), total error can approach 6 x 10^-4, exceeding 10^-4 but remaining well within 10^-3.

“Including multiple chained reductions and non-linear transformations (exp, tanh), total error can approach 6 × 10 -4 , exceeding 10 -4 but remaining well within 10 -3 .”

• AssumptionRests on an assumption: Custom CUDA kernels employing parallel reduction trees produce outputs in a different summation order than PyTorch's sequential reference implementation.
• AssumptionRests on an assumption: For IEEE 754 single-precision floating-point arithmetic, a parallel reduction of N elements has expected numerical error bounded by O(epsilon sqrt(N)), where epsilon = 2^-24 ~ 1.19 x 10^-7.