KernelBench

Properties

axis: coding
verification: det-test
whatItTests: 250 PyTorch ML workloads where an agent must generate a GPU kernel that is both functionally correct and faster than a baseline by an adjustable threshold, scored via the fast_p metric.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: fast_p jointly scores correctness and speedup, and the paper shows iterative refinement using execution/profiling feedback improves results, a direct build-verify-reflect analog where the 'test' is a performance threshold rather than a pass/fail unit test. Evaluation in the paper is single-agent only.
source: KernelBench: Can LLMs Write Efficient GPU Kernels? (arXiv:2502.10517)
notes: Distinct from every other proposed coding benchmark: verification is a joint correctness-and-performance oracle (execution plus profiling), not just pass/fail, and the domain is low-level GPU kernel generation rather than application-level code. The four-level task breakdown reported by third-party summaries is not in the fetched abstract, so it is omitted here.

axis: coding
verification: det-test
whatItTests: 250 PyTorch ML workloads where an agent must generate a GPU kernel that is both functionally correct and faster than a baseline by an adjustable threshold, scored via the fast_p metric.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: fast_p jointly scores correctness and speedup, and the paper shows iterative refinement using execution/profiling feedback improves results, a direct build-verify-reflect analog where the 'test' is a performance threshold rather than a pass/fail unit test. Evaluation in the paper is single-agent only.
source: KernelBench: Can LLMs Write Efficient GPU Kernels? (arXiv:2502.10517)
notes: Distinct from every other proposed coding benchmark: verification is a joint correctness-and-performance oracle (execution plus profiling), not just pass/fail, and the domain is low-level GPU kernel generation rather than application-level code. The four-level task breakdown reported by third-party summaries is not in the fetched abstract, so it is omitted here.