Verifier finds 39.5% of LLM-generated GPU kernels are broken
The case for letting AI write GPU kernels just met its sharpest audit yet: a new verifier argues that the field's standard correctness test is so loose it waves through broken code most of the time.
A team of researchers built a contract-grade verifier for LLM-generated GPU kernels and found that 39.5 percent of machine-generated kernels a public system's own harness had accepted as correct are broken beyond any tolerance argument — and 62.1 percent carry at least one violation. The paper's premise is blunt: the reported high correctness rates come from a single loose test — run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference. A kernel can pass that and still be silently wrong, returning an ordinary number where the true answer is a NaN or an infinity, differing from run to run, breaking when the shape changes, or accumulating in fp16 where the reference keeps an fp32 total.
The verifier checks twelve adversarial properties, several of them tolerance-free, so no threshold choice can explain a failure away. Auditing 2,638 machine-generated kernels, it rejects 1,487 that the standard test accepts — against only 14 the other way. The authors defend the finding four independent ways: a 7/7 positive control, a threshold-calibration sweep, 98.5 percent agreement with the reference benchmark's own correctness code, and a stratified hand-audit. And to prove the verifier isn't just a wall of no, they used it to validate their own kernel: a native Blackwell tcgen05 training backward pass for the gated-linear-recurrence family, checked against a double-precision oracle and used to train five models.
Why it matters: the "AI writes CUDA" pitch is only as good as the checks it passes, and this paper says the checks are the weak link — the correctness signal behind reported progress in kernel generation is far weaker than the numbers suggest. For teams shipping LLM-written kernels into production, a loose eval isn't a formality; it's a silent failure mode in the making. Expect verification tooling to become the next battleground in the agentic-coding stack.
What to watch: whether kernel-generation systems adopt tolerance-free contracts as a standard eval — and how their advertised pass rates survive the stricter test.
If model-generated code passes today's tests yet ships broken kernels, what should count as "correct" before it lands in production? Tell us in the comments.
Sources: arXiv paper · Hacker News discussion
