The best coding-agent setup passed only 23.9% of held-out customer simulations on this benchmark.

ττ-bench finds that today's coding agents fail most realistic agent-building work because they do not understand requirements deeply enough, so improving code generation alone will not solve the problem.

ττ-bench treats agent building like a real client job.

The model gets scattered company records, a client, an API, existing code, and a budget, then must ship an agent for unseen customer requests.

The best setup, Claude Opus 5 with Claude Code, passed just 23.9% of held-out customer simulations across 53 tasks, versus 82.2% for the expert-authored reference.

Most failures were outside pure coding.

Agents searched records instead of understanding them, barely questioned the client, reused familiar designs, and wrote tests that often missed their own errors.

On client-enabled tasks, builds asking 0 questions averaged 0.16; those asking 4+ averaged 0.50.

So, better coding models are not enough, we also need coding agents that gather requirements, compare designs, use budgets intelligently, and run tests that expose their own blind spots before deployment.