Great paper from Alibaba and Cornell on verification in coding-agent harnesses.

(bookmark it)

If you build agent harnesses, this one is worth your time.

They find that a separate reviewer agent adds 5.9 to 9.5 points on Terminal-Bench at 1.13 to 1.25x the baseline cost.

They compared 17 verification designs across seven frontier coding models on Terminal-Bench and SWE-bench Verified. The designs check the agent's work at four points: the initial instructions, reminders during the run, a gate before the agent can finish, and an independent critic at the end.

Instructions and reminders gave small, model-dependent gains. Completion gates gave larger gains, sometimes at much higher cost. The critic gave the best accuracy per dollar, adding 3.9 to 6.6 points on SWE-bench Verified at almost no extra cost. More review rounds added up to 3.5 more points on Terminal-Bench.

https://arxiv.org/abs/2610.34577

Chat with Paper: https://academy.dair.ai/papers/rethinking-verification-in-agent-harnesses-2610.34577