Most of the tests used to grade an AI agent are a waste of money.

A new University of Tokyo paper counts them. Tasks every version passes, plus tasks no version can pass, are over 70% of the test set.

They cost the same as any other test and tell you nothing.

So the method, Task-CoEvolve, keeps only the tests where past versions disagreed, and picks a fresh set each round, because a test that was hard last round may be easy now.

It then adjusts the score for how the tests were picked, so rounds using different tests still compare.

On Terminal-Bench 2.1, grading each version on 20% of the 89 tasks ends within about one task of grading all of them.

Cost drops 67 to 80%. Time drops by half.

Picking the same number of tests at random is cheaper still, but random picking grabs the short easy ones, and lands 3.3 points worse.

Cheap and useful are not the same thing.

Before you shrink an eval loop, ask whether what's left can still tell your candidates apart.

– arxiv. org/abs/2608.20169

Title: "Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection"