A skill-enabled agent can look better overall while performing worse on the exact tasks where it retrieved skills,

This paper finds a nasty evaluation failure: positive retrieval gains can hide negative performance on the very tasks where retrieval happened.

Most evaluations make a flawed comparison: they check whether tasks where the agent chose to retrieve a skill scored better than tasks where it chose not to.

But those may be completely different kinds of tasks. If the agent tends to retrieve skills on easier problems, retrieval will look helpful even if the skill itself did nothing, or even made the answer worse.

This paper’s fix, RAE, is simple: when retrieval happens, rerun that exact task without skill access and compare the result.