A model can know the answer and still fail because you asked the same question differently.

New IBM paper introduces BenchDrift, an auditing framework for existing benchmarks, that systematically rephrases the same questions without changing their answers, then measures how much a model's score moves purely because of wording.

Across 8 models and 3 benchmarks, the gap between best-case and worst-case accuracy averaged 74.7 percentage points.

Stronger models were actually more exposed.

For every model-benchmark pair above 60% baseline accuracy, rephrasing broke more correct answers than it recovered.

And confidence did not solve this.

Even in the highest-confidence bucket, 18.5% of correct answers were lost after meaning-preserving rewording.

So if your model selection depends on small score differences, test several equivalent phrasings or report the range.

– arxiv. org/abs/2608.11694

Title: "The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance"