Are decision models useful for reranking?

This new paper compares decision models with fine-tuned rerankers in a controlled experiment.

Before you replace your reranker with a general decision model, run your current reranker and the decision model on your own queries and compare their rankings.

The paper uses the same backbone model and the same candidate passages for every version it compares.

The authors tested six Qwen backbones from 0.6B to 27B.

Every version reranked the same BM25 top-100 candidates on ten collections. Five come from TREC DL 2019-2023, and five come from BEIR. They compared three versions of each backbone. The first is the untuned model asked zero-shot. The second is a decision fine-tune. The third is a plain reranker fine-tune.

From 0.6B to 9B, the reranker fine-tune ranks better than the other two on every backbone by 5 to 29 points in nDCG@10. The gap is statistically significant on 48 of 60 comparisons. The reranker fine-tune also wins on collections that none of the fine-tunes were trained on.

At 27B, the reranker fine-tune and the decision fine-tune score about the same. No reranker fine-tune had been released at that size, so the authors trained one.

Up to 9B, decision fine-tuning stays within 2.5 points below and 0.7 points above the untuned model asked the same question. Reranker fine-tuning adds 6 to 30 points.

Decision models are mainly promoted for their calibrated probabilities. The paper shows that the decision models' calibration depends on how the question is worded. When asked whether a passage answers the query, decision models from 4B up are better calibrated than published reranker fine-tunes. With a slightly different question, taken from prior work, the decision models are no longer better calibrated.

Better calibration also does not fix the ranking. Up to 9B, a reranker fine-tune keeps better passages at a fixed threshold.

https://arxiv.org/abs/2610.32318

Chat with Paper: https://academy.dair.ai/papers/system-one-models-are-not-better-rerankers-2610.32318