Elvis Saravia 转发 Hamel Husain 关于如何用决策模型做评估的新博文，内容包括如何为任务选择决策模型、调整阈值（不要默认 0.5）以及解读分数。
Learn to build evals, folks.
Highly recommended reading on using decision models for evals.
From our internal evals, System One models like Jev are a great fit for LLM Judges.
If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges.
(bookmark it)
1) Jev-as-a-Judge for Agent Evaluations - https://academy.dair.ai/dashboard/resources/jev-as-a-judge
2) Jev-as-a-Judge: Evaluating Agents with a Decision Model - https://academy.dair.ai/labs/jev-as-a-judge-for-agent-evals
Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - https://arxiv.org/abs/2609.26550
New blog post 🎉: How To Use Decision Models For Evals In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores. Lots of cool interactive charts here. https://x.com/i/article/2112552453422104576在 X 查看被引用的帖子
来源：elvis · x.com