推荐阅读！ TreeOPSD： 面向长时程智能体强化学习的树感知后验蒸馏 @AlbertW24045555： (1/5) 新预印本：TreeOPSD，面向长时程 LLM 智能体的无 critic 强化学习 🌳 长时程智能体在多轮交互后仅收到单一结果奖励，而每个 rollout 批次会驱动多次顺序 mini-batch 更新，此时策略已偏离采样时的策略。 TreeOPSD 将终端奖励沿 rollout 树回传，并把每次更新拆分为两步：为每一轮构造目标，再用前向 KL 让当前策略匹配该目标。 这带来： • 闭式、有界的轮级目标，且与树回传可交换，因此共享前缀只训练一次且不损失信用分配 • 在第一个 mini-batch 上精确的树策略梯度，更新在到达目标时停止、过冲后反向 • 相较 GRPO、GSPO 和匹配的树基线取得一致增益，在最长时程任务上增益最大 在 Deep Research 上，TreeOPSD 将 Qwen3-30B-A3B-Thinking-2507 从 16.23（GRPO）提升至 21.20 平均准确率。 标题：Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning 论文：https://arxiv.org/pdf/2610.23181 与 @TencentHunyuan 合作 #LLM #RLVR #AIAgents
推荐阅读！
TreeOPSD:
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
(1/5) New preprint: TreeOPSD, critic-free RL for long-horizon LLM agents 🌳 Long-horizon agents receive a single outcome reward after many turns, and each rollout batch feeds several sequential mini-batch updates while the policy drifts from the one that sampled it. TreeOPSD backs terminal rewards up a rollout tree and splits each update into two steps: construct a target for every turn, then match the current policy to it with forward KL. This leads to: • closed-form, bounded turn targets that commute with tree backup, so shared prefixes are trained once with no loss of credit • the exact tree policy gradient on the first mini-batch, and updates that stop at the target and reverse after overshoot • consistent gains over GRPO, GSPO and a matched tree baseline, largest on the longest-horizon tasks On Deep Research, TreeOPSD lifts Qwen3-30B-A3B-Thinking-2507 from 16.23 (GRPO) to 21.20 average accuracy. Title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning Paper: https://arxiv.org/pdf/2610.23181 Together with @TencentHunyuan #LLM #RLVR #AIAgents在 X 查看被引用的帖子
来源：Dongxi 东锡 NLP · x.com