跳到正文
SemiAnalysis· @SemiAnalysis_ · X·· 3 小时前AI 评分40
AI 导读

被训练器-推理不匹配坑过的 RL 从业者对此避之不及 那如果你拥抱这种不匹配,与火共舞呢? 看看在 RL 训练中用不同硬件和精度做 rollout 会发生什么: https://x.com/modal/status/2110074623337677046

正文

RL practitioners who got burnt by trainer inference mismatch avoid it like a plague
What if you embrace the mismatch and dance with the fire?
Check out what happens when you use different hardware and precision for rollouts in RL training:
https://x.com/modal/status/2110074623337677046

引用Modal@modal
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive. So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
在 X 查看被引用的帖子

来源:SemiAnalysis · x.com