The core problem is credit assignment.

A long-horizon rollout may take tens of hours, while a single terminal reward must be attributed across thousands of interactions.

This is where value-free RL methods such as GRPO begin to struggle.