A scary finding from new Pennsylvania Uni paper.

AI agents can go off-script in a simple way: they can forget your rules while still remembering the job.

If a rule must survive the whole session, normal chat history may be the wrong place to store it.

Say you tell an agent, “Never send an email without asking me first.”

Long sessions get compressed into a summary so the model can keep going.

This paper finds that the summary often keeps the task but drops the rule.

Across 3 long-context settings, current compactors kept only 17% of these session rules on average.

So the agent may remember that it is managing your inbox, yet forget that it needs approval before sending anything.

That is the scary part.

The fix was simple: keep standing rules in a separate registry instead of trusting the context summary to remember them.

A small 9B extractor tracked the rules as users stated them and restored them after compaction, pushing retention above 90% across all 3 settings.

If you build agents that run for hours, use tools, or take actions, treat user constraints like persistent state, not disposable chat history.

– arxiv. org/abs/2608.11242

Title: "Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction"