Interesting paper from Sakana AI on memory for recurrent models.

Recurrent models are good at state tracking, but they usually keep everything in one hidden vector, so short-term computation and long-term storage compete for the same space.

The Continuous Memory Machine gives the model two memory matrices. One short-term memory tracks recent neuron activity, and the other long-term memory stores information for later steps. A Transformer reads and writes both at every step.

It builds on Sakana's Continuous Thought Machine and beats LSTM, DNC, RMC, and CTM baselines on copy, associative recall, sorting, few-shot regression, and maze solving. It also generalizes to longer inputs than earlier memory-augmented networks.

The attention maps show the model uses long-term memory for algorithmic and in-context tasks and skips it when the task does not need it.

Paper: https://academy.dair.ai/papers/continuous-memory-machines-2610.07907