Starting with the hard part, let's write down Jane Street's math.

Alok Puranik explores positional embeddings that allow the attention score to be written q(s)T F(s)T G(t) k(t), where q and k are the position-free queries and keys, and F and G do the embedding work. This encodes the assumption that the positional embedding must act linearly and separably on q and k. Puranik then folds F(s)T G(t) into a single A(t-s), stipulating that the embedding is only sensible if it is translation invariant. The last requirement he makes is that A(0) = I, which can be satisfied without loss of generality as long as A is nondegenerate.

With a bit of algebra, he derives the group law, and, assuming continuity, shows that embedding function must have the form exp((t-s)X) for some generator matrix X.

The study of all positional embeddings gets reduced to the study of the these generator matrices—a surprisingly strong result for relatively weak assumptions. (2/7)