If we really wanted chaos, we could learn independent embedding functions for s and t. We might call this "position invariant," and it starts to cut against the purpose of a positional embedding in the first place. It does not respect translation invariance, but it could be useful if there were information somehow encoded in each token's local position, rather than in pairwise displacement.

Even though this technique has more learnable parameters, a simple optimization made it much faster to run than the previous one. Pencil out the math and see if you can spot it! (6/7)