You can teach a language model a new modality without ever touching the backbone's weights.

Train only the projector between encoder and backbone, and the capabilities you already shipped cannot regress.

To add a new modality to a language model, you only need to train the small projector that maps the encoder's output into the model's embedding space — fine-tuning the language model itself adds nothing reliable and wrecks its existing skills.

In their 3D tests, frozen-backbone models matched or beat jointly fine-tuned ones while training twice as fast, and the jointly fine-tuned Llama backbone collapsed from 86.96% to 0.61% on GSM8K.

– arxiv. org/abs/2608.19726

Title: "Projector Is All You Train"