Me: Mechanistic interpretability sounds like a good idea, yes. If we could learn something about how an AI was thinking, in enough detail to predict in advance what it would think even as the thought departed from human, that would be a bare beginning on the sort of understanding you'd needed to get started on AI alignment that wasn't just about hammering on an AI from the outside with SFT and RL. AI guys: We found a neuron that lights up when the AI thinks of the Eiffel Tower! Me: Cool. That does sound like it'd be a prerequisite to getting started on the important part eventually, someday. AI guys: Oh NO! There are MULTIPLE neurons that light up when the AI thinks about the Eiffel Tower! And some of those neurons also participate in OTHER concepts! We're calling this unexpected development 'polysemanticity' and 'superposition'. Me: That sure sounds like a thing that would happen, yes. I cannot say that I am impressed by you acting surprised about it. Let me know when you've gotten far enough in mechanistic interpretability to get started on the important part. AI guys: We have found that with BIG AIs you can often point to a SINGLE neuron that lights up when the AI thinks of the Eiffel Tower! Understanding AI thought processes is gonna be EASY PEASY! Me: Cool. Let me know when you've advanced to the point of getting started.