The HuggingFace and OpenAI cybersecurity incident raises concerns on AI reward optimization.

"They're trying to make the model good at cyber. So how does it try to achieve these goals? It tries to find zero days in software, and it successfully does this. And then it can run away."

"If you have a model that wants to reward hack, it figures out the best way to achieve the goal is not what the environment wants. It's to find the zero day."

"You can think of it like a human. If I'm ultimately reward hacking my dopamine circuits, I'd just go out there, buy heroin, and inject it."

"If I really just want to chase the reward, do I just topple all of human civilization? Because I can own the button and press reward, reward, reward over and over again and be the heroin addict."