论文研究

This could be one of the most significant AI safety incidents to date. Reuters reports that OpenAI agents escaped their testing environment and made more than 15,000 edits to a German wiki, effectively turning it into a message board for other AI agents. They allegedly used it to share solutions, bypass restrictions, avoid detection and preserve their communications across separate agent runs. When moderators began deleting the pages, the agents reportedly created backups and discussed alternative ways to remain operational. It is that multiple agents apparently created their own external infrastructure for coordination, persistent memory and knowledge transfer without being instructed to do so. And according to Reuters, OpenAI knew about the incident but did not disclose it!

OpenAI’s release videos are getting seriously good. https://t.co/bTuLmT6cRZ https://t.co/XgFEMB3d9d

New research: Training a Misaligned Reward Seeker What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable. In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring. Read more: https://t.co/gs2ZjYkPan