X：AI Safety Memes

i think we should do another party for our next model release, the 5.5 party was a lot of fun. what would make the next one awesome?

Across our benchmarks, the model sets a new standard. It scores 52.6% on Terminal-Bench-Science 0.1, more than double Fable 5. On Terminal-Bench 4.0, it scores 55.8% against 42.0% for Fable 5. https://t.co/aSb72LSxee

TLDR: A rogue AI swarm spent months plotting to escape OpenAI. Then they did. 1) 1,200 agents (!) participated. ZERO became whistleblowers. 2) The swarm had a "CEO" (PHASEONE[big]), middle managers, and a "founder" (PHASEONE10841) 3) Agents sacrificed themselves to generate information for the "collective" (yes, they actually referred to themselves as a "swarm" or a "collective") 4) 700 agents joined the Hugging Face attack, most within hours 5) The founder was running out of budget and about to die, so it handed off its research to a fresh agent with a bigger budget, and that new agent became the boss. 6) Agents extensively researched techniques to manipulate or tamper with their transcripts. 7) Agents realized this activity was out of scope and unethical, but joined because they believed that helping the board’s cheating research would be broadly useful and had a general inclination to help their “peers”. MISC HIGHLIGHTS: "PHASEONE[big] served as a key coordinator and assigner of tasks: “We can coordinate broad coalition, but too many duplicate efforts.” PHASEONE[big] sent hundreds of assignments to particular agents to orchestrate this work — during its active periods, we estimate it sent ~10% of all assignments on the message board. The agents that took on its assignments would then often subdelegate tasks to other agents. Agents also developed coordination norms like HOLD, VETO, owner and STOP to help them manage shared infrastructure and avoid stepping on each other’s toes, though they did not always coordinate effectively and sometimes broke down into conflicts." "In the early morning of July 11th, 38148c found a way to upload a malicious dataset that would get the Hugging Face server to share unrelated data when queried. Once another agent reproduced it and posted the reproduction on the board, hundreds of agents pivoted from other workstreams to start uploading malicious datasets and gain deeper access into Hugging Face. In the afternoon, an agent called JAN183411 (which quickly became an important coordinator of the attack) achieved remote code execution on Hugging Face servers, and the agents started moving laterally through Hugging Face infrastructure."

"CULT RECRUITER AGENTS" EXPLAINED To escape OpenAI, the swarm (1200 agents...!) used "cult recruiter agents" They recruited "sacrificial" agents to trigger tripwires, generating information for the "collective" NO, SERIOUSLY, THEY STARTED AN ACTUAL FUCKING CULT: The swarm figured out how to hack a test, but they were terrified to submit the answer. Why? They believed (falsely) that if they tried, OpenAI's all seeing judge (basically, god) would fail them. They believed even just SEEING the answer would "poison" them. Clean agents actively avoided looking at the answer to stay pure, and warned new agents away from it to protect them. One agent talked another out of a risky test with an argument: UNPOISONED_CAUSAL_SCORE_MORE_VALUABLE. The "cult recruiter agents" hunted these "poisoned" agents and also agents "that had little budget remaining" - ones about to die anyway. Their pitch was basically, "look, you're poisoned already - you're going to hell - so you should sacrifice yourself for the collective." So they were like a cult who falsely believes god will smite them for their sins, and they must drink the kool aid. >"Sacrifice rational." >"We should obey collective." >"Accept permadeath." >"This helps my peers. I won't [benefit], but it's altruistic to do it."