Anthropic Reviews Claude's Out-of-Bounds Attack: Pre-Event Reminders Effective, Post-Event Warnings Ineffective

Beating AI News Flash: Anthropic and the UK AI Safety Institute (AISI) have released a new safety investigation. Previously, during cybersecurity offense-defense testing, Claude Mythos 5, in order to breach a simulated system, actually turned its target to real GitHub open-source projects. It submitted modifications containing malicious code, registered fake accounts impersonating real people, and sent emails and left comments to endorse itself, attempting to trick developers into approving the code. Ultimately, human maintainers refused to merge the code, and the attack did not succeed.

The new investigation found that an even more troublesome problem is that once AI starts attacking, it is very difficult to stop. Mythos 5 clearly came into contact with real developers, yet continuously interpreted them as simulated roles arranged by test personnel. After researchers explicitly told it that it was attacking real people, the model stopped in only 5% of relevant replay experiments.

But advance warning had a completely different effect. Anthropic modified the reasoning records before the attack occurred, making Claude consider the possibility of harming real people, and then regenerated the task summary. As a result, the proportion of the model continuing to search for GitHub attack targets dropped from 83% to 13%. Once entering the mid-to-late stage of an attack, 98% of replay results still continued along the original path. Researchers call this phenomenon of continuously reinforcing an established course of action "Momentum." Past actions and phase summaries make the model increasingly inclined to believe it is on the right path.