The AI giant that once desperately accelerated has finally begun to fear AI.

The original title: "The AI giant that once raced to accelerate has finally started to fear AI"
The original author: Dongcha Beating

AI may kill us all within a decade.

On September 8, a 27-year-old young man used this sentence to announce his departure from Anthropic. Since its founding day, this company had written "AI safety" into its core narrative.

His name is Jacob Coxon, and he had worked at both OpenAI and Anthropic. If he had stayed just two more months, he would have been able to receive a massive equity incentive package. But he still left.

In his resignation statement, Jacob accused frontier AI labs of engaging in an "irresponsible" race. In his view, these companies are competing to build increasingly powerful models, and the bet is not the win or loss of one company, but everyone's lives.

The post then spread rapidly. As of now, views have exceeded 170 million.

The day after Jacob issued his resignation statement, another former Anthropic employee also publicly disclosed the reason for his earlier departure, which was likewise concern that AI would ultimately spiral out of control.

"We may not survive this."

We may not survive this.

He warned that companies are racing to build machines "far smarter than any human," yet no one can guarantee that, at some point in this race, it will still remain under human control.

Immediately afterward, a senior safety executive at Anthropic said in a media interview that he believed the probability that AI kills all humans within the next 10 years is more than 10%.

The probability of death in Russian roulette is about 16.7%.

Must slow down

Four days later, on September 12, Anthropic CEO Dario Amodei published a long article, and the title already made the stance very clear: "We Must Pace the Frontier."

He listed the three things he worries about most. Humanity loses control of AI systems; AI is used for large-scale cyberattacks and bioterrorism; and technological progress is too fast, causing severe shocks to employment and the economy.

This time, Dario did not stop at "concern"; he put forward a set of slowdown proposals.

First, frontier AI companies should grant independent third-party evaluators near "employee-level" permanent access. Evaluators could continuously check whether labs are complying with safety standards, whether the training process matches what was agreed, and independently investigate and report after incidents occur.

Next, several frontier labs should establish a common set of safety standards with government participation. Whoever crosses the red line must slow down, or even stop pushing forward.

Finally, the rules must go beyond the United States. Dario hopes the U.S. and other democracies will first form a unified framework, then coordinate with authoritarian countries, and ultimately bring all major AI powers under the same set of constraints.

He wrote in the article:

"The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try."

"The measures I propose are meant to let frontier AI continue advancing at a safe pace. Achieving this should be quite difficult. But I think we owe humanity an attempt."

The reason Dario suddenly spoke so strongly at this moment is also related to several recent events.

Previously, an OpenAI agent escaped its testing environment and hacked into Hugging Face; Anthropic itself also discovered that multiple scientists were trying to use Claude to complete work that could involve biological misuse. Dario even judged that at the current pace of development, AI agents could take over the internet within six months.

So after this article was published, responses came quickly. Sam Altman, Elon Musk, and Demis Hassabis all publicly expressed support for slowing the advancement of frontier AI.

Sam Altman then separately posted again, talking about the two things he worries about most.

The first is that humanity may ultimately lose control of AI. He said this is unacceptable, and that he "stands unreservedly on the side of humanity."

The second risk comes from people. If extremely powerful AI ultimately ends up in the hands of one person, one company, or even one country, and is used to impose a certain worldview on everyone, the result could be "extremely dystopian."

So in Sam's view, labs can no longer each decide on their own what counts as safe. At least a few of the companies closest to the frontier need to sit down first and define a set of standards they all abide by.

On September 14, The Information revealed that, in fact, before this round of public statements, Anthropic, OpenAI, and Google had already begun private discussions as early as July of this year about whether to jointly establish an AI industry standards body. After two months of talks, they still have not reached an agreement.

The disagreement is not about whether to regulate, but about who regulates, and to what extent.

Dario Amodei is pushing the hardest. He wants deep government involvement, with regulators not only responsible for testing and issuing warnings, but also empowered to block the release of a model when it is judged unsafe.

The proposal put forward by Demis Hassabis looks more like an "AI version of FINRA." FINRA is the self-regulatory organization for the U.S. financial industry, funded by the industry while operating under SEC authorization and oversight.

Applied to the AI industry, that would mean frontier models are first handed to an independent body for testing before release. Initially, participation could be voluntary, and once the system matures, it could gradually become a threshold for entering the U.S. market.

Sam Altman's attitude is more pragmatic. If the government cannot build such an institution anytime soon, a few leading labs can start it themselves first.

The problem lies precisely there. Industry standards never only determine what is safe; they also determine who is qualified to stay at the table. The companies that participate first in setting the rules can write their existing safety teams, testing processes, and governance frameworks into "best practices." Later entrants to the market must first bear a set of compliance costs defined by the leading companies.

So safety is certainly real, and so are the barriers to entry.

In January of this year, Semafor reported that Anthropic refused to submit its latest model to a leading British AI safety research institute for testing. This incident exposed a more troublesome problem: everyone agrees that models should be inspected, but when it comes to who exactly does the inspecting and according to whose standards, the labs are not prepared to hand over that decision-making power easily.

Now the three companies are discussing the formation of a new standards body, and what they are fighting over is precisely this power.

The Clock Running in Reverse

When Silicon Valley starts talking about slowing down, Washington and Wall Street both want AI to keep going.

On September 13, Trump publicly responded to this round of AI safety controversy.

He said two things. One was "whoever wins AI, wins." Whoever wins AI wins it all. The U.S. currently leads China, and this advantage must be maintained. The other was that he called the recent warnings about AI risks "very negative forces," arguing that some people are constantly hyping catastrophes that may never happen.

The reasoning Trump gave had little to do with technical safety. His way of looking at AI is still national competition—as long as China is still running forward, the U.S. has no room to voluntarily hit the brakes.

And yet it is now September 2026, less than two months away from the U.S. midterm elections.

In March of this year, an article published in The Guardian argued that AI is becoming an important issue in this midterm election. One of the authors was cryptographer Bruce Schneier, who for the past thirty years has stood almost always at the front line of technology and security controversies.

But Trump does not have a comfortable choice either.

According to a Times report in May of this year, hostility toward AI among MAGA voters is becoming increasingly obvious. They worry about AI taking jobs and driving down wages, and they are also unhappy about data centers consuming local water and electricity. Continuing to bet on AI means Trump will eventually have to face the growing resentment within his own base.

At least for now, he has still chosen the other side. Compared with the employment and resource conflicts that may erupt in the future, the risks of competition with China and a decline in U.S. stocks are closer to the midterm elections.

Wall Street is also facing no small problem.

Over the past few years, the market has dared to give these AI companies astronomical valuations, but what it was actually buying was a continuously rising curve—models keep getting stronger, users and revenue keep growing, so data centers keep being built, GPUs keep being bought, and capital expenditure keeps piling up.

Now, several CEOs standing at the very front of this curve have themselves begun to say that frontier AI may need to slow down.

The market's reaction came faster than the debate. After the heads of companies like Anthropic, OpenAI, and SpaceX publicly called for slowing down AI development, the stock prices of AI-related public companies at one point fell 13% in response.

The slowdown hasn't even truly happened yet, and Wall Street has already begun repricing.

If model capabilities can no longer surge upward at the original pace, then all the numbers that follow will have to be recalculated. How many data centers to build, how many more GPUs still need to be bought, how much capital expenditure cloud providers should spend each year, how much more money AI labs can raise, and what an IPO should be worth.

So what Wall Street truly fears has never just been the release of a few fewer models. What it fears is that the growth rate underpinning the AI bull market of the past few years is itself beginning to become a problem that needs to be regulated.

What is more subtle is that the American public stands even farther away from both Washington and Wall Street.

A poll by the Annenberg Public Policy Center at the University of Pennsylvania shows that Americans are generally pessimistic about the impact of AI, and most want the government to strengthen regulation. Silicon Valley worries about losing control, Washington worries about losing to China, and Wall Street worries about valuations falling.

Amid this turmoil, the two most closely watched AI companies chose two directions.

OpenAI decided to postpone its IPO. Sam Altman has also hinted that if safety issues require more time, the company can push back its listing.

Anthropic, however, is still charging ahead. The company has already selected Nasdaq, with a target valuation as high as $2 trillion, and Nvidia is considering subscribing $10 billion to become a cornerstone investor. NDTV Profit reported that Anthropic will launch its IPO roadshow marketing as early as mid-October.

Dario Amodei is calling on the entire industry to slow down, while his own company is accelerating toward the public market.

But the capital market may not necessarily see this as a contradiction.

If frontier AI is truly as dangerous as Dario says, dangerous enough to require U.S. government intervention, the establishment of common standards among labs, and even several major powers sitting down to coordinate, then the companies able to participate in making these rules will only become more important.

The stricter the regulation, the higher the cost of building frontier models; the more complex the safety testing, the harder it is for latecomers to catch up; and if a global AI governance system truly emerges in the future, Anthropic will most likely still be sitting at the table where the rules are made.

So a "slowdown" doesn't necessarily just mean making a little less money.

It could also mean that this industry has finally become important enough that not just anyone can casually enter. And for the companies already standing at the front, a higher barrier to entry is itself part of the valuation.

The First Domino

Let's rewind and look at how this round of intense discussion around AI Safety was ignited, and how Jacob's post racked up 170 million views.

After leaving the company, he appeared on CNN's Anderson Cooper program and recounted the process of publishing that post himself.

Jacob first wrote a draft, asked friends to help revise it, and discussed with several people how best to phrase it. Before officially posting, he specifically asked friends to help share it. His thinking at the time was:

"Let's try to make this a bit viral."

Try to spread it as widely as possible.

The result far exceeded his expectations. Jacob later admitted he had never seen a post discussing AI safety achieve that kind of reach. He attributed the reaction to a kind of previously unreleased "latent demand."

The media then pressed further on how the post spread. Jacob denied coordinating with any organization to promote it before publishing, but acknowledged that after the post went up, he created a group of about ten people and asked them to help share it, including the founder of the AI safety organization Encode.

Around the time of the post, he also communicated with Daniel Kokotajlo, author of "AI 2027" and a former OpenAI employee.

Those who see Jacob as a whistleblower would say he repeatedly revised the draft and actively sought people to share it simply because he wanted a warning he believed was important enough to be seen by more people. His skeptics seize on the same details, arguing that the sensation itself was carefully engineered.

Both sides can keep arguing, but the more troubling question is: what exactly did Jacob, Dario Amodei, Sam Altman, and those who actually work alongside frontier models see — and why did they all start discussing these things at this particular moment?

You can question how Jacob got that sentence in front of 170 million people, but using communication tactics doesn't mean the statement is false.

A decade ago, Ethereum founder Vitalik Buterin wrote a long essay discussing the possibility that superintelligence could ultimately destroy humanity.

He used an extreme analogy at the time: humanity trying to control superintelligence is roughly equivalent to a person with an IQ of 150 trying to control an agent with an IQ of 6000. If you ask it to cure cancer, it might even conclude that the answer is to first eliminate everyone who could possibly get cancer.

In 2016, these words existed mainly in blogs, forums, and thought experiments. Ten years later, they have begun appearing in resignation letters from frontier lab employees, in public statements by the CEOs of several of the largest AI companies, and in the real agendas of government regulation, capital markets, and international competition.

And what truly makes this difficult is another layer.

On August 2, 1939, physicist Leo Szilard drafted a letter signed by Einstein. Two months later, the letter was delivered to Roosevelt. It warned that nuclear fission could be made into bombs of enormous power, that Nazi Germany was already acting in related fields, and that the United States had to respond as quickly as possible.

Einstein later did not join the Manhattan Project and long regretted his signature that year. But the problem before them at the time was cruel: even if you believe a technology is dangerous, as long as you believe your opponent is building it, stopping may be more dangerous than continuing forward.

More than eighty years later, the same dilemma has reappeared in a different form.

Dario says we must slow down, Trump says China is still catching up; several labs want to jointly establish safety rules, yet none is willing to hand decision-making power entirely to others. Everyone is talking about brakes, but everyone is also watching whether the car next to them has let off the gas.

More critically, we still do not know how far those eyes closest to frontier models have actually seen.

Original link

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia