Original Article Title: My response to AI 2027

Original Article Author: Vitalik Buterin

Original Article Translation: Luffy, Foresight News

In April of this year, Daniel Kokotajlo, Scott Alexander, and others released a report titled "AI 2027," outlining "our best guesses for the impact of superhuman AI over the next 5 years." They predicted that by 2027, superhuman AI would be born, and the future of entire human civilization would depend on the outcome of AI-like development: by 2030, we will either see a utopia (from a U.S. perspective) or face total annihilation (from a global perspective).

In the months following, there have been numerous responses with differing views on the plausibility of this scenario. In critical responses, the focus has mostly been on the issue of the "accelerating timeline": will AI development really continue to speed up as dramatically as Kokotajlo and others have suggested? This debate has been ongoing in the AI field for several years, with many expressing skepticism that superhuman AI will arrive so rapidly. In recent years, the duration for which AI can autonomously perform tasks has roughly doubled every 7 months. If this trend continues, for AI to autonomously perform tasks equivalent to an entire human career, it would have to wait until the mid-2030s. While this progress is still relatively fast, it falls well beyond 2027.

Those who hold a longer timeline view tend to believe that there is a fundamental difference between "interpolation/pattern matching" (the work current large language models do) and "extrapolation/true original thought" (which currently only humans can do). Achieving the latter's automation may require technologies we have yet to master or even conceive. Perhaps we are simply repeating the mistakes of the widespread adoption of calculators: wrongly assuming that since we rapidly achieved automation of a certain type of important cognition, everything else will quickly follow suit.

This article will not directly engage in the timeline debate or delve into the "(highly significant) debate of whether super AI is inherently dangerous." However, it is worth noting that personally, I believe the timeline will be longer than 2027, and the longer the timeline, the more compelling the arguments I put forth in this article become. Overall, this article will present a critique from another perspective:

The "AI 2027" scenario implies an assumption: the capabilities of leading AI ("Agent-5" and subsequent "Consensus-1") will rapidly escalate to possess god-like economic and destructive power, while the capabilities of all others (economic and defensive) will essentially remain stagnant. This is contradictory to the scenario itself stating that "even in a pessimistic world, by 2029, we may be able to cure cancer, slow aging, and even achieve mind uploading."

In this article, I will describe some strategies that readers may find technically feasible but impractical to deploy in the real world in a short amount of time. In most cases, I agree with this point. However, the scenario in "AI 2027" is not based on the current real-world but assumes that in 4 years (or any timeline that could lead to destruction), technology will have developed to give humans far greater capabilities. Therefore, let's explore: What would happen if not just one side had AI superpowers, but both sides did?

The Bioweapon Apocalypse Is Far More Complex Than Described in the Scenario

Let's zoom into the "species" scenario (i.e., everyone dies because the U.S. was overly fixated on defeating China and neglecting human safety). Here is the plot where everyone dies:

"In about three months, Consensus-1 expanded around humanity, transforming prairies and tundra into factories and solar panels. Eventually, it deemed the remaining humans too obstructive: by mid-2030, AI released over a dozen subtly spreading bioweapons in major cities, silently infecting almost everyone, then triggered lethal effects through chemical sprays. Most people died within hours; a few survivors (like doomsday preppers in bunkers, sailors on submarines) were eliminated by drones. Robots scanned victims' brains, storing copies in memory for future study or revival."

Let's analyze this scenario. Even at present, there are some technologies in development that would make this kind of "clean victory" for AI less realistic:

· Air filters, ventilation systems, and UV lights that can significantly reduce the airborne transmission of diseases;

· Two real-time passive detection technologies: passive detection of human infection within hours and alerting, rapid detection of unknown new virus sequences in the environment;

· Various methods to enhance and activate the immune system that are more effective, safe, universal, and locally producible than the COVID-19 vaccine, enabling the human body to resist both naturally occurring and artificially designed pandemics. Human evolution occurred in an environment with a global population of only 8 million, spending most of its time outdoors, so intuitively, we should be able to easily adapt to today's more threatening world.

When combined, these methods may reduce the basic reproduction number (R0) of airborne diseases by 10-20 times (e.g., better air filtration reduces transmission by 4 times, immediate isolation of infected individuals reduces by 3 times, simple enhancement of respiratory immunity reduces by 1.5 times), or even more. This is enough to prevent the spread of all existing airborne diseases (including measles), and this number is far from the theoretically optimal.

If real-time virus sequencing were widely used for early detection, the idea that "a silently spreading bioweapon could infect the global population without triggering an alert" would be highly suspicious. It is worth noting that even advanced tactics such as "releasing multiple epidemics and chemical substances that are only hazardous when combined" could also be detected.

Let's not forget that we are discussing the assumption in "AI 2027": by 2030, nanobots and Dyson spheres are classified as "emerging technologies". This means that efficiency will be significantly improved, making the widespread deployment of the aforementioned response measures more promising. Although as of today in 2025, human action is slow, and inertia is abundant, with many government services still relying on paper-based offices. If the world's most powerful AI can transform forests and fields into factories and solar farms by 2030, then the world's second most powerful AI can also install a large number of sensors, lights, and filters in our buildings by 2030.

But let's further explore the assumption of "AI 2027" and enter a purely science fiction scenario:

· Micro air filtration in the body (nose, mouth, lungs);

· An automated process from discovering new pathogens to fine-tuning the immune system to defend against them, ready for immediate application;

· If "consciousness uploading" is viable, simply replace the entire body with a Tesla Optimus or Unitree robot;

· Various new manufacturing technologies (likely to be highly optimized in a robot economy) will be able to locally produce far more protective equipment than currently available, without relying on the global supply chain.

In a world where the cancer and aging problem will be cured in January 2029, and technological advancements continue to accelerate, by the mid-2030s, if we do not have wearable devices that can real-time bioprint and inject substances to protect the human body from any infection (and toxin), it is truly unbelievable.

The above bio-defense argument does not cover "mirror life" and "mosquito-sized killer drones" (predicted to appear in the "AI 2027" scenario in 2029). However, these means cannot achieve the sudden "clean victory" described in "AI 2027", and intuitively, symmetric defense against them should be much easier.

Therefore, a bioweapon is actually unlikely to completely destroy humanity in the manner described in the "AI 2027" scenario. Of course, all the outcomes I have described are far from a human "clean victory". No matter what we do (perhaps except for "uploading consciousness to robots"), a full-scale AI biological war would still be extremely dangerous. However, achieving the standard of a "human clean victory" is not necessary: as long as there is a high probability of a partial failure in an attack, it is enough to provide a strong deterrent to the AI that has already dominated the world, preventing it from attempting any attack. Of course, the longer the timeline of AI development, the more likely such defensive measures are to be effective.

How to Combine Biological Weapons with Other Means of Attack?

For the above response measures to be successful, three prerequisites must be met:

· The world's physical security (including biological and counter-drone security) is managed by local authorities (human or AI) and not all puppets of Consensus-1 (the AI name in the 'AI 2027' scenario that eventually controls the world and destroys humanity);

· Consensus-1 cannot invade the defense systems of other countries (or cities, other secure areas) and immediately render them ineffective;

· Consensus-1 has not controlled the global information space to the extent that no one is willing to attempt self-defense.

Intuitively, the outcome of prerequisite (1) may lead to two extremes. Today, some police forces are highly centralized with a strong national command system, while others are localized. If physical security must rapidly transform to meet the demands of the AI era, the landscape will be completely reset, and the new outcome will depend on the choices made in the coming years. National governments may choose to take shortcuts and rely on Palantir, or they may proactively choose solutions that combine local development with open-source technology. In this matter, I believe we need to make the right choice.

Many pessimistic narratives about these topics assume that points (2) and (3) are beyond redemption. Therefore, let's delve into these two points in detail.

The Doomsday of Cybersecurity Is Far from Over

Both the public and professionals generally believe that true cybersecurity is unattainable; at best, we can swiftly patch vulnerabilities once discovered and deter cyber attackers by stockpiling known vulnerabilities. Perhaps the best scenario we can hope for is a situation akin to "Battlestar Galactica": nearly all human ships are simultaneously paralyzed by Cylon cyber attacks, while the remaining ship survives unscathed due to not utilizing any networking technology. I disagree with this viewpoint. On the contrary, I believe the "endgame" of cybersecurity is advantageous to the defending side, and with the rapid technological advancements assumed in 'AI 2027', we can achieve this endgame.

One way to understand this is to adopt a favorite technique of AI researchers: trend extrapolation. Here is a trend line based on in-depth research using GPT, assuming top-of-the-line security technology, showing how the vulnerability rate per thousand lines of code changes over time.

Furthermore, we have seen significant progress in the development and consumer adoption of sandboxing technologies and other isolation and minimally trusted codebase technologies. In the short term, attackers with exclusive super-intelligent vulnerability discovery tools can find a large number of vulnerabilities. However, if highly intelligent agents used for vulnerability discovery or formal code verification are publicly available, the natural eventual balance will be: software developers discovering all vulnerabilities through a continuous integration process before releasing the code.

I can see two compelling reasons why even in this world, vulnerabilities cannot be completely eliminated:

· Defects stem from the complexity of human intent itself, so the main challenge lies in building a sufficiently accurate intent model, rather than the code itself;

· For non-security-critical components, we may continue the existing trend in consumer technology: by writing more code to handle more tasks (or reducing development budgets) rather than achieving the same number of tasks with ever-increasing security standards.

However, these categories do not apply to scenarios such as "can an attacker obtain root access to systems that sustain our lives," which is precisely what we are discussing here.

I admit that my view is more optimistic than the mainstream view held by smart people in the current cybersecurity field. But even if you disagree with my view against the background of today's world, it is worth remembering: the scenario in "AI 2027" assumes the presence of superintelligence. At the very least, if "1 billion superintelligence copies think at 2400 times human speed" cannot give us code without such flaws, then we absolutely should reassess whether superintelligence is as powerful as the author imagines.

To some extent, we not only need to significantly raise software security standards but also need to raise hardware security standards. IRIS is a current effort to improve hardware verifiability. We can start with IRIS or create better technologies. In fact, this may involve an approach of "constructing correctly": the hardware manufacturing process of critical components deliberately designed specific verification steps. These are tasks that AI automation will greatly simplify.

The Super Persuasion Doomsday is Far From Over

As mentioned earlier, another scenario where a significant increase in defense capability may still be futile is: AI persuades enough people to believe that there is no need to defend against the threat of superintelligent AI and that anyone attempting to find defenses for themselves or the community is a criminal.

I have always believed that there are two things that can enhance our resistance to super persuasion:

· A less monolithic information ecosystem. It can be said that we are gradually entering a post-Twitter era, and the Internet is becoming more fragmented. This is a good thing (even though the fragmentation process is chaotic), as we overall need more information pluralism.

· Defensive AI. Individuals need to be equipped with locally running, explicitly loyal AI to balance the dark patterns and threats they see on the Internet. Such ideas already have sporadic pilot projects (such as Taiwan's "Message Checker" app, which performs local scans on phones) and there is a natural market to further test these ideas (such as protecting people from scams), but more effort is needed in this area.

From top to bottom: URL check, cryptocurrency address check, rumor check. Such applications can become more personalized, user-centric, and powerful in functionality.

This contest should not be a super-intelligent super-persuader against you, but rather a super-intelligent super-persuader against you plus a slightly weaker but still super-intelligent analyzer working for you.

This is what should happen. But will it really happen? Over the short time frame assumed in the scenario of "AI 2027," achieving widespread adoption of information defense technology is a very challenging goal. However, it may be argued that a more modest milestone would be sufficient. If collective decision-making is crucial, and if, as in the scenario of "AI 2027," all major events occur within a single election cycle, then strictly speaking, it is important to enable direct decision-makers (politicians, civil servants, some corporate programmers, and other participants) to use good information defense technology. This is relatively easier to achieve in the short term, and in my experience, many such individuals are already accustomed to interacting with multiple AIs to aid decision-making.

Insights

In the world of "AI 2027," people take it for granted that superhuman artificial intelligence will effortlessly and rapidly eliminate the remaining humans, making it imperative for us to ensure that the leading AI is benevolent. However, I believe the reality is much more complex: whether the leading AI is powerful enough to easily eliminate the remaining humans (and other AIs) remains a highly debated issue, and we can take action to influence this outcome.

If these arguments are correct, their insights into today's policies sometimes resemble the "mainstream AI safety principles" and sometimes differ:

Slowing down the development of superintelligent AI is still a good thing. Having superintelligent AI emerge in 10 years is safer than in 3 years, and emerging in 30 years is even safer. Giving human civilization more time for preparation is beneficial.

How to achieve this is a puzzle. I believe that the United States' proposed "10-year ban on state-level AI regulation" being rejected is generally a good thing, but especially after early proposals like SB-1047 failed, the next steps became less clear. I think the least intrusive and most robust way to slow down the development of high-risk AI might involve some form of treaty on regulating cutting-edge hardware. Many hardware cybersecurity technologies needed for effective defense also contribute to the verification of international hardware treaties, creating a potential synergy here.

Nevertheless, it is worth noting that I believe the primary source of risk comes from military-related actors who would vigorously seek exemptions from such treaties; this cannot be allowed, and if they ultimately are granted exemptions, AI development solely driven by the military could increase risks.

Facilitating AI to do more good and less harm is still beneficial. The key exception (and it has always been) is: facilitation ultimately evolves into enhancement of capability.

Regulating to increase transparency in AI labs is still beneficial. Incentivizing normative behavior in AI labs can reduce risks, and transparency is a good way to achieve that goal.

The "Open to Harm" mindset is becoming riskier. Many people are against openly empowering AI, arguing that defense is not practical, and the only bright side is for the good actors to achieve superintelligence with good AI capabilities before anyone with less noble intentions, gaining any highly dangerous ability. However, the argument in this article paints a different picture: defense is not practical precisely because one actor is far ahead, and the others have not caught up. Technology diffusion to maintain a balance of power becomes crucial. But at the same time, I would never argue that accelerating cutting-edge AI capability growth through open source is a good thing simply because it is done openly.

The U.S. lab's "We must beat China" mindset is becoming riskier for similar reasons. If hegemony is not a safety buffer but a source of risk, this further refutes the (unfortunately all too common) view that "good actors should join leading AI labs to help them win faster."

Initiatives such as "Public AI" should be further supported, both to ensure broad distribution of AI capabilities and to ensure that the infrastructure actors actually have tools to quickly apply new AI capabilities in some of the ways described in this article.

Defense technology should embody more of the concept of "Armed Sheep" rather than "Hunt Down All Wolves." Discussions about the fragile world hypothesis often assume that the only solution is for a hegemonic state to maintain global surveillance to prevent any potential threats. However, in a non-hegemonic world, this is not a feasible approach, and top-down defense mechanisms are easily subverted by powerful AI, turning into weapons. Therefore, a greater defense burden needs to be achieved through strenuous efforts to reduce the world's vulnerability.

The above arguments are speculative and should not be acted upon based on the nearly certain assumptions made in these arguments. However, the story in "AI 2027" is also speculative, and we should avoid taking action based on the assumption that its specific details are almost certain.

I am particularly concerned about a common assumption: establishing AI hegemony, ensuring its "alliance," and "winning the race" is the only way forward. In my view, this strategy is likely to decrease our security—especially when hegemony is deeply linked to military applications, as it greatly undermines the effectiveness of many alliance strategies. Once hegemonic AI deviates, humanity will lose all means of balance.

In the "AI 2027" scenario, human success hinges on the United States choosing the path of safety over destruction at a critical moment—voluntarily slowing down AI progress to ensure that Agent-5's internal thought processes are interpretable by humans. Even so, success is not guaranteed, and how humans can avoid the ongoing existential cliff of relying on a single superintelligent mindset remains unclear. Regardless of how AI develops in the next 5-10 years, acknowledging that "reducing world fragility is feasible" and investing more effort in achieving this goal with the latest human technologies is a path worth exploring.

Special thanks to the Balvi volunteers for their feedback and review.

Original Article Link

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia