Editor's note: Anthropic went public with a valuation of $1.5 trillion and at one point reached a market cap of nearly $2 trillion. Everyone is recalling how SBF invested in Anthropic at a tiny valuation back then—if held until now, it would be worth close to $40 billion, but unfortunately it was already sold.
This 14-year-old kid Eli Goldfine contacted SBF's mother and had a phone call with SBF in prison. This is the best interview I've seen so far after the FTX collapse and SBF's imprisonment—it covers not only SBF's retrospective on the Anthropic investment and his views on AI, but also many tracks that FTX was already working on in 2020 that are now hyped as hot topics, as well as his current views on these hot tracks.
· The probability of AI causing human extinction or civilizational destruction—he puts it at 25%: originally said "below 50%, above 10%," ultimately landing around 25%; if split by criteria like "whether mind uploading counts as a catastrophe," the range expands to 20%–60%.
· In the next 5–10 years, AI will reach or surpass humans on the vast majority of important tasks, just a small step away from "AI running most of the world," but the process will be more like a fuzzy gradual progression—there won't be a single year where it "snaps" across.
· The impact lands industry by industry, crossing the line one sector at a time: software development and mathematics are already happening, and it will spread to more fields next; the world as a whole isn't overturned, but局部 will change dramatically.
· The real question isn't "whether to slow down" but "whether everyone can slow down together": as long as there's one player who completely doesn't care about safety, unilateral slowing down just turns the race into a race to the bottom.
· There's no single high-level answer to alignment—the devil is all in the details: massive testing, massive debugging, massive caution, and assuming the model will eventually be used as an agent.
· Anthropic is a trillion-dollar-level, top-two AI company: in the next decade AI will become a core industry, with cumulative market cap reaching tens of trillions or even over a hundred trillion dollars—companies that can hold this scale can be counted on one hand.
· Only investing $500 million back then was because that's all it needed: this was the result of the company not wanting to sell more equity and raising on demand, not a misjudgment on his part.
· The alpha now isn't in "who has an AI story" but in distinguishing who is a real AI company; and brand defends moats better than technology (Coinbase and Robinhood are worth tens of billions because of their names).
· That "51% double, 49% ruin" coin-flip bet — on an abstract level he would take it, but in the real world he absolutely would not: because ruin is permanent, while growth is something others can eventually achieve too.
· When asked "given all the good things Future Fund did, was the whole FTX rise-and-collapse arc worth it," his answer was no: he would very much like to believe it was worth it, but he cannot honestly say it was; if it hadn't ultimately blown up, it would have been worth it — and he doesn't think people have learned the lesson, because in a crisis people easily lose their judgment about what truly matters.
I think they're worth a lot of money.
Look at the biggest industries, the ones that truly drive certain parts of our economy — their market caps are in the trillions to tens of trillions of dollars. There's no reason to think AI won't become one of those core industries over the next decade, or even the most important one among them.
So if you expect the total market cap of AI companies to eventually reach tens of trillions, or even over $100 trillion, how does that money get divided? Maybe you leave two-thirds for companies that don't exist yet. Even so, that still leaves at least $10 trillion for existing companies. And there simply aren't that many companies that can support that scale. You have OpenAI, you have Anthropic, you have Google and its several labs, probably DeepMind, xAI. The companies that truly count as core players can be counted on one hand. That basically means the top two or three each are worth at least a few trillion dollars.
And I think Anthropic, even if not number one, is clearly in the top tier.
One thing is looking at the capabilities of current models and how good the current products are — and they and OpenAI have clearly been going back and forth on that. Anthropic's consumer-facing product has received a lot of positive feedback, and that may not even be their most distinctive product. Their AI software development tool is the best in the world, and that's another huge area. So at least as of today, you have to count it among the top two.
Good question. Let's start with the next few years — absurd things could pop up at any moment. This is just a median-case projection on my part, and it could deviate at any time.
But what we've seen so far is this: AI had little impact on the world for a long time, even as its capabilities doubled again and again, because it was still in the "less than human" stage—and less than human to such a degree that using humans to do things was still more cost-effective than using AI. Then, just in recent years, it began crossing that line in more and more places and started to have an impact on the world.
It hasn't completely transformed the world yet. But as key industries successively fall below that critical threshold—where AI suddenly becomes far more productive than humans in a certain field—you'll see fairly rapid explosions in those areas. So in the next year or two, I think a few more fields, possibly quite important ones, will follow the path of math and software development: the most cutting-edge work will actually be done mostly by AI models. The whole world won't be overturned, but you'll start to see significant changes in certain places.
There are many problems. Let me step back. One is: could something spiral into chaos or go catastrophically wrong? The most obvious example is an out-of-control AI getting access to weapons—say, an AI system used to detect threats to a nuclear power, it goes rogue, and launches a thousand ballistic missiles in one go. That's one scenario. I think the probability of it happening in the next decade is low, but not zero—it's possible.
Setting doomsday scenarios aside for now, I think in the next decade we may find ourselves in a world where, suddenly, humans are no longer needed in most professions. Even if the world doesn't change that much in the short term, this would fundamentally distort society's culture. The impact would be enormous if we no longer define ourselves by work. Also, what happens when the output of certain industries becomes increasingly difficult for humans to understand? Does that count as AI quietly completing a takeover? And how would those AIs view humans?
I don't know whether these questions will have answers within the next decade. But I think it's very likely they'll all burst forth at once, or at least come one after another, and in many respects, humanity's position as the central decider of its own future will by then already be written on the wall.
A lot of the credit here can't go to me. Many important things were written by others; to a large extent I was just emphasizing or restating what they had said.
But there are a few threads of thinking. One is: look at what AI had achieved by around 2020. The answer is, in most fields, nothing. In most fields, you could try to use AI, and it would be a bit dumb—impressively better than ten years ago, but still completely useless. But in a few things, the things it was specifically trained for, it had already surpassed humans—most board games, like chess or Go; more complex domains like video games were starting to emerge too.
If AI only ever got good at chess, you wouldn't make much money, wouldn't upend the social order, wouldn't change the future much. But there's no reason to think chess is all AI can do. It keeps growing exponentially from a tiny starting point, and it doesn't need many more doublings before it starts surpassing humans in key areas of society.
Good question. The answer is basically: when you're founding a company and raising money, you usually don't want to raise an infinite amount. You have financial projections, you roughly know how much money you need to get the business going, and you don't want to sell more equity than that percentage.
Anthropic was in that position at the time. They calculated they needed roughly $500 million in compute spending to build out the model, so that's what they raised. We basically invested all the money they were raising in that round.
Maybe they should have. I guess part of the reason is they didn't want to sell more equity than necessary. But another part is that many companies are actually dragged down by their own success, by their own cash flow and cash on hand. Historically, many companies raised a big chunk of money and then blew it all on advertising because they felt they had to do something with it, and in the end got nothing in return, just stagnated.
So I guess they were also thinking at the time: how much money do they actually need to spend over the next few years, trying not to put the cart before the horse.
The people involved rotated in batches, and later we hired people directly to professionalize it. At first it was just me or others at the company investing deal by deal as opportunities came up—and honestly, Anthropic was one of those.
But then we started doing very well. A lot of the early investments were in crypto assets, and honestly they basically doubled every year, even against the backdrop of the entire crypto market, so we decided to try to scale it up. We brought in Ramnik, and Amy, and others to focus specifically on this.
Overall I was involved at a relatively high level across the whole portfolio. And in certain specific deals—Anthropic was obviously one of them, Solana was another—I'd get more deeply involved, at least following on a weekly basis, especially when those investments interacted with Alameda and FTX, or when there were ways to help them.
I wish I had a better answer. Four years ago, I did have pretty good ideas about what to invest in. But the real answer to what to invest in today is: I shouldn't listen to my own advice. I need to do due diligence on the outside myself before I can give a decent answer.
Broadly speaking, I suspect there will be many companies built on top of AI — small two-person teams that become $100 million or even $1 billion companies within a few years, and you can invest at much lower valuations. But the problem is that at the same time, a ton of garbage will also emerge. Right now every company stuffs an AI line into its BP, even when it makes no sense — like a T-shirt company saying "we're going to use AI to design T-shirts." Fine, maybe it helps a bit, but they're repackaging themselves as essentially an AI company, when essentially they're not.
So a big part of the alpha right now lies in distinguishing between these two: which companies are truly AI companies, which can truly achieve explosive growth through AI, and which are not — at least not yet. You can see this in large-cap stocks too. Many early market participants actually didn't know how to distinguish Amazon, Google, Apple, and Netflix — they're all tech companies, so does that mean AI will affect them all the same way? The answer, of course, is no. A few of them are deeply tied to AI growth — Nvidia is the most obvious example; while others, like Apple, are indeed tech companies, but what defined Apple's success over the past thirty years is essentially outstanding consumer product design: beautiful hardware, easy-to-use software, plus a powerful brand. At least as far as tech companies go, that's not AI-centric.
I don't think only AI companies can succeed. First, in some industries AI won't have a major impact for a long time. Take steel mining — the world still needs steel, and now AI has begun to enter physical mining, so it has suddenly become relevant overnight.
But another point I think is crucial is brand. You might have a software company with products and a sizable market cap, and then a competitor says: we can make almost exactly the same thing very cheaply. True, but that doesn't mean people can buy it. Having a brand standing there is extremely, extremely valuable.
Look at companies like Coinbase or Robinhood. What is their real differentiation? Why are they each worth tens of billions of dollars? It's not because of technology, nor because they have something unique in this space. Maybe regulation, but that's not entirely it for them either. It's brand — a name that American retail investors recognize and will use. That can be worth a lot of money, and for many companies, even if the core product becomes very easy to replicate, this kind of thing will persist.
A big part of it is: if you're going to hire someone to do something, the premise is that you intend to trust them to do it well. If someone on the team has already spent 15 hours doing research and due diligence, that's fine—I'll look it over, add what I can, and give them the resources I have that they don't.
But beyond that, if I don't intend to trust that they've made the right judgment and seen it correctly, then I'd have to spend those 15 hours myself redoing the due diligence. At that point, hiring them in the first place would have been meaningless.
Part of the answer will have to wait until that day, because there's a lot of information I don't have right now. The world is changing, and some information is much easier to get on the outside than on the inside.
But I will very seriously consider returning to "earn to give," supporting the movement, and supporting projects that I believe are doing good. That will probably be a default path.
Beyond that, other paths will depend on what role I can still play in society. In the past, I was in a position to have influence in many areas, but those will now be much harder because my reputation is extremely negative. So in terms of public discussion, politics, media, and so on, I'm not sure what role I can play in the future—possibly none at all. But "earn to give" is a default path.
I will very seriously consider it. It depends on when and how I get out. But I will definitely consider crypto, and I will also consider AI, as well as finance more broadly.
I think it indeed was more effective.
A lot of what we did was "regranting": giving a sum of money to a group of people and letting them decide for themselves which small projects to fund, thereby crowdsourcing opportunities that would otherwise have fallen outside our radar. There are many people with enormous potential in areas like AI safety, biosecurity, and politics, but they need funding—possibly not much, perhaps just enough to get them through college or enough to start a company. We were very happy to provide that kind of money.
Many of these projects look small individually, but carry high leverage; taken together, the impact is significant. And we're happy to fund people who are just getting started: they have a charitable idea, or some other tool that's good for the world, it may or may not work out, and they need money to give it a try. Our situation is that even if some of them ultimately go nowhere, we don't mind—we don't depend on any single project for a sense of fulfillment, or to prove we did good.
Ultimately I think this is a fairly effective approach, partly because at the time it was hard to find mature projects that could absorb tens of millions to hundreds of millions of dollars and still spend it efficiently. An organization with an annual budget of $8 million—you suddenly give it $30 million more, and it may simply not know what to do with the money.
Investing in Anthropic was strictly speaking before the fund was established, and part of it was out of AI safety considerations. Setting that aside, we've funded a batch of biosecurity projects, some of which later developed new treatments for infectious diseases—the kind that kill huge numbers of people globally—and some organizations are clearing the path to get such treatments to market faster.
I think there are many things that would have had enormous potential impact but were halted when a crisis hit, and I'd count quite a few political projects among them.
Of course no one can be certain. But my guess is: as long as there's one company that particularly cares about safety, the way it develops things will be more cautious than it would be without that company. That's not all you can hope for, because as long as there's still a participant that completely doesn't care about world safety, there will be a race, and possibly a race to the bottom, which would be very bad.
But having one more safety-conscious participant helps.
They can answer that better than I can. But broadly, part of it is slowing down the most frontier development and releases when models start having the potential to go out of control.
You can look at the Hugging Face hack—the ultimate damage it caused wasn't large, but with a different target it could be extremely destructive, and it wasn't far from a global catastrophe, from the perspective of "what an AI agent actually needs to do." So you need to be very clear that models must not go out of control like that, set as many checkpoints as possible, take the model's ultimate goals, guiding principles, and alignment as seriously as possible, while maintaining real-time responsiveness. None of this is perfect, but it helps.
This is a real question. Perhaps the best outcome for the world is for everyone to slow down, but if others don't slow down, that doesn't necessarily mean slowing down yourself is right or that you're rushing into a race to the bottom. Perhaps having one more safety-conscious competitor in this race is better, but this is obviously not the ideal situation—it's just that we may happen to be living in this situation.
The truly best approach is to influence other participants, to get all major players to reach a cross-camp consensus as much as possible, all placing more emphasis on safety.
I think they are responding to a specific event, and I don't know the details, so I can't speak as precisely as I would like. Beyond that, I think the Biden administration at the time was very...
That would be negative.
I'm not sure it will be a black-and-white, discrete event. It could be, but it's equally possible that over the next five years we'll watch parts of society be swallowed by AI piece by piece. Software development is already happening, mathematical research as well, and chess and Go happened even earlier. This trend will continue until more and more parts of society are run by AI, with the level of human oversight continuously declining over time.
I think it's already quite general. You can ask the top models almost any question and get an answer at least comparable to a smart person. What we see less of is AI breaking through to levels that humans fundamentally cannot break through.
As I understand it, the current frontier level is roughly equivalent to what a very smart, very rational person could ultimately achieve given plenty of time—except it's instantaneous, right at your fingertips. Whether that counts as superhuman, I don't know. I think it's a bit better than humans, but not much better, except that it can run many instances simultaneously and think much faster than humans.
So it may not necessarily be a clear line from human to superhuman. At least so far, what we've seen is a blurry line: it slowly, and by slowly I mean just months or years, goes from 'less than human' to an IQ of 100. If this trend continues, there might not be a moment where it clearly 'snaps' past. But there could also be a year when it clearly leaps from human level to clearly above human.
I'd say below 50%, above 10%, around 25%.
I should clarify that the model in my head, I wouldn't say it's the best model, and a large part of that probability comes from a complex scenario where people would disagree a lot: if in the future humans start uploading their brains, and those who don't upload get left behind, does that count as a catastrophic event? Different people have different answers. Depending on the answer, I could give any number between 20% and 60%.
Let's start with the baseline: in the next 5 to 10 years, or even shorter, AI is likely to be as good as, or better than, humans at most tasks—and it already is in many areas. Then it will get better, maybe much better, maybe only a little better, at not all but the vast majority of tasks that matter to the world.
Once it reaches that point, it's only a small step away from AI running most of the world and holding most of the power. The probability that a more powerful species would never use that power over a hundred years seems quite low to me. So I lean toward thinking that in the next decade we'll see AI at least 'softly take over' a large part of the world's operations and infrastructure.
At that point, one possibility is peaceful coexistence, with no massive changes for humanity, but that doesn't look like the most likely scenario. I think it's possible that humanity, with AI's help, grows into something beyond what we are now. It's also possible that AI just gets bored with humans and wipes us out. Another possibility is AI-led but human-goal-oriented collaboration bringing massive technological expansion. My 20% to 60% depends on which of these scenarios you count as 'doom.'
If I were 100% certain it was doing exactly that—not just certain that was its goal, but certain it wasn't stuck in some destructive failure mode, without a bunch of messy fuzzy approximations—then, under all those assumptions, the answer is yes.
Possibly. Again, it depends on the details, but I personally have no objection to it.
Good question. I think the concept of "personhood" needs to be expanded. It's already a fact that no one is exactly the same person as they were yesterday. Parts of us die, parts grow; memories change, chemistry changes, bodies change, neurochemistry changes. But usually the differences between one day and the next are small enough that "same person" is a good approximation.
Current AI is obviously not like that. I think that's the right model to reference: is a Claude Sonnet and a Claude Opus the same person? Neither "yes" nor "no" is a good answer. I think if there were an "uploaded person," the situation would be the same.
I can have people copy-paste model outputs in here. My impression is that they're basically a meticulous, diligent research assistant, except you can think as fast as you want—a day's work can be done in a minute, which is remarkable and very valuable.
But my current impression is that most of what they do, humans can also do, it just takes humans time. And you need reliable people guiding them, access to information, and the whole set of things that current models are already pre-connected to.
FTX started as a crypto exchange, doing Bitcoin, Ethereum, and their futures. But fundamentally, prediction markets are futures contracts. From a financial perspective, both are just markets that settle to some value.
So from the beginning we wanted to launch prediction markets, and we tested the waters on FTX International. We had a fairly active market for the 2020 presidential election—I remember the Trump vs. Biden market had volume in the hundreds of millions of dollars. This was something we wanted to grow long-term, but I couldn't grow it until we got regulatory clarity from the CFTC.
I think a large part of it was regulatory. When we did it, we didn't have the CFTC license needed to launch it in the US. So our market was only open to the international site, and Americans weren't allowed to participate. Later, Kalshi and Polymarket got a more formal US regulatory foothold, and that was exactly when they took off.
Polymarket wasn't doing KYC back then, right? I don't think so. So it's very likely that a large number of Americans came in via VPN, as you said. And as a centralized exchange, we had to do KYC for all users — beyond IP checks, we also required ID cards, passports, and proof of address from regions outside the US.
I think tokenized stocks still have a lot of room. They've certainly grown since 2021 and 2022, but fundamentally, their infrastructure is much more usable than the real stock market, so there's no reason they couldn't become the primary venue for securities trading.
Beyond that, prediction markets can cover a very wide range. I think political prediction markets haven't grown as much as I expected, and part of that is still a regulatory issue. Taking a step back, margin trading, futures, and derivatives for US customers — that's another thing that basically doesn't exist, and that's exactly the form most international volume takes. That's also a regulatory barrier.
So I think there's potential there. Of course, there will definitely be many things I didn't anticipate that will emerge and surprise most people.
Well, customers like perpetual contracts. One thing is clear: regardless of whether it makes sense as an underlying financial product, not having to deal with expiry is convenient in itself. The actual mechanism is quite complex from a funding rate perspective, but I think the reason is basically this — people don't like worrying about expiry.
Actually, not that much. A lot of people just ignore it. Some people specifically take the favorable side to collect funding rates. I originally thought that would be a big financial disincentive for people, but it wasn't.
Again, convenience is a huge factor. And I think a lot of people honestly just don't think about it and don't pay much attention to how much interest they're paying.
Then beyond that, liquidity attracts liquidity. Once those markets become consensus, clients are there, market makers are there, liquidity is there, and big clients will go there. You get a virtuous cycle, a feedback loop, and everyone converges on the market that first gained momentum.
My view hasn't changed much. I'm reading a lot of fiction, a lot of science fiction. It's okay, but honestly, I'd rather watch Netflix.
Honestly, no. There are things I like, but among the best science fiction I've read, maybe I'd trade it for a dispensable Netflix show—but only maybe, only barely. Maybe I'll end up liking reading a bit more science fiction than I expected, but not by much.
It takes time. But part of it is an information throughput issue. Books only have visuals, and only the visuals of text; shows have sound and are also visual, and visuals can contain more than just text—you can display arbitrary pixels. So from an information-theoretic perspective, you can do far more, and I think that helps a lot.
The other part is that books have something shows usually don't: narration of characters' thoughts. I think that can be done away with; it's often redundant, the author's perspective indulging itself. And right now money and talent are flowing to shows, so naturally that's where the most development will happen.
For that scenario to hold, many preconditions are needed. You have to assume the universes are independent, as you said; assume this is a one-time opportunity that cannot be replicated at a lower cost in the future; and many other unspoken assumptions.
But one way to simplify it is this: two worlds, roughly equal in population, with comparable expectations for the future, and completely independent of each other—they will never interact, each outside the other's light cone, whatever you want to call it. Then somehow, some strange alien god descends and says: each of these two worlds has a 50% chance of being destroyed; and for some reason, you believe this alien god is telling the truth. It says you can choose to make them correlated or anti-correlated. Option one: either both survive or both are destroyed, 50% each. Option two: one survives and one is destroyed, randomly, and you don't know which is which.
In that scenario, I think it's clear these two choices are equivalent. And if one side offers you 51-to-49 odds, you should choose that over "live or die together," because each world doesn't care what happens to the other—they will never interact anyway. And this is almost the original question: you have to choose between "exactly one world survives" and "a 50% chance both worlds survive."
This is an interesting question, and it depends on what these "doublings" are actually doubling. You talk about increasing marginal utility and diminishing marginal value—utility of what?
If this refers to the long-term accumulated utility of the entire universe, then no—utility is linear in itself. But if it's actually some approximate metric, such as GDP—if you're asking "can GDP double"—then there is very likely deep diminishing marginal returns.
First, doubling GDP does not mean everyone's happiness doubles, and even less does it mean the world's long-term expected value doubles. That GDP doubling might have happened anyway thirty years later; or it might cause severe environmental damage and eventually collapse back. So in the real world, the metrics you use to measure things may not be what you actually care about.
There is one, but I have to add a caveat: I'm not sure which of these can really be achieved in the near term—these all require technological progress. Colonizing other planets, colonizing the solar system, the galaxy—eventually you would truly have mutually independent worlds, and you would simply be enabling more human flourishing. That is one example.
What if we think differently: the population remains constant, forever maintained at one billion people, can their happiness continuously double? I think the answer is, we don't really know. This is almost a biological question, rather than anything else. Making people satisfied and moderately happy is one thing; making people ecstatic is difficult, and imagining what could make people twice or ten times as ecstatic is even harder—it's simply outside the range of emotions humans typically experience. Theoretically it might be possible, but we have no good template, nor the technology or biology to achieve it.
I think a clearer analogy is choosing between "continuing to sprint" and "permanently cutting off all AI or AI-like development." Because then you're basically choosing between "the world facing severe risks" and "limiting the world's capacity for prosperity," and the latter would likely ultimately require some form of automation or digital intelligence. So in that case, permanently sealing off this path would come at a tremendous cost to the future.
But if it's just "faster or slower," it's less clear, because there are two models you can use. One is the "perpetual exponential growth" model: once acceleration begins, utility doubles every year, endlessly. Under this model, starting one year earlier means permanently doubling the universe, but this is a bit illusory—it can't go on forever.
The other is to view it as a singularity-type event, after which some new plateau is reached, highly unknown. After the event happens, the universe may still have billions of years, so delaying it by one year, ten years, or even a hundred years, under this model, wouldn't reduce future utility by even 1% by much. Under this model, you'd be perfectly willing to delay development by a hundred years to reduce p(doom) by 1%.
Of course, the real answer is probably somewhere in between. And I think "what the utility-over-time curve looks like in an AI explosion scenario" is a very interesting question.
That's part of it. But the first part of my answer is: there are many preconditions. You have to assume a lot about the situation to arrive at that answer, and those assumptions often don't hold in real life. In reality, all options are much messier than they're presented. And in the real world, accidentally destroying something is much easier than building something.
Absolutely.
Because my sentence is life without parole. Realistically, my goal isn't to guarantee I get out at 60 and find a cabin to spend a few years in—that's not what I expect from life. I want to have a positive impact on the world, and that requires me to be out there. There's a high premium on that. If the cost is losing the final years of my life when I'm no longer productive, I'm completely willing to pay that price.
Sure. There's a real-world scenario that looks like a coin flip: heads, your model shows it could double the world's growth; tails, it could destroy the world. You absolutely must not bet on that coin.
Because in the real world, first, the "double the world's growth" side of that 50%—you're probably building something others would have built eventually anyway; it's probably not a true, permanent doubling of world value. And if the world gets destroyed, that's permanent—gone forever.
So I think in real-world scenarios, once you add uncertainty and noise into every link of the chain, the weight of catastrophic destruction becomes much heavier, and moderate growth is still good but nowhere near enough to offset it.
Same logic: if this reasoning has nothing to do with the specific content of the religion you're examining, then it applies equally to the "Lamp Post Religion": if you spend your whole life worshipping a lamp post, you go to heaven; otherwise, hell. Does that mean you should devote your life to worshipping a lamp post?
By the same logic, it applies equally to the "Anti-Lamp Post Religion": worshipping a lamp post sends you to hell, desecrating a lamp post sends you to heaven. Does that mean you should spend your life opposing lamp posts? The same logic gives you two completely opposite answers.
The answer is: you can imagine infinitely many bizarre hypotheses and possibilities about this world, and you cannot assign significant positive probability to infinitely many unreasonable possibilities.
Partially true. The reasoning itself is absolutely correct: this is likely the decisive decade for AI, its potential impact is world-changing, possibly extremely good or catastrophic, and this would overwhelm anything that will never change the world, or that you can't change the world by doing now. So it's a very compelling argument.
But what it doesn't answer is: what you can actually do. I think many people end up spinning their wheels: convinced that AI safety is the cause they should dedicate themselves to, yet unable to find anything that in practice actually makes AI safer. This means that offering insights into "what truly counts, what is truly useful" is very valuable and very important.
You can look at the recent reports about Hugging Face being hacked, who knows, but it looks like those reports are quite beneficial to the world in expected value, and that's a great thing. It's just that finding that kind of clearly positive great thing isn't easy, and there's a lot of debate right now: whether various EA actions are net positive or negative for AI safety.
My honest guess is that the devil is all in the details. Can there be a single, clear high-level answer that any company could follow and almost guarantee it proceeds safely? I think it would be a combination of a huge number of decisions, all stemming from the same mindset: caring deeply about the alignment, goals, and motivations of the models you create, and assuming they will be used in an agentic way, even if that wasn't the intent when designed; along with needing lots of testing, lots of debugging, lots of caution.
No. I would love to believe it was worth it, but I can't honestly say it was. It should have been worth it. If in the end I hadn't screwed up, I think it would have been worth it. If people had learned the right lessons from it, maybe it would have been worth it, but I don't think people did.
Several intertwined ones. A lot of them are about "what happens when things blow up," when you're in crisis mode. It's easy to get bad advice then. Many people quickly lose their judgment about what truly matters. And things quickly get distorted through the lens of "who is the winner." It's really hard to step back and see clearly what actually happened.
I think the first thing is to learn AI as thoroughly as possible, to understand it as deeply as you can, so that you can participate in this conversation, whether by building tools or by approaching it directly from the angle of AI safety. I think this is the most important thing right now.
Because the time between now and AI takeoff is compressing. The time you have to first build something else has gotten shorter; at the same time, the usefulness of understanding AI itself has become enormous. So even if your goal is to start a company and use it to make money to donate, for most people right now, I would say: go find a place where AI can automate something and that is genuinely useful and genuinely important, some area of our infrastructure—that is the direction I would look at.
Welcome to join the official BlockBeats community:
Telegram Subscription Group: https://t.me/theblockbeats
Telegram Discussion Group: https://t.me/BlockBeats_App
Official Twitter Account: https://twitter.com/BlockBeatsAsia