V God details the limitations of the expansion of the base infrastructure

原文标题：《 首发丨 V 神最新研究：区块链可扩展性的局限性，反驳马斯克的狗狗币扩展方式 》

Translated by: Babbitt

原文作者： 以太坊创始人 Vitalik Buterin

How scalable can blockchain really be? Is it possible to "reduce fees by a factor of 100 by shortening block time by a factor of 10 and increasing block size by a factor of 10", as Elon Musk hopes, without resulting in extreme centralization and undermining the essential nature of the blockchain? If not, how scalable can blockchain really be? What if you choose to extend it by changing the consensus algorithm? More importantly, what if you changed the technology to introduce a feature like ZK-snark or sharding? Theoretically, a sharded blockchain could keep adding more shards, would that lead to more things being added?

As it turns out, there are some important and rather subtle technical factors that limit the scalability of blockchains, even one that uses sharding. In many cases, we have scalability solutions, but these solutions have limitations. This article explores many of the problems with these scalability scenarios.

All the problems can be solved by simply adding parameters. But at what cost?

Ordinary users will also be able to run nodes, which is critical for blockchain decentralization

At 2:35 a.m., you receive an urgent call from a partner on the other side of the world who is helping you manage a mine pool (or possibly a pledge pool). Your partner tells you that starting about 14 minutes ago, your pool and several other pools suddenly split off the chain, with the original chain still carrying 79% of the network's computing power. Depending on your node, the block produced by this majority chain (original chain) is invalid. This presents a balance error: the key block appears to have mistakenly assigned 4.5 million additional tokens to an unknown address.

An hour later, you're talking to two other pools just as confused as you are, as well as a few block browsers and exchanges in a telegraphic group. Finally you see someone send out a link to Twitter. "The new On-Chain Sustainable Agreement Development Fund is officially launched," the tweet read.

By morning, Twitter and community forums were full of uncensored arguments, and discussions were everywhere. But by then, many of the 4.5 million new tokens had already been converted to other assets on the chain, leading to billions of dollars in Defi transactions. 79% of consensus nodes and all major browsers and light wallets are following this new chain. Maybe this new developer fund will fund some of these developments, or maybe all of this money will be gobbled up by leading mining pools, exchanges, etc. But, whatever the outcome, the fund is a fait accompli for all intents and purposes, and the average user is powerless to fight back.

The show is about to begin. Maybe it could be funded by Molochdao or some other agency

Could this happen to your blockchain? The elite of your blockchain community, including mine pools, block browsers, and managed nodes, are likely to coordinate well. They were probably all in the same telegraph group and WeChat group. If they really want to make a sudden change to the rules of the agreement to advance their interests, then they may do so. The Ethereum blockchain once completely resolved a consensus failure in 10 hours; If your blockchain has only one client implementation, and you only need to deploy code changes to dozens of nodes, then coordinating changes to the client code can be faster. The only reliable way to nullify this coordinated social attack is through the active defense of decentralized communities, i.e., users.

Imagine if users were running nodes that validated the blockchain (either directly or through more advanced indirect techniques) and automatically rejected blocks that broke the rules of the protocol, even though more than 90% of miners or mortgagers supported the protocol. If each user was running an authentication node, this attack would quickly fail: some pools and exchanges would fork, but this would seem silly. However, even if only some users run the authentication node, the attack does not give the attacker a victory. On the contrary, this leads to confusion, as different users will see different chains. At the very least, the ensuing market panic and possible continued chain splits would significantly reduce the profits available to attackers. The amount of time required to resolve the conflict will itself discourage the attacker from attacking.

Let's be clear: Your resistance to malicious protocol changes comes from a culture of user-validated blockchains, not PoW or PoS.

- Hasu

What does Hasu say about this

If your community consists of 37 node runners and 80,000 passive listeners who are responsible for checking signatures and block headers, the attacker wins. If everyone in your community is running the node, the attacker will fail. We don't know the exact threshold for herd immunity against coordinated attacks, but one thing is absolutely clear: more nodes is better, fewer nodes are worse, and we certainly need more than a few dozen or a few hundred.

So, how much work do we need to do at the full node?

To maximize the number of users that can run nodes, we will focus on general consumer hardware. You can add some network capacity by requiring specialized hardware that is easy to buy (from Amazon, for example), but this doesn't really add much to scalability.

There are three key limitations to the ability of a full node to handle a large number of transactions:

Computing power: What percentage of CPU power is required to safely run a node?

Bandwidth: Given the current reality of Internet connectivity, how many bytes can a block contain?

Storage: How many gigabytes of disk storage can we ask users to store? Also, how fast is the read speed required? Can HDD mechanical hard disk be used? Or do you need a SSD?)

Many of the misconceptions about how far blockchain can be scaled using "simple" technologies result from being overly optimistic about each of these numbers. We can introduce these three factors one by one:

Ability to calculate

Error: 100% of CPU power can be used for block validation

Correct: Approximately 5-10% of CPU power can be used for block validation

There are four main reasons for such a low percentage:

We need a margin of security to deal with the possibility of a DoS attack (a transaction by an attacker to exploit a weakness in the code takes longer to process than a regular transaction)

Nodes need to be able to synchronize chains when offline. If I disconnect from the Internet for a minute, I should be able to sync up again within seconds

Run nodes should not drain the battery too quickly or slow down all other applications

Nodes also need to perform other non-block production tasks, which revolve around validating and responding to incoming transactions and requests over the P2P network.

Note that until recently, most responses to the question "Why only 5-10%?" "Concentrates on a different problem: because PoW blocks are random, it takes a long time to validate blocks, increasing the risk of creating multiple blocks simultaneously. There are many ways around this problem (for example, Bitcoin NG, or using POS). But these fixes don't address the other four issues, so they don't provide the huge scalability benefits that many people initially thought they would.

Parallelism is not a panacea. Often, even seemingly single-threaded blockchain clients have been parallelized: signatures can be verified by one thread, execution done by another, and a separate thread handles the transaction pool logic in the background. Also, the closer you get to 100% utilization of all threads, the more energy you consume running nodes, and the lower your margin of safety against DOS.

bandwidth

Error: If we were producing 10 MB blocks every 2-3 seconds, most users would have a low network speed. 10 MB/s, so of course they can handle such blocks

Correct: Maybe we can process blocks of 1-5 MB every 12 seconds. It's hard enough.

These days, we often hear advertising statistics about how much bandwidth an Internet connection can provide: 100 Mbps or even 1 Gbps is common. However, there is a big difference between the advertised bandwidth and the expected actual bandwidth for several reasons:

"Mbps" means "millions of bits per second"; A bit is 1/8 of a byte, so you need to divide the bit number of the AD by 8 to get the number of bytes in the AD.

Like all companies, Internet providers often lie.

There are always multiple applications using the same Internet connection, so nodes cannot take up the entire bandwidth.

P2P networks inevitably come with their own overhead: nodes often end up downloading and re-uploading the same block multiple times (not to mention transactions broadcast through MBlockQuotePool before being included in the block).

StarkWare did an experiment in 2019 when they first released a 500 KB block because the reduction in the cost of gas for transaction data made this possible for the first time, but there were actually several nodes that could not handle blocks of that size. Since then, the ability to process large blocks has improved and will continue to improve. But no matter what we do, we still can't naively capture average bandwidth in megabits per second and convince ourselves that we can accept latency of 1 second and be able to have blocks of that size.

storage

Error: 10 TB

Correct: 512 GB

As you might guess, the main argument here is the difference between theory and practice. In theory, you can buy an 8-TB SSD from Amazon (you do need SSD or NVME; HDDs are too slow to store blockchain state). In fact, the laptop used to write this blog has 512 GB of storage, and if you let people buy their own hardware, many of them will just become lazy (or they can't afford the $800 8 TB SSD) and use a centralized provider. Even if you can install a blockchain onto a storage, a high level of activity can be easily burned to disk quickly and force you to keep buying new disks.

A team of blockchain protocol researchers surveyed how much disk space each person had. I know the sample size is small, but it still tells you something

In addition, the storage size determines the time it takes for new nodes to come online and start participating in the network. Any data that the existing node must store is data that the new node must download. The initial synchronization time (and bandwidth) is also a major barrier to the user running the node. At the time of writing this blog, it took me about 15 hours to synchronize the new Geth node. If Ethereum usage increases by a factor of 10, it will take at least a week to synchronize a new Geth node, which will most likely result in your network connection being limited. This is especially important during an attack when a successful response to an attack involves a new user who has never run a holiday point.

The interaction effect

Moreover, there is an interaction between the three costs. Because databases internally use a tree structure to store and retrieve data, the cost of retrieving data from a database increases as the logarithm of the database size increases. In fact, because the top layer (or layers at the top) can be cached in RAM, the disk access cost is proportional to the size of the database and is a multiple of the size of the data cached in RAM.

Don't take this picture literally. Different databases work in different ways, and the portion of memory is usually just a single layer (but large) (see the LSM tree used in LevelDB). But the basic principle is the same

For example, if the cache is 4 GB and we assume that each layer of the database is 4 times larger than the previous layer, then Ethereum's current ~ 64 GB state will require approximately 2 accesses. However, if the state size is increased four times to ~ 256 GB, then this increases to ~ 3 accesses. Thus, a fourfold increase in the GAS limit actually translates into an approximately sixfold increase in block validation time. The effect can be even stronger: Reading and writing usually takes longer when the hard drive is full than when the hard drive is running low.

So what does this mean for Ethereum?

Today, in the Ethereum blockchain, although still feasible on regular hardware (I'm just syncing a node on my laptop as I write this!), , but running a node has become a challenge for many users. So we are approaching a bottleneck. The biggest concern for core developers is storage size. Therefore, for the time being, it is unlikely that heroic efforts to address computing and data bottlenecks, or even changes to consensus algorithms, will result in a significant increase in the accepted GAS limit. Even solving Ethereum's biggest prominent DOS vulnerability would only increase the gas limit by 20%.

The only way to address the storage issue is Statelessness and State expiry. Statelessness allows a class of nodes to validate the blockchain without maintaining permanent storage. State expiration releases recently unaccessed states, requiring the user to manually provide evidence that they continue to be maintained. Both paths have been in use for a long time, and proof-of-concept implementations of statelessness have begun. These two improvements, combined, could significantly alleviate those concerns and open up room for a significant increase in the gas limit. But even after statelessness is implemented and the state expires, the GAS limit may safely only increase about threefold before other limitations start to dominate again.

What happens after sharding?

Sharding fundamentally overcomes these limitations, as it decouples the data contained in the blockchain from the data that individual nodes need to process and store. Instead of the nodes validating the blocks by downloading and executing them themselves, advanced mathematics and cryptography are used to validate the blocks indirectly.

As a result, a sharded blockchain can safely have very high transaction throughput that a non-sharded blockchain cannot. It does take a lot of cryptographic ingenuity to create effective alternatives that can successfully reject full validation of invalid blocks, but it can be done: the theory has been established and a proof-of-concept based on the draft specification is under study.

Ethereum is planning to use secondary sharding, and its overall scalability is limited by the fact that nodes must be able to handle individual shards and beacon chains, which in turn must perform some management work for each shard. If the shard is too large, the node will no longer be able to process a single shard. If there is too much sharding, the node will no longer be able to process the beacon chain. The product of these two constraints forms a kind of upper bound.

You could conceivably go further by cubic sharding or even exponential sharding. In this design, data availability sampling will certainly become more complex, but it can be done. But Ethereum will not adopt sharding any higher than secondary sharding. The reason is that sharding from the sharding of transactions does not actually achieve additional scalability gains unless other risks start to become unacceptably high.

So what are these risks?

Minimum number of users

It is conceivable that a non-sharded blockchain can work even if only one user is willing to participate. A sharded blockchain is not like this: no single node can process the entire chain, so you need enough nodes so that they can at least process the blockchain together. If each node can handle 50 Tps and the whole chain can handle 10,000 Tps, then the chain needs at least 200 nodes to survive. If the chain has fewer than 200 nodes at any point, then either the node can no longer keep up with the chain or the node stops detecting invalid blocks, or other undesirable conditions may occur, depending on how the node software is set up.

In fact, due to the need for redundancy (including for data availability sampling), the minimum number of safe nodes is several times higher than simply "chain TPS divided by node TPS"; For the example above, we say 1000 nodes.

If the capacity of the sharded blockchain is increased by a factor of 10, the minimum number of users will increase by a factor of 10. Now, you may be asking: Why don't we start at a time when capacity is small, increase capacity when we see a lot of users, and decrease capacity when the number of users decreases?

There are actually some problems with this:

The blockchain itself cannot reliably detect how many unique users there are, so this will require some kind of governance to detect and set the number of sharding. Managing against capacity constraints can easily become a source of disruption and conflict.

What if many users quit suddenly and unexpectedly?

Increasing the minimum number of users required to start a fork makes a hostile takeover more difficult.

The minimum number of users is 1,000, which is almost certainly possible. On the other hand, if you set the minimum number of users to 1 million, that's not going to work. Even setting the minimum number of users to 10,000 is a bit risky. Therefore, it seems difficult to justify a sharded blockchain with more than a few hundred shards.

History is easily detectable

The important attribute of a blockchain that users really value is permanence. When the company goes bankrupt or loses interest in maintaining the ecosystem, the digital assets stored on the servers cease to exist within 10 years. NFT on Ethereum, on the other hand, is permanent.

Yes, in the year 2372, people will still be able to download and check your crypto cat.

But once the capacity of the blockchain gets too high, it becomes more difficult to store all this data, until at some point there's a big risk that some part of history will eventually... No one stores it.

Quantifying this risk is easy. Take the data capacity of the blockchain (MB/SEC) as the unit, and multiply by ~ 30 to get the amount of data stored in TB per year. The data capacity of the current sharding plan is about 1.3 MB/s, so it is about 40 TB/year. If you increase it by a factor of 10, it becomes 400 TB per year. If we want the data not only to be accessible, but also to be easily accessible, then we need metadata (for example, decompressing aggregate transactions), so that's 4 petabytes per year, or 40 petabytes after 10 years. The Internet Archive uses only 50 petabytes. Therefore, this is the upper limit of reasonable size for the security of the sharded blockchain.

So it looks like the Ethereum sharding design has actually been roughly aiming for a fairly reasonable maximum security on both dimensions. You can increase the constant a little bit, but not too much.

conclusion

There are two ways to try to scale the blockchain: basic technological improvements and simply adding parameters. First, adding parameters sounds appealing: if you're doing the math on a napkin, it's easy to convince yourself that a home laptop can handle thousands of transactions per second, without the need for ZK-snark, summarization, or sharding. Unfortunately, there are many subtle reasons why this approach is fundamentally flawed.

A computer running a blockchain node cannot spend 100% of its CPU power to validate the blockchain; They need a large margin of security against unexpected DoS attacks, they need spare capacity to perform tasks such as processing transactions in the memory pool, and you don't want to have nodes running on a computer that can't be used by any other applications at the same time. Bandwidth is also tricky: a 10 MB/s connection doesn't mean you can have 10 MB of blocks per second! Maybe 1 to 5 MB blocks every 12 seconds. This is the same as with storage. Increasing the hardware requirements for running nodes and limiting the running of nodes to specialized actors is not a solution. For a decentralized blockchain, it's critical for the average user to be able to run nodes and develop a culture where running nodes is a normal activity.

On the other hand, fundamental technological improvements are possible. Currently, Ethereum's main bottleneck is storage size, and statelessness and state expiration can solve this problem and allow it to increase by up to about 3 times -- but not more, as we want to run a node much easier than it is now. A sharded blockchain can scale further because a single node in a sharded blockchain does not need to process every transaction. But even with a sharded blockchain, there are limits to capacity: as capacity increases, the minimum number of safe users increases, and the cost of archiving the blockchain (and the risk of data loss if no one goes to archive the chain) goes up. But let's not worry too much: these limits are high enough that we can process more than a million transactions per second while keeping the blockchain fully secure. But some work will have to be done without sacrificing the decentralized nature that makes blockchains so valuable.

Special thanks to Fepx Lange, Martin Swende, Marius van der Wijden and Mark Tyneway for their feedback and comments on this article.

The original link

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia