As the Ethereum system becomes increasingly difficult to use, what solutions does God V have in mind?
The original title: the view | & have spent the etheric fang state scale management proposal"
Originally written by Vitalik Buterin
Jian is an Ethereum enthusiast
One of the most enduring and unresolved challenges facing the Ethereum protocol is the growing scale of state data. Many operations on the Ethereum blockchain (creating an account, writing to a contract storage slot, sending ETH to a new account...) Will add state content to Ethereum (that is, add data objects to state data), and all full nodes must store a full amount of state data in order to validate and manufacture new blocks. These operations require the sender of the transaction to pay a one-time Gas metering fee, but impose a permanent persistence cost on the entire network, as nodes need to store this new data (and future nodes need to download it during synchronization).
This is a significant imbalance in the system design that could make the Ethereum system increasingly difficult to use, as the state is filled with "junk data" that is no longer useful. The purpose of this article is to explain in detail the root cause of the problem, as well as some ways to resolve it. If we could implement a solution, it would pave the way for a significant increase in the block Gas cap to be safe.
The areas of research discussed in this article are still evolving, and newer, better ideas, and more elegant tradeoffs are possible at any time.
Introduction: What's the problem?
"State" refers to the information that a node must have if it is to process newly generated blocks and transactions. State is quite different from "history," which is information about past time that a node can keep for later rebroadcast or archiving, but is not required to process a blockchain.
In the Ethereum protocol, the status information includes:
· Account ETH balance & NBSP; And & have spent Nonce (serial number)
· Code of smart contract
· Storage items of smart contract
· Data related to the consensus mechanism (recent block hashes, tertiary blocks; Consensus data for proof of stake also includes the verifier's public key and its activities recorded on the beacon chain, etc.)
Historical information is made up of old blocks and receipts. There is no opcode in EVM that allows you to access old blocks, old transactions and content, and receipt output, so the node can still validate the new block by dropping this data, so this is historical information.
The last item in this list of status information -- data on consensus mechanisms -- has been carefully designed to limit its size, so we don't need to worry about it much. But the first three are too big for your head. The size of these three types of status messages grows over time as new users join the network, create new accounts, create new contracts, join contracts, receive tokens, etc.
The trouble is that many states just lie there once used (never to be touched again); Once a user deactivates an app, there is a "junk state" -- something that is never used again, but is there forever.
In theory, users can do "not fall garbage." Users can only publish with SELFDESTRUCT Conditional contract. When they can no longer use the contract, they will call this opcode to remove the contract and empty the token balance. They can also use a smart contract wallet to send transactions through an existing external holding account (EOA) without having to generate a new EOA (EOA status cannot be deleted).
In practice, however, such incentives are few and far between, and the technical complexity of proper state cleanup is too great. In many contracts, anyone is given such a call. SELFDESTRUCT Are inappropriate (people want "unterminable" apps!). Also, it adds a lot of complexity to the user experience and code. In fact, since SELFDESTRUCT With very limited use and very high side effects, I prefer to remove this opcode forever. If we really want to control the size of state data, what we need is a network of nodes that can. The default & have spent A way to discard a "garbage state" that is no longer used.
Stateless client
One type of solution to this problem is based on the concept of a "stateless client" (this article is the source of this concept, and here is a video of the presentation). The basic idea is to make block validation no longer conditional on having a global state. Instead, blocks bring their own evidence (or "witness" data) to prove the value of the state they accessed. As with the current design, the block will contain a "state root" to which the values accessed can be proved. Ethereum's current state tree scheme (Merkle Patricia Tree) supports such proof techniques, as can more efficient schemes like binary trees or Verkle Trie. The witness data will also prove the correctness of the new state root after processing the block.
Statelessness comes in two forms:
· Weak statelessness: the block producer still needs the complete state to generate witness data for the (self-manufactured) block; But the phase of validating blocks can be stateless;
· Strong statelessness: no node needs a complete turntable. In turn, it is the transaction sender who needs to provide witness data, which the block producer can aggregate. The transaction sender itself is responsible for storing part of the state tree needed to generate witness data for the account of concern.
Statelessness is a very strong "elegant" solution, because it is the responsibility completely transferred to the user, while in order to guarantee good user experience in practice, we need to create some type of agreement to help users maintain state, do not run individual nodes and handle user need interactions with accounts of the unexpected situation. Such agreements are hard to craft.
In addition, statelessness of all types increases the data bandwidth required by the network; Strong statelessness also requires the transaction to declare the accounts it interacts with and the keys to store the items (conceptually this is called an "access list").
A more modest solution: state expiration
More modest solutions boil down to different forms of "state expiration" solutions. The state must be continuously accessed to remain "active"; Long periods of neglect become "deactivated" (or "expired"). There are many options for what mechanism to use to update the state (such as prepaying a "rent" or simply accessing that state), but the general rule is that unless a state object is explicitly updated, it is in some form of a deactivated state. Therefore, any activity that creates new state objects (and updates existing state objects) can only become a burden on the node for a period of time, rather than a permanent burden as it is now.
The inactivated state, as the name implies, is not part of the "state"; Nodes that want to process blocks or create blocks do not need to store the deactivated state. However, the inactivation state is not completely deleted! In all types of state expiration proposals, there is a predetermined way to "resurrect" a deactivated state.
The general principle is that the active state is used as it is currently, and the inactive state is used through the mechanism of the stateless client described above. A transaction that resurrects an expired state object needs to provide evidence (witness data) that the object is part of the deactivated state. In order to be able to generate such evidence, users themselves need to store and maintain at least a portion of the deactivated state (the portion corresponding to the deactivated state object of their concern).
Will expire
There are many designs for determining expiration conditions. The most common ones are:
· Direct rent: Block by block "rent" is collected and paid directly out of the balance of each account (or other status object); When the balance of the status object drops to zero, the account expires.
· Remaining Survival Time Value: Each status object stores a "Remaining Survival Time" value, which can be increased by paying a fee.
· Touched-on-refresh: Each status object stores a "time remaining alive" value, which is incremented every time the account is read or written.
· All state objects expire periodically (for example every 6 months) : that is, the Regenesis proposal.
I myself have come to like the "touch and refresh" approach because (1) it avoids the need for an application to create a complex economic model to make the user pay the status rent; And (2) it ensures that there is a clear upper limit on the size of the active states (block Gas cap/Gas consumption of the touch state object × how long the state lives). Make a lot of state in accordance with the laws of time interval expired scheme (i.e. ReGenesis) have the same benefits, but there are some interesting trade-off: the key benefit is that expired scheme is simpler (without traversing the whole state tree one by one by one to inactivate state object), but the key is insufficient, crossing an expiration time, when you activate their state object again, how much data may be contact with you for the timing of the state object.
Expiration at the account level vs. Expiration at the slot level
The logic of state expiration can be run at either the account level or the individual storage slot level. Currently, I strongly prefer to implement the state expiration scheme at the storage slot level. Since the number of slots in many contract accounts is unlimited, any user can join a contract and increase the number of slots under the contract (for example, airdrop is an example that has been shown before). Regardless of the account layer expiration scheme used, to actually limit the size of the state, the amount of rent must be proportional to the number of storage slots in the contract (or inversely proportional to the lifetime). As a result, users will still be able to impose contracts and their users for a one-time fee. Perpetual sustainability costs.
To solve this problem, the contract either adds complex internal logic to "pass" the rental of the storage operation to the user, or redesigns the schema of its own contract to use the Create2 opcode to create new contracts and use those contracts to act as storage slots. Either way, you end up with the equivalent of an outdated solution at the slot level. Therefore, I personally believe that we should implement the state expiration scheme only at the contract storage slot level.
However, the slot level expiration scheme has its own disadvantages: each slot adds a metadata to indicate when it has expired (or whether it has been deactivated), which means that "resurrection conflict issues" (more on this below) can affect slots as well as accounts.
Remove from the state tree vs. Arrange a "retired" part of the state tree
Another technical point of view to distinguish between different state expiration proposals is "one tree flow" and "two tree flow". That is, do we have a single state tree as we do now, but some states are marked as expired; Or do you simply remove the deactivated state from the main state tree and move it to a separate tree (or other data) that contains only expired states?
A tree flow
The active nodes are marked in white and the inactive nodes are marked in gray
Note that even the middle nodes in the tree are marked as active or on fire (or, more realistically, each node is marked with the date of inactivation, so its activity can be easily checked); Tagging can be done at each node in the state tree (leaf node and intermediate node).
The second tree flow
The white tree contains the active states; The gray tree stores the deactivated state
The benefit of a tree stream is that, at the very least, it will look like the current state tree, and the deactivation and resurrection processes are relatively simple: the resurrection process simply refreshes the "expiration date" parameter of the relevant node in the tree, while the deactivation process is automated. The drawback is that it requires a tree structure that can store intermediate information in this way within nodes, and it doesn't scale well to Verkle trees. In addition, it requires additional merkle proof elements that can not only sink to the leaf node, but also stop at the intermediate node (in case it is necessary to prove that a part of the state has expired).
The advantage of binary streams is that the current, pure form state accumulator can support such schemes without the need to add metadata for each node. The downside is that it requires some deeper changes to the entire protocol, and requires an explicit process to inactivate the state (so expiration is no longer automated). Also, there is no built-in solution for reviving the conflict dilemma (see the next section), so you have to choose between two approaches.
Note that in a binary stream, the data structure that stores the deactivated state is not a tree. In fact, it's entirely possible to resurrect a state object by simply providing a Merkl tree pointing to the receipt for when the object was deactivated, along with some cryptographic evidence that the object hasn't been resurrected before (or recently re-expired).
The resurrection of conflict
Then we come to one of the key conundrum of state expiration schemes: "resurrecting conflicts." The concept of resurrecting conflict is as follows. Suppose an account is generated from address A; This account is overdue; Then address A creates A new account (for example, using the CREATE2 opcode to ensure that the address is the same for both generated accounts); Finally, address A tries to resurrect the original account. What happens at this point?
Here are a few possible solutions:
1. Explicitly "account merging" process: similar to the rule that "except for the accumulation of the ETH balances of the two accounts, the state of the old account shall prevail" or "except for the accumulation of the ETH, the state of the new account shall prevail"; Even more, the contract code of the old account can dictate the special consolidation process
2. Ensure that resurrection conflicts do not occur by eliminating the ability to duplicate deployments to the same address: i.e., adjust the functionality of CREATE2 to include the current time in the data primitive that is eventually hashed to the address, so that even if the same data is generated in the future, the same address cannot be obtained
3. Add a "stub" to the status object to prevent new accounts from being generated in the same place (the one tree stream method above automatically implements this function)
4. Requiring a new account to be generated with proof that the account has not expired before: In a sense, this is equivalent to a stub scheme, except that the stub is placed in a separate part of the state, so any user who wants to create a contract account must keep track of this part of the state
(Note that if we were to use a slot expiration scheme, either of the above solutions would have to extend to the single slot level and not stop at the account level)
The main concerns were :(1) it would add a lot of complexity to the application and they would need to incorporate logic; (2) Once this is done, unless an address is "registered" on the chain, users can no longer easily obtain an address with which they can interact and accumulate assets (such as ERC20 Tokens). The unregistered address is important: any user who receives ETH for the first time is using an unregistered address. At the root of this (2) concern is the fact that unregistered addresses are effectively time-limited, so if a user generates an address, receives funds, but forgets to send a transaction (that is, forgets to "register") within the next year, his funds will be locked.
Note that EOA is not immune. It seems possible, though, because the merge process for EOA is relatively simple (just add the old ETH balance to the new one, or EIP 169 for Nonce). There are, however, two problems. First, the goal of account abstraction is to replace EOA with contracts, and the merging process of contracts for account abstraction may not be straightforward. Second, not only the EOA itself is affected by expiration and resurrection events, but also the associated storage keys (such as the ERC20 token balance) in the application in which the EOA is involved, so complex merge logic is still required.
So, from my perspective, the least disruptive is some form of stub solution. However, there is an information theory problem with the stub scheme that can lead to some strange results. To prevent new state objects from being created at N state object locations that have expired, a collection covering the N addresses (and/or storage keys) must be part of the state. If the set is information-minimized (that is, contains only these addresses), then the size of the set will be O(N), and therefore the size of the state will be O(N); So, the size of the activated state is going to be proportional to the size of the inactivated state, so we haven't really solved the problem.
Tree rot
The only way to solve this problem is to cover more than those N accounts; In effect, we would have to make the entire tree inaccessible. (Again, that's the essence of a treestream solution: If two accounts get expired, all the space in between them also gets expired implicitly.)
And here's another problem: This creates a form of "tree rot" that, over time, makes all parts of the state tree inaccessible to the creation of a new account, at least to users who aren't keeping track of the region's expired state.
Secondary problems caused by moldy trees must also be addressed. For example, if a contract is to create a subcontract, it must be able to create a contract in a state area that is either not moldy or where the user has witness data (perhaps with a "prompt" from the user). A solution to the tree moldy problem can be found here: persistently open new areas for account creation. Another idea is for each user to select some region of the state (for example, 1/256 of the state), track changes to that region (including expiration state) so that witness messages can be created, and create accounts only in that region.
Another problem with tree moldy is that it requires an explicit data structure to store and check ranges. It is best if a tree has data that can be placed in a node to indicate which parts below that node have expired (as used by a treestream solution), but a key-value pair store can be quite difficult to do.
So let's go back to strong statelessness
Many of the problems that arise from using tree structures in state expiration schemes can be traced back to the fact that we need to agree on which states are active and which are deactivated. In the two-tree model, this point is more obvious; But even in a treestream mode, the state tree needs to be explicitly marked so that an Ethereum node that recently downloaded state using a fast synchronous download can determine whether a transaction that attempted to access an account without providing a witness message should succeed or fail. Can we do that without having to make that distinction clear?
Wouldn't it solve this problem if we achieved complete statelessness and then could help transaction senders and block producers reliably get the state they need to witness message generation? So what can be done to help transaction senders and block producers do this?
A natural approach is for nodes in the network to store only a portion of the state tree, for example, that has been accessed in the past year. Simply add a voluntary setting to the client Settings. If we want to be more reliable, we can force the miner to store at least some data by introducing a Proof of Custody scheme.
One thing to note is that if the consensus layer is not aware of which states are active and which are inactivated, the Gas overhead for accessing recent and old states is the same. This leads to two outcomes:
1. The GAS overhead of accessing recent state also needs to be further increased
2. The upper limit on the block size containing witness messages can be very large if a block is full of transactions accessing old state (probably 800 bytes * 12.5M gas / 2400 gas per access ~= 4.1MB
If we want to avoid these disadvantages, we need to keep track of which state objects (including areas of the address space that have not yet been filled) are active in the consensus, which brings us back to properties close to the state expiration scenario. Once again, statelessness vs. state expiration (state rent) is a spectrum, a complex trade-off space, not an either-or choice.
Rollup needs, and can, use the same solution
An important medium-term scalability solution for Ethereum is Rollups. However, Rollup itself doesn't need to worry about the size of state data anymore; In fact, the issue of the state size of the Rollup system is exactly the same as that of the Ethereum chain itself.
Fortunately, if we can come up with a solution, then at least EVM Rollup (the Rollup solution that tries to replicate the Ethereum runtime environment to the maximum extent possible) will be able to use the same solution to address the scale of its internal state. Therefore, state size management is complementary to scalability solutions such as rollup and sharding (state size management is complementary to rollups, sharding and other scaling strategies).
(The term "complementary" in this context is highly misleading.)
conclusion
State size is a worsening problem, and a solution to state size could also pave the way for a significant increase in the block Gas cap. Some form of state expiration should be agreed upon and implemented. However, there are significant technical tradeoffs between the different solutions, especially if we also want to preserve some of the important attributes of the current design.
Some attributes we may need to sacrifice include:
· Users can generate accounts offline and receive funds at that address, and can silence attributes for any length of time before making the address visible on the chain
· Address length is maintained at 20 bytes (rolling state expansion schemes require a larger address space, although address length may need to change quickly for collision resistance in the first place)
· The state can be thought of as a "pure" key-value pair stored property, as well as a property that does not need to store metadata in each node of the state tree
· Existing applications require varying degrees of rewriting to ensure that users can generate witness data without storing all the deactivated states
· Gas consumption; Or the difficulty of creating a new contract and writing to a new slot
If we are prepared to make sacrifices, there are options that can be started very quickly. On the other hand, perhaps in time we can fix or better aggregate these concepts, reduce problems, and especially make them technically easier to implement (for example, allowing "pure" key-value pair storage). We should have a deeper understanding of what sacrifices we are more/less willing to accept and continue to actively study proposals for improvement.
Welcome to join the official BlockBeats community:
Telegram Subscription Group: https://t.me/theblockbeats
Telegram Discussion Group: https://t.me/BlockBeats_App
Official Twitter Account: https://twitter.com/BlockBeatsAsia
Related Articles
