Delphi Digital: Fully read the Ethereum sharding roadmap

The Hitchhiker's Guide to Ethereum

Original author: Delphi Digital

The Way of DeFi

The key points

Ethereum is the only major protocol that builds a scalable unified billing and data availability layer

Rollups scales computing while taking advantage of ethereum's security

All roads lead to the end of centralized block production, decentralized trustless block verification and anti-censorship

Innovations such as proposer-builder separation and weak statelessness unlock this separation of authority (build and validate) to achieve scalability without sacrificing security or decentralization

Mevs are now front and center -- extensive design is planned to mitigate their hazards and prevent their centralization

Danksharding combines multiple leading-edge research approaches to provide the scalable base layer required for Ethereum's rollup-centric roadmap

I do hope to achieve danksharding in our lifetimes

Introduction to the

I've been skeptical of The Merge timeline ever since Vitalik said that a person born today has a 50-75% chance of living to 3000 years and that he wants to live forever. But call it a joke, but look ahead to ethereum's ambitious roadmap.

This is not a quick read. If you want a broad and nuanced view of Ethereum's ambitious roadmap -- give me one hour of focus and I'll save you months of work.

A lot of ethereum research has to be tracked over time, but everything ultimately fits into one overarching goal -- scaling computing without sacrificing decentralized validation.

I hope you're familiar with Vitalik's famous "Endgame" article. In his article, he acknowledged the need for some Centralization to achieve scale. The "C" word is scary in blockchain, but it's a fact. We just need to maintain that power through verification of decentralization and distrust. There is no compromise here.

Professionals will build the modules for L1 and beyond. Ethereum is still very secure with simple decentralized validation, and Rollup inherits their security from L1. Ethereum then provides settlement and data availability, allowing Rollup expansion. All of the research here is ultimately looking to optimize these two roles while making the full validation chain easier than ever.

Here is a glossary for abbreviating words that will appear ~ 43531756765713534 times:

DA - Data availability

DAS - Data availability sampling

PBS - Proposer - builder separation

PDS - original danksharding

DS - Danksharding

PoW -- Proof of work

PoS - Proof of interest

Part I: The Road to Danksharding

Hopefully you've heard by now that Ethereum has moved to a rollup-centric roadmap. No more execution sharding -- Ethereum will instead be optimized for rollups that require a lot of data. This is done through data sharding (Ethereum's plan) or large blocks (Celestia's plan).

The consensus layer does not interpret sharded data. It has one job to do -- make sure the data is available.

I assume that you are already familiar with basic concepts such as Rollup, fraud, and ZK proof, and why DA is important. If you're not familiar with it or just need a refresher, Can's recently released report on Celestia covers it all.

Raw data sharding design - Independent sharding proposer

The design described here has been abandoned, but it is a valuable scenario. For simplicity, I'll call this "Sharding 1.0."

Each of the 64 sharded blocks has its own proposer and committee, rotated from verifier to verifier. They separately verify that their shard data is available. Initially this is not a DAS -- it relies on the majority of each shard's validator set to fully download the data.

This design introduces unnecessary complexity, a worse user experience, and attack vectors. Shuffling verifiers between shards is tricky.

Unless you introduce very strict synchronization assumptions, it is difficult to guarantee that voting will take place in a single slot. Beacon block proposers need to collect votes from all individual committees, and there may be delays.

DS is completely different. The verifier performs the DAS to verify that all data is available (no more separate sharding committees). A specialized builder creates a large block with a beacon block and all shard data identified together. PBS is therefore necessary for DS to remain decentralized (building large blocks together is resource-intensive).

Data availability sampling

Rollups publishes a lot of data, but we don't want nodes to download all that data. This would mean high resource demand, which would undermine decentralization.

In contrast, DAS allows nodes (and even light clients) to easily and securely verify that everything is available without having to download everything.

- Simple solution -- just check a bunch of random blocks in the block. If they all sign, I'll sign. But what if you miss out on a deal that gives all ETH to Sifu? Money is no longer safe.

- Smart solutions - Erase data first. Extend the data with the Reed-Solomon code. This means that the data is interpolated as a polynomial, which we then evaluate in many other places. This is done in one go, so let's break it down.

For those of you who have trouble with math, let's do this very quickly. (I promise this won't be a really scary math -- I had to watch some Khan Academy videos to write these parts, but I get it now).

A polynomial is an expression summing over any finite number of CXK terms. Degree is the highest index. For example, 2 x3+6 x2+2 x-4 is a cubic polynomial. You can reconstruct any d-degree polynomial from any D +1 coordinates located on that polynomial.

Now let's take a concrete example. Below we have four data blocks (d0 to d3). These data blocks can be mapped to the evaluation of the polynomial F (X) at a given point. For example, f (0) = d0. Now you have the least degree polynomial that runs through these estimates. Since these are four blocks, we can find cubic polynomials. We can then extend this data to add four evaluations along the same polynomial (e0 through e3).

Remember the key polynomial property -- we can reconstruct it from any four points, not just our original four data blocks.

Back to our DAS. Now we just need to ensure that any 50% (4/8) erasure code data is available. From this, we can reconstruct the entire block.

Therefore, an attacker must hide more than 50% of the block to successfully trick the DAS node into thinking that the data is available when it is not.

After many successful random samples, < The probability of 50 percent being available is very small. If we successfully sample the erasure code data 30 times, then It's 2-30 out of 50.

KZG commitment

Ok, so we did a bunch of random samples, and they were all available. But we have another question -- is the data erasure code correct? Otherwise, maybe the block producer just added 50% of the garbage when he extended the block, and we sampled the crap. In this case, we would not actually be able to reconstruct the data.

Usually we just submit a lot of data by using Merkle roots. This is valid for proving that the set contains some data.

However, we also need to know that all the original and extended data are on the same low-degree polynomial. Merkle root doesn't prove it. So if you use this scheme, you'll also need proof of fraud in case you get it wrong.

Developers are working in two directions:

- Celestia is taking the anti-fraud route. One needs to note that if the block is incorrectly erasing the encoding, they will submit proof of fraud to alert everyone. This requires standard honest minority assumptions and synchronization assumptions (that is, in addition to someone sending me proof of fraud, I also need to assume THAT I am connected and will receive it within a limited time).

- Ethereum and Polygon Avail are taking a new route -- the KZG Promise (aka Kate Promise). This eliminates the honest minority and synchronous security assumptions about proof of fraud (although they are still there for reconstruction, which we will cover shortly).

Other solutions exist but are not actively pursued. For example, you can use ZK proof. Unfortunately, they are computationally impractical (for now). However, they are expected to improve in the coming years, so Ethereum may shift to STARK in the future as KZG promises to be quantum-resistant.

Going back to THE KZG commitment -- this is a polynomial commitment scheme.

A commitment scheme is simply a cryptographic way of proving a commitment to some value. The best analogy is to put a letter in a locked box and hand it to someone else. Once inside, the letter cannot be changed, but it can be opened and authenticated with a key. Your new submission is a promise, and the key is proof.

In our example, we map all the raw and extended data onto an X,Y grid and then find the least degree polynomial that passes through them (a process called Lagrange interpolation). This polynomial is what the prover will promise:

Here are the key points:

- We have a polynomial f of X.

- The prover makes a "commitment" to the polynomial C (f)

This relies on elliptic curve cryptography with trusted Settings. For more details on how it works, see a great post from Bartek

For any "evaluation" of the polynomial y = f (z), the prover can compute the "proof" π(f,z)

Given the promise C (f), the proof π(f,z), any position z, and the evaluation y of the polynomial at z, the verifier can confirm that indeed f (z) = y

Explanation: The prover provides these fragments to any verifier, who can then confirm that the estimate of a point (where the estimate represents the underlying data) is correctly placed on the promised polynomial

This proves that the original data is correctly extended, since all estimates are on the same polynomial

Notice that the verifier doesn't need the polynomial f (X)

Important attributes - This has O (1) commitment size, O (1) proof size, and O (1) verification time. Even for the prover, the promise and proof generation are only extended on O (d), where D is the degree of the polynomial

Explanation: Even if n (the number of values in X) increases (i.e., as the dataset increases with larger shard blobs) - the commitment and proof remain constant in size, and validation requires constant effort

Both the promise C (f) and the proof (f,z) are just an elliptic curve element on a pair-friendly curve (this will use BL12-381). In this case, they are only 48 bytes each (very small)

As a result, the prover submits a large amount of raw data and extended data (multiple evaluations representing polynomials) that are still only 48 bytes long, and the proof is only 48 bytes long

TLDR - This is highly extensible

Then, the KZG root (polynomial promise) is similar to the Merkle root (which is a vector promise) :

The original data is the polynomial f (X) computed at f (0) to f (3), and then we extend it by computing the polynomial at f (4) to f (7). All points f (0) through f (7) are guaranteed to be on the same polynomial.

Bottom line: DAS allows us to check whether erasure code data is available. KZG's commitment proves to us that the raw data has been properly expanded and promises all of this.

Good job. That's algebra for today.

KZG commitment and fraud proof

Now that we know how KZG works, take a step back and compare the two approaches.

Disadvantages of KZG - it is not post-quantum-secure, it requires a trusted setup. None of this is worrying. STARK offers a post-quantum alternative, whereas a trusted setting (open participation) requires only one honest participant.

Advantages of KZG - lower latency than fraud proof setup (although, as mentioned earlier, GASPER is not fast deterministic by any means), and it ensures correct erase encoding without introducing the few assumptions of synchronization and honesty inherent in fraud proof.

However, considering ethereum still reintroduces these assumptions for block reconstruction, you don't actually remove them. The DA layer always needs to plan for scenarios that initially provide blocks, but then nodes need to communicate with each other to put them back together. This reconstruction requires two assumptions:

You have enough nodes (light nodes or full nodes) to sample the data so that they are sufficient to regroup them together. This is a fairly weak and inevitable honest minority assumption, so it's not a big problem.

Reintroduce the synchronization assumption -- nodes need to be able to communicate over a period of time before they can be put back together.

Ethereum validators fully download sharded BLOBs in PDS, and with DS, they will only perform DAS (download allocated rows and columns). Celestia will require the verifier to download the entire block.

Note that in either case, we need synchronous assumptions for reconstruction. If the block is only partially available, the full node must communicate with other nodes to put it back together.

If Celestia were to switch from requiring verifiers to download all the data to performing only DAS (although no such switch is currently planned), the delay advantage of KZG would come to light.

Then, they also need to implement the KZG promise -- waiting for proof of fraud would mean a significant increase in block spacing, and the risk of the verifier voting for the miscoded block would be very high.

I recommend the following for a deeper exploration of the way KZG promises to work:

Elliptic curve cryptography (relatively easy to understand) introduction

Explore elliptic curve pairing -- Vitalik

KZG polynomial promises -- Dankrad

How does the trusted setting work? - Vitalik

The proposer - builder separation in the agreement

Today's consensus node (miner) and merged (verifier) play two roles. They build the actual block and then propose it to other consensus nodes that validate it. Miners "vote" by building on the previous block, and after the merge, verifiers vote directly on whether the block is valid or invalid.

PBS separates them -- it explicitly creates a new intra-protocol builder role. Professional builders will put blocks together and bid for proposers (verifiers) to select their blocks. This combats the centralized power of MEV.

Think back to Vitalik's Endgame -- all roads lead to centralized block production, with untrusted and decentralized verification. PBS compiled it. We need an honest builder to provide services to the network to increase activity and censorship-resistant (two are needed for an efficient market), but the set of validators needs an honest majority. PBS makes the proposer role as simple as possible to support verifier decentralization.

Builders receive priority fee tips as well as any MEVs they can draw. In an efficient market, competitive builders will bid for the full value they can extract from the block (minus their amortized costs, such as powerful hardware, etc.). All value permeates a decentralized set of validators -- exactly what we want.

The exact PBS implementation is still under discussion, but two-slot PBS might look something like this:

- Builders use their bid to commit block headers

- Beacon block proposer selects the winning block head and bid. Even if the builder fails to produce the body, the bidder will win the bid unconditionally

- The reference Committee confirms the winning header

- The builder publishes the winning body

- Independent References Committee Elected by the winning body (if the winning builder disagrees, it is voted to be absent)

The proposer is selected from the validator set using the standard RANDAO mechanism. We then use the commitment-reveal scheme, which does not reveal the full block body until the committee confirms the block header.

Commit-reveal is more efficient (sending around hundreds of complete block bodies might overwhelm the bandwidth of the P2 P layer), and it also prevents MEV theft. If the builder is going to commit their full block, another builder can see it, figure out that strategy, merge it, and quickly release a better block. In addition, a mature proposer can detect the MEV policy in use and replicate it without compensating the builder. If this MEV theft were balanced, it would incentivize merging builders and proposers, so we used commit-reveal to avoid this situation.

After the proposer chooses the winning block head, the committee confirms and fixes it in the fork selection rule. The winning builder then publishes the complete "Builder block" body of their win. If released in time, the next committee will testify. If they fail to publish on time, they still pay the proposer the full bid (and lose all MEVs and fees). This unconditional payment eliminates the proposer's need to trust the builder.

The downside of this "two-slot" design is delay. The merged blocks will be fixed at 12 seconds, so if we don't want to introduce any new assumptions, here we need 24 seconds to get the full block time (two 12-second time slots). Eight seconds per slot (16 seconds of block time) seems like a safe compromise, although research is ongoing.

crList

Unfortunately, PBS gives the builder a higher level of ability to vet transactions. Maybe the builders just don't like you, so they ignore your deal. Maybe they did such a good job that all the other builders gave up, or maybe they just overpaid for the block because they really didn't like you.

CrLists checked this capability. The exact implementation is again an open design space, but "mixed PBS" seems to be the most popular. The proposer specifies a list of all eligible transactions they see in the memory pool, and the builder will be forced to include them (unless the block is full) :

The proposer publishes a crList and a summary of the crList that includes all eligible transactions

The builders create a proposed block body, then submit a bid that includes a hash of the crList summary, proving they have seen it

The proposer accepts the bid and block header from the winning builder (they haven't seen the body yet)

Builders publish their blocks with proof that they have included all transactions from the crList or that the block is full. Otherwise, the fork selection rule will not accept the block

The prover checks the validity of the published body

There are still important issues to be resolved here. For example, the dominant economic strategy here is to have the proposer submit an empty list. Even the review builder can win the auction as long as the highest bid is made. There are some ideas to address this and other issues, but only to emphasize that the design here is not set in stone.

Two dimensional KZG scheme

We saw how THE KZG commitment allowed us to commit data and prove that it was properly extended. However, I have simplified the actual operation of Ethereum. It does not commit all data in a single KZG promise -- a single block will use many KZG promises.

We already have a dedicated builder, so why not let them create a huge KZG commitment? The problem is that it takes a powerful supernode to rebuild. We can accept the super node requirement of the initial build, but we need to avoid the reconstruction assumption here. We needed entities with fewer resources to handle rebuilding, and breaking it up into many KZG commitments made this feasible. Given the amount of data at hand, reconstruction might even be fairly common, or the base case assumption in this design.

To make reconstruction easier, each block will contain m shard BLOBs encoded in m KZG promises. Naively doing so would result in a lot of sampling -- you would execute the DAS on each shard blob to know it was all available (m* K samples, where K is the number of samples per blob).

Instead, Ethereum will use the 2d KZG scheme. We use the Reed-Solomon code again to extend the M promise to the 2 m promise:

We make it a 2-D scheme by extending the additional KZG commitment (256-511 here) on the same polynomial as 0-255. Now we just need to DAS the table above to ensure data availability for all shards.

The 2-d sampling requirement for 75% of the data available (as opposed to the previous 50%) meant that we did more fixed samples. Before I mention the 30 samples of the DAS in the simple one-dimensional scenario, but this would require 75 samples to ensure the same probabilistic probability of reconstructing the usable blocks.

Shard 1.0 (with a one-dimensional KZG commitment scheme) requires only 30 samples, but 64 shards need to be sampled if you want to check the full DA of 1920 samples. Each sample is 512 B, so this requires:

(512 B x 64 fragments x 30 samples) /16 seconds = 60 KB/s bandwidth

In practice, the verifier is simply shuffled without individually checking all the shards.

Now, combined blocks using the 2d KZG commitment scheme make checking the full DA trivial. It requires only 75 samples of a single unified block:

(512 B x 1 block x 75 samples) /16 seconds = 2.5 KB/s bandwidth

Danksharding

PBS was originally designed to weaken the concentration power of MEV on the validator set. However, Dankrad recently took advantage of this design and realized that it unlocked a much better sharding structure, DS.

DS utilizes specialized builders to create beacon chains that perform tighter integration of blocks and shards. We now have a builder working together to create the entire block, a proposer and a committee. Without PBS, DS would not be feasible -- regular validators cannot handle the huge bandwidth of blocks filled with Rollup data bloBs:

Shard 1.0 includes 64 independent committees and sponsors, so each shard may not be available individually. The tighter integration here allows us to secure DA once and for all. Data is still "sharded" behind the scenes, but from a practical point of view, Danksharding is starting to feel more like big blocks, which is great.

Danksharding - Honest majority verification

The verifier proves that the data is available as follows:

This relies on an honest majority of verifiers -- as a single verifier, my columns and rows being available are not enough to make me statistically confident that the entire block is available. It depends on the honest majority of people who say it is. Decentralized validation is important.

Note that this is different from the 75 random samples we discussed earlier. Private random sampling is a way for resource-poor individuals to easily check availability (for example, I can run the DAS light node and know that the block is available). However, the verifier will continue to use the row and column methods to check availability and guide block rebuilding.

Reconstructing Danksharding -

As long as 50% of a single row or column is available, the sample validator can easily reconstruct it completely. When they rebuild any missing blocks in rows/columns, they reassign those blocks to orthogonal lines. This helps other validators rebuild any missing blocks from intersecting rows and columns as needed.

The safety assumption for rebuilding the available blocks here is:

- Enough nodes to perform sample requests so that together they have enough data to rebuild the block

- Broadcast synchronization assumptions between nodes in their respective blocks

So, how many nodes is enough? There are roughly 64,000 individual instances (so far, more than 380,000). This is also a very pessimistic calculation that assumes no crossover among nodes run by the same validator (a far cry from nodes being limited to 32 ETH instances). If you sample more than 2 rows and columns, the chance that you can retrieve them collectively increases because of the crossover. This starts to scale quadratically -- if verifiers run 10 or 100 verifiers, 64,000 might be several orders of magnitude less.

If the number of online validators starts to get unusually low, you can set DS to automatically reduce the shard data BLOB count. Therefore, the safe assumption will be lowered to the safe level.

Danksharding - Private random sampling of malicious majority security

We see that DS validation relies on honest majorities to prove blocks. As an individual, I can't prove to myself that a block works by downloading only a few rows and columns. But private random sampling can give me that assurance without trusting anyone. As mentioned earlier, this is where the node checks the 75 random samples.

The DS will not initially include private random sampling, as this is a very difficult problem to solve on the network side (PSA: maybe they could actually use your help here!). .

Note that "private" is important, because if an attacker de-anonymizes you, they can trick a small number of sample nodes. They can just return the exact block you requested and keep the rest. Therefore, you don't know that all data is available just from your own sampling.

Danksharding -- Key points

Besides being a sweet name, DS is also very exciting. It finally realized ethereum's vision of a unified settlement and DA layer. This tight coupling between Beacon blocks and shards essentially pretends not to be shard.

In fact, let's define why it's even considered "sharding." The only remnant of sharding is that the verifier is not responsible for downloading all the data. And nothing more.

So if you're questioning now whether this is really still shard, you're not crazy. This distinction is why PDS (which we'll cover soon) is not considered "shard" (even though it has "shard" in its name, and yes, I know it's confusing). PDS requires each verifier to fully download all shard BLOBs to prove their availability. The DS then introduces sampling, so each verifier downloads only a portion of it.

Fortunately, minimum shard means a much simpler design than slice 1.0 (so fast delivery, right?). . To simplify, they include:

- There may be hundreds of lines less code in the DS specification than in the Shard 1.0 specification (thousands less on the client side)

- There is no sharding committee infrastructure, committees only need to vote on the main chain

- Individual shard BLOb confirmations are not tracked, now they are all confirmed in the main chain, or they are not

One good result of this is the market for data consolidation charges. Shard 1.0 with different blocks made by different proposers distracts this.

The elimination of sharding committees also strengthens the fight against bribery. The DS validator votes the entire block once per epoch, so the data is immediately validated by 1/32 of the entire validator set (32 slots per epoch). Shard 1.0 verifiers also vote once in each epoch, but each shard has its own committee reorganized. Thus, each shard is only confirmed by 1/2048 of the validator set (1/32 divided into 64 shards).

As mentioned earlier, the combination of blocks with the 2D KZG commitment scheme also makes DAS more efficient. Shard 1.0 requires a bandwidth of 60 KB/s to check the full DA for all shards. The DS only requires 2.5 KB/s.

There is another exciting possibility for DS -- synchronous calls between ZK-Rollups and L1 Ethereum implementations. Transactions from a sharded BLOb can be immediately acknowledged and written to L1 because everything is generated in the same beacon chain block. Sharding 1.0 eliminates this possibility due to separate sharding validation. This allows for an exciting design space, which can be very valuable for something like shared mobility (e.g. DAMM).

Modular base layers scale gracefully -- more decentralization leads to more scaling. This is fundamentally different from what we see today. Adding more nodes to the DA layer allows you to safely increase data throughput (that is, more room for rollups to exist at the top).

There are still limits to how scalable blockchain can be, but we can scale orders of magnitude higher than anything we see today. Secure and extensible base layers allow execution to proliferate above them. Improvements in data storage and bandwidth will also allow for higher data throughput over time.

It is certainly possible to exceed the DA throughput envisaged here, but it is hard to say where this maximum will end up. No clear red lines, but areas where assumptions would start to make people uncomfortable:

Data storage - This has to do with DA and data accessibility. The purpose of the consensus layer is not to guarantee data accessibility indefinitely. What it does is make it available long enough for anyone who cares to download it to satisfy our security assumptions. Then it gets dumped anywhere -- which is comfortable, because history is one in N trust assumptions, and we don't actually talk about that much data in the grand scheme of things. As throughput increases by several orders of magnitude, this could enter uncomfortable territory in a few years.

Verifier - the DAS needs enough nodes to rebuild the block together. Otherwise, attackers may wait and only respond to the queries they receive. If these queries provided are not enough to rebuild the block, then the attacker can keep the rest and we're out of luck. To safely increase throughput, we need to add more DAS nodes or increase their data bandwidth requirements. This is not the throughput issue discussed here. However, if throughput increases further by several orders of magnitude than this design, this can be uncomfortable.

Note that the builder is not the bottleneck. You need to generate KZG certificates quickly for 32 MB of data, so you need a GPU or fairly powerful CPU with at least 2.5 GBit/s bandwidth. In any case, this is a specialized role and a negligible business cost to them.

Proto-danksharding (EIP-4844)

The DS is great, but we have to be patient. PDS is designed to tide us over -- it implements the necessary forward compatibility steps on DS on an accelerated timeline (for the Shanghai hard fork) to provide orders of magnitude of expansion in the interim. However, it does not actually implement data sharding (that is, the validator needs to download all the data separately).

Today's Rollups use L1 "CallData" for storage, which is always on the chain. However, Rollup only needs DA for a reasonable period of time so that anyone interested has enough time to download it.

Ip-4844 introduces a new bloB-enabled transaction format that Rollup will use for future data stores. Blobs carry large amounts of data (~125 KB), and they are much cheaper than similar amounts of CallData. The data BLOB is then removed from the node after a month, which reduces storage requirements. This is enough time to satisfy our DA safety assumptions.

Current Ethereum blocks typically average about 90 KB (call data is about 10 KB of that). PDS unlocks more DA bandwidth for bloBs (target ~ 1 MB and Max ~ 2 MB) as they are pruned after a month. They don't become a permanent drag on nodes.

A BLOB is a vector of 4096 field elements, each 32 bytes long. PDS allows up to 16 blocks per block, while DS will increase to 256.

PDS DA bandwidth = 4096 x 32 x 16 = 2 MiB per block, and the target is 1 MiB

DS DA bandwidth = 4096 x 32 x 256 = 32 MiB per block, target 16 MiB

Each step expands by an order of magnitude. PDS still requires consensus nodes to fully download data, so it is conservative. The DS distributes the load of storing and propagating data between the validators.

Here are some good things that IP-4844 introduced on the road to DS:

Transaction format that carries data BLOBs

KZG's commitment to BLOB

All the execution-level logic required by DS

All execution/consensus cross-validation logic required by DS

Layer separation between BeaconBlock validation and DAS BLOb

Most of the BeaconBlock logic required by DS

Blob's self-adjusting independent gas Prices (Multidimensional IP-1559 with index pricing rules)

Then DS will add further:

PBS

Data acquisition system

2 D KZG scheme

Escrow certificate or similar protocol requirement for each verifier to verify the availability of a specific portion of sharded data in each block (may be about a month)

Note that these data BLOBs are introduced as a new transaction type on the execution chain, but they do not impose additional requirements on the execution side. EVM only looks at commitments attached to the BLOB. The execution-level changes made with IP-4844 are also forward compatible with DS, and no further changes are required in this regard. Then upgrading from PDS to DS requires only changing the consensus layer.

The data BLOB is fully downloaded by the consensus client in the PDS. The BLOB is now referenced in the Beacon block body, but not fully encoded. Instead of embedding the entire content into the body, you propagate the bloB's content separately as a "sidecar." Each block has a BLOB sidecar, which is fully downloaded in the PDS, and then DAS will be performed on it using the DS validator.

We discussed earlier how to commit bloBs using the KZG polynomial promise. However, INSTEAD of using KZG directly, IP-4844 implements what we actually use -- versioned hashing of it. This is a 0 x01 byte (representing this version) followed by the last 31 bytes of KZG's SHA256 hash.

We did this to make it easier to achieve EVM compatibility and forward compatibility:

EVM compatibility - KZG promises to be 48 bytes, while EVM more naturally uses 32 byte values

Forward compatibility - If we switch from KZG to something else (STARKs is available for anti-quanta), these promises can continue to be 32 bytes

The PDS eventually creates a custom data layer -- the data BLOB will have its own unique expense market, with separate floating gas prices and limits. Therefore, even if a certain   NFT  The project sells a bunch of monkey lands on L1, and your Rollup data costs don't go up (although the settlement costs prove to be). This acknowledges that the primary cost of any Rollup today is publishing their data to L1 (rather than proving it).

The gas fee market remains unchanged and the data BLOB is added as a new market:

Blob fees are charged by GAS, but it is a variable amount adjustment based on its own EIP-1559 mechanism. The long-term average number of BLOBs per block should equal the target.

You actually have two auctions running in parallel -- one for computation and one for DA. This is a huge leap forward in efficient resource pricing.

Here are some interesting designs. For example, it might make sense to change the current gas and BLOB pricing regime from linear IP-1559 to a new exponential IP-1559 regime. The current implementation does not average our target block size in practice. Today, the basefee is not fully stabilized, resulting in the average gas used per block being observed exceeding the target by about 3% on average.

Part II: History & AMP; State management

Here's a quick review of some basics:

History -- everything that happens on the chain. You can paste it on your hard drive because it doesn't require quick access. In the long run, 1 of N honestly assumes.

Status - a snapshot of all current account balances, smart contracts, and so on. The full node (currently) requires this to validate the transaction. It's too big for RAM and the hard drive is too slow - it sits in your SSD. High-throughput blockchains inflate their state far beyond what the rest of us can keep on our laptops. If everyday users can't hold state and they can't fully verify, say goodbye to decentralization.

TLDR - These things get really big, so it's hard to run a node if you let the node store them. If running a node is too hard, we ordinary people won't do it. That's bad, so we need to make sure that doesn't happen.

Calldata Gas Cost reduction and total Calldata limitation (IP-4488)

PDS is an important stepping stone to DS, and it checks many final requirements. Implementing PDS over a reasonable time span can advance the timeline on the DS.

Easier to implement band-Aids is IP-4488. It's not as elegant, but it still solves the cost emergency. Unfortunately, it does not implement steps in the implementation of DS, so all the inevitable changes will still need to be made later. If it's starting to feel like THE PDS is going to be a little slower than we'd like, it might make sense to quickly go through IP-4488 (it's just a few lines of code change) and then implement the PDS about 6 months later. The exact timetable is still uncertain.

Ip-4488 has two main components:

Reduce the calldata cost from 16 gas per byte to 3 gas per byte

Add a limit of 1 MB of call data per block and an additional 300 bytes per transaction (theoretical maximum is about 1.4 MB)

Restrictions need to be added to prevent a worst-case scenario -- a block full of Calldata would reach 18 MB, which is far beyond ethereum's processing capacity. Ip-4488 increases ethereum's average data capacity, but due to this Calldata limit (30 million gas/ 16 gas per CallData byte = 1.875 MB), its burst data capacity actually decreases slightly.

Ip-4488 has a much higher sustained load than PDS because this is still calldata with BLOB, which can be pruned after a month. Ip-4488 will significantly accelerate historical growth, making it a bottleneck for operational nodes. Even if IP-4444 is implemented with IP-4488, this will only remove the execution load history after one year. A low sustained load of PDS is clearly desirable.

Binding historical data in the execution client (IP-4444)

IP-4444 allows clients to choose to locally prune historical data (title, body, and receipt) that is more than a year old. It requires the client to stop providing this trimmed history data at the P2 P layer. Pruning history allows customers to reduce user disk storage requirements (currently hundreds of GB and growing).

This is already important, but if IP-4488 is implemented (since it significantly increases history), it will basically be mandatory. Anyway, hopefully this will be done relatively soon. Eventually some form of history will need to expire, so this is a good time to deal with it.

Full synchronization of chains requires a history, but it is not required to validate new blocks (it only requires state). Thus, once the client is synchronized to the top of the chain, the history data is retrieved only when explicitly requested via JSON-RPC or when a peer attempts to synchronize the chain. With the implementation of IP-4444, we need to find alternative solutions for these.

Clients will not be able to use Devp 2p  as they do now; Do "full synchronization" -- instead they will "checkpoint synchronization" from weak subjective checkpoints, which they treat as creation blocks.

Note that weak subjectivity is not an additional assumption -- it is inherent to the transition to PoS anyway. Due to the possibility of remote attacks, this requires synchronization with effective weak subjective checkpoints. The assumption here is that clients will not synchronize from invalid or old weak subjective checkpoints. The checkpoint must be within the period in which we begin pruning the historical data (in this case, within a year), otherwise the P2 P layer will not be able to provide the required data.

This will also reduce bandwidth usage on the network as more clients adopt lightweight synchronization strategies.

Retrieving historical data

IP-4444 prune history data after a year sounds good, while PDS prune blob faster (after about a month). We absolutely need these, because we can't ask the nodes to store all this and remain decentralized:

IP-4488 - Long term may include about 1 MB per slot to add about 2.5 TB of storage per year

PDS - Target about 1 MB per slot, adding about 2.5 TB of storage per year

DS - Target about 16 MB per slot and add about 40 TB of storage per year

But where did the data go? Don't we need it anymore? Yes, but note that there is no risk to the protocol for losing historical data -- just for a single application. The job of ethereum's core protocol, then, should not be to perpetuate all agreed data.

So, who's going to store it? Here are some potential contributors:

Individual and institutional volunteers

Block browsers (such as EtherScans.io), API providers, and other data services

Third-party indexing protocols like TheGraph can create incentive marketplaces where customers can pay servers for historical data as well as Merkle certificates

Clients in the Portal Network (currently under development) can store random portions of the chain's history, and the Portal Network automatically directs data requests to the nodes that own it

BitTorrent, for example. A 7 GB file containing BLOB data from the block is automatically generated and distributed every day

Application-specific protocols, such as Rollup, can require their nodes to store sections of history relevant to their application

The long-term data storage problem is a relatively simple one because it is the 1 of N trust assumption we discussed earlier. We are years away from being the ultimate limit on blockchain scalability.

Weak stateless

Ok, so we've got history management pretty well covered, but what about state? State issues are actually the main bottleneck for improving Ethereum TPS right now.

The full node takes the pre-state root, performs all transactions in the block, and checks if the post-state roots match what they provide in the block. To know if these transactions are valid, they currently need the state at hand -- validation is stateful.

Enter the stateless age - we will not need states to function. Ethereum is pushing for "weak statelessness," which means that states are not required to validate blocks, but are required to build blocks. Validation becomes a pure function -- give me a completely isolated block and I can tell you if it's valid or not. It basically goes like this:

Thanks to PBS, the builder still needs the states to be acceptable -- they will be more centralized, high-resource entities anyway. Focus on going to the center to test your credentials. Weak statelessness creates more work for the builder and much less work for the verifier. It's a great trade-off.

You achieve this magical stateless execution through Witness. These are proof that the builder will start to include the correct state access in each block. You don't actually need the entire state to validate a block -- you just need the state that transactions in that block read or affect. Builders will begin to include fragments of the state affected by the transaction in a given block, and they will prove that they have accessed that state correctly through witness.

Let's take an example. Alice wants to send Bob an ETH. To authenticate a block with this transaction, I need to know:

Before the transaction -- Alice has 1 ETH

Alice's public key -- so I know the signature is correct

Alice's random number -- so I know that the transactions were sent in the correct order

After executing the trade -- Bob gains 1 ETH, Alice loses 1 ETH

In a weakly stateless world, the builder adds the above witness data to the block and verifies its accuracy. The verifier receives the block, executes it, and decides if it is valid. That's all!

Here are the implications from the verifier's point of view:

The huge SSD requirement for holding state is gone -- a key bottleneck for scaling today.

Because you now have to download witness data and certificates, the bandwidth requirements increase. This is the bottleneck of the Merkle-Patricia tree, but it is gentle, not the bottleneck of Verkle's attempt.

You still perform the transaction for full validation. Stateless recognition is not currently the bottleneck for scaling Ethereum.

Weak statelessness also allows Ethereum to loosen its self-imposed restrictions on the throughput of its execution, and state bloat is no longer a pressing issue. Raising the gas limit by about three times might make sense.

Most user execution will take place on L2 anyway, but higher L1 throughput will still be beneficial even for them. Rollups relies on Ethereum for DA (publishing to shards) and settlement (which requires L1). As Ethereum expands its DA layer, the amortized cost of publishing proof may become a larger share of Rollup costs (especially for ZK-Rollups).

Verkle Tries

We obscure how these testimonies actually work. Taifang currently uses merkle-Patricia trees to represent states, but the required Merkle proof is too large for these witnesses to implement.

Ethereum will turn to Verkle for state storage. Verkle's proof is much more efficient, so they can serve as viable witnesses to weak statelessness.

First let's review what a Merkle tree looks like. Every transaction is hashed -- the hashes at the bottom are called leaves. All hashes are called "nodes" and they are hashes of the two "child" nodes below them. The resulting final hash is a "Merkle root".

This is a useful data structure for proving the inclusion of transactions without having to download the entire tree. For example, if you want to verify that the transaction H4 is included, you only need H12, H3, and H5678 in the Merkle proof. We have H12345678 from the block header. Therefore, light clients can ask full nodes for these hashes and then hash them together based on the route in the tree. If the result is H12345678, then we have successfully proved that H4 is in the tree.

However, the deeper the tree, the longer the path to the bottom, so you need more projects to prove it. Thus, a shallow and wide tree would seem to facilitate efficient proof.

The problem is, if you try to make the Merkle tree wider by adding more children under each node, it's very inefficient. You need to hash all siblings together to climb the tree, so you need to receive more siblings for Merkle proof. This will make the proof size huge.

This is where the promise of efficient vectors comes in. Note that the hashes used in Merkle trees are really vector promises -- they are effectively only committed to two elements. So we want vector commitment, we don't need to accept all the siblings to validate it. Once we have that, we can make trees wider and reduce their depth. That's how we get the size of the valid proof -- by reducing the amount of information that needs to be provided.

A Verkle trie is similar to a Merkle tree, but it uses a valid vector promise (hence the name "Verkle") instead of a simple hash to promise its children. So the basic idea is that each node can have a lot of children, but I don't need all of them to prove that. Regardless of width, it is proof of constant size.

In fact, we've already introduced a good example of one of these possibilities -- a KZG commitment can also be used as a vector commitment. In fact, that's what ethereum developers originally planned to use here. They have since turned to Pedersen to commit to a similar role. These will be based on elliptic curves (Bandersnatch in this case), and they will each submit 256 values (much better than two!) .

So why not have the widest possible depth tree? This is very useful for verifiers who now have super compact proofs. But there is a practical tradeoff that the prover needs to be able to compute the proof, and the wider it is, the harder it is. Therefore, these Verkle attempts will be located between 256 value wide extremes.

State of maturity

Weak statelessness removes the state inflation constraint for the verifier, but the state does not magically disappear. Transaction costs are limited, but they impose a permanent tax on the network by increasing state. State growth remains a permanent drag on the network. Something needs to be done to address the underlying problem.

This is where the state expires. Being inactive for a long period of time (say, a year or two) can even be removed from the list of things a block builder needs to carry. Active users don't notice anything, and useless states that are no longer needed can be discarded.

If you need to restore the expired state, all you need to do is display a certificate and reactivate it. So this goes back to the 1 of N storage assumption here. As long as someone still has a full history (block browser, etc.), you can get what you need from them.

Weak statelessness reduces the immediate need for state expiration at the base layer, but in the long run, especially as L1 throughput increases, this is a good thing. This would be a more useful tool for high-throughput summaries. L2 states will grow at an order of magnitude faster, even dragging down high performance builders.

Part III -- MEV

PBS is necessary for the safe implementation of DS, but recall that it was actually originally designed to counter the centralized power of MEV. You'll notice a recurring trend in Ethereum research today -- MEV is now front and center in cryptoeconomics.

Designing blockchains with MEV is critical to maintaining security and decentralization. The basic protocol-level approach is:

Minimize harmful MEVs (e.g., single slot deterministic, single secret leader election)

Democratize the rest (e.g., MEV-Boost, PBS, MEV smoothing)

The rest must be easily captured and propagated between verifiers. Otherwise, since it cannot compete with complex searchers, it will centralize the set of validators. This is exacerbated by the fact that the combined MEV will have a higher share of the verifier award (the pledge issue is well below the inflation rate given to miners). Not to be ignored.

MEV supply chain today

Today's sequence of events is as follows:

Mineral pools play a builder's role here. MEV searchers relay trade packages (and their respective bids) to the mine pool via Flashbot. The pool operator aggregates a complete block and passes the block headers to individual miners. Miners prove this by assigning weights to PoW in fork selection rules.

Flashbots were created to prevent vertical integration of the entire stack -- which would open the door to censorship and other unsavory externalities. By the time Flashbots appeared, the pool had already begun making exclusive deals with trading companies to extract MEVs. Instead, Flashbots gives them an easy way to aggregate MEV bids and avoid vertical integration (by implementing MEV-GEth).

After the Ethereum merger, the mining pool will disappear. We want to open the door to validator nodes that work properly in the home. This requires someone to fill a specialized construction role. Your home verifier node may not be as good at capturing MEVs as a hedge fund with quantified salaries. If left unchecked, if ordinary people can't compete, this will centralize the set of validators. If structured properly, the agreement can redirect MEV revenue to the pledge revenue of the daily verifier.

MEV-Boost

Unfortunately, the PBS in the agreement simply won't be ready for the merger. Flashbots once again provides a stepping stone solution - MEV-Boost.

The merged validator will default to receiving common memory pool transactions directly into its executing client. They can package it up, give it to a consensus client, and broadcast it to the network. (If you need to understand how ethereum's consensus and execution clients work together, I'll cover that in Part 4).

But as we discussed, your verifier doesn't know how to extract MEV, so Flashbots offers an alternative. Mev-boost will plug into your consensus client, allowing you to outsource specialized block building. Importantly, you still have the option of using your own execution client as a backup.

MEV searchers will continue to play the role they do today. They will run specific strategies (statistical arbitrage, atomic arbitrage, sandwiches, etc.) and bid on their bundled trade packages. The builders then aggregate all the bundles they see, as well as any private order flows (for example, from Flashbots Protect), into the best complete block. The builder passes the block header to the verifier only through a relay running to the MEV-Boost. Flashbots intends to run Repeaters and builders and plans to decentralize over time, but whitelisting other builders may be slow.

Mev-boost requires the verifier to trust the repeater -- the consensus client receives the block header, signs it, and displays the block body. The purpose of the repeater is to prove to the proposer that the body is valid and exists so that the verifier does not have to trust the builder directly.

When the PBS within the protocol is ready, it encodes the content provided by MEV-Boost in the meantime. PBS provides the same separation of powers, allowing easier decentralization of builders and eliminating the need for proposers to trust anyone.

Committee-driven MEV smoothing

PBS also opened the door to another cool idea -- committee-driven MEV smoothing.

We see that the ability to extract MEV is a central force of the verifier set, but so is the distribution. The high variability of MEV rewards from block to block encourages the pooling of many verifiers to smooth out your returns (as we see today in mining pools, though to a lesser degree here).

The default setting is to provide the builder's full payment to the actual block proposer. MEV smoothing will instead distribute payments to many verifiers. A committee of verifiers will examine the proposed block and certify whether it is indeed the highest bidder. If all goes well, the block proceeds and the award is divided between the committee and the proposer.

This also solves another problem - out-of-band bribery. For example, proposers could be incentivize to submit suboptimal blocks and directly accept out-of-band bribes to conceal their payments from the principal. This certificate checks the proposer.

Intraprotocol PBS is a prerequisite for MEV smoothing. You need to understand the builder market and the specific bids submitted. There are several open research questions here, but this is an exciting proposal that is once again crucial to ensuring decentralized verifiers.

Single slot transaction finality

Quick finality is great. Waiting around 15 minutes is not optimal for UX or cross-chain communication. More importantly, this creates a MEV reorganization problem.

The combined Ethereum already provides more powerful confirmation than today -- thousands of validators proving that each block competes with miners and may mine at the same block height without a vote. That would make restructuring highly unlikely. However, this is still not really the end of the deal. If the last block has some fat MEV, you might just trick the verifier into trying to reorganize the chain and steal it for yourself.

Single-slot determinism eliminates this threat. Restoring a final block requires at least one-third of the verifiers, and their pledge tokens are immediately cut (millions of ETH).

I wouldn't pay too much attention to the underlying mechanics here. Single slot finality is way down the road in Ethereum's roadmap, it's a very open design space.

In today's consensus protocol (with no slot finality), Ethereum only needs 1/32 of a verifier to prove each slot (approximately 12,000 out of approximately 380,000 currently). Extending this voting to a full set of validators using BLS signature aggregation in a single slot requires more work. That would compress hundreds of thousands of votes into a single validation:

Vitalik breaks down some interesting solutions here.

Single Secret Leader Election (SSLE)

SSLE attempts to patch up another MEV attack vector that we will face after the merger.

The beacon chain verifier list and the upcoming leader selection list are public and easy to de-anonymize and map their IP addresses. You can probably see the problem here.

More sophisticated verifiers can use tricks to better hide themselves, but smaller verifiers are particularly susceptible to Doxxed and subsequent DDOSd. This can easily be used for MEVs.

Let's say you're the proposer of block N, and I'm the proposer of block N +1. If I know your IP address, I can do a cheap DDOS attack on you and you will time out and not be able to generate your blocks. I can now capture our slot's MEV and double my reward. The elastic block size of IP-1559 (maximum gas per block is twice the target size) exacerbates this situation, so I can cram what would otherwise be two block transactions into a single block twice as long as I am now.

Family verifier nodes may abandon validation because they are under constant attack. SSLE makes it impossible for anyone but the proposer to know when it is their turn to block such attacks. This will not take effect at the time of the merger, but hopefully it can be implemented soon.

Part IV -- Consolidation

Okay, just so you know, I was joking. I actually think (and hope) that ethereum consolidation will happen relatively quickly.

We're all talking about this topic, so I feel obligated to at least give it a brief introduction.

The combined client

Today, you run a single client that handles everything (for example, Go Ethereum, Nethermind, etc.). Specifically, the full node performs the following operations simultaneously:

Execution -- Each transaction in the block is executed to ensure validity. Get the pre-state root, perform all operations, and check that the generated post-state root is correct

Consensus - Verify that you are on the heaviest (highest PoW) chain that gets the most work done (i.e. Satoshi Consensus)

They are indivisible because the full nodes follow not only the heaviest chain but also the heaviest effective chain. This is why they are full nodes rather than light nodes. Even if a 51% attack occurs, the whole node will not accept invalid transactions.

Beacon chain currently only runs consensus to give PoS a test run. Does not include execution. The final total difficulty will be determined, at which point the current Ethereum execution block will be merged into the beacon chain block to form a single chain:

However, the full node will run two independent clients in the background that can interoperate:

Execution client (aka "Eth1 client") - The current Eth 1.0 client continues to process execution. They process blocks, maintain memory pools, and manage and synchronize state. The PoW was partially torn out.

Consensus client (aka "Eth2 client") -- The current beacon chain client continues to process PoS consensus. They follow the head of the chain, Gossip and the proof block, and get the verifier reward.

The client receives the beacon chain block, executes the client run transaction, and if all goes well, the consensus client will follow the chain. You will be able to mix and match execution and consensus clients of your choice, all of which will be interoperable. A new engine API will be introduced for clients to communicate with each other:

Or:

The consensus after the merger

Today's Satoshi consensus is simple. Miners create new blocks and add them to the heaviest effective chains observed.

The merged Ethereum turned to GASPER -- a combination of Casper FFG (the ultimate deterministic tool) and LMD GHOST (the forking selection rule) to reach consensus. The TLDR here - this is an activity that is good for consensus, not good for security.

The difference is that security-friendly consensus algorithms (such as Tendermint) stop when they fail to get the necessary number of votes (set here to two-thirds of verifiers). Pro-chain activity (e.g., PoW + Satoshi Consensus) will continue to build an optimistic ledger no matter what, but they will not reach final certainty without enough votes. Today's Bitcoin and Ethereum will never be finalized -- you just assume there won't be a reorganization after a sufficient number of blocks.

However, Ethereum will also get enough votes through regular checkpoints to achieve finality. Each 32 ETH instance is an individual verifier, and there are already over 380,000 beacon chain verifiers. Epochs consists of 32 slots, all verifiers split and prove one slot within a given epoch (meaning about 12,000 proofs per slot). The fork selection rule LMD Ghost then determines the current head of the chain based on these proofs. A new block is added to each slot (12 seconds), so the epoch is 6.4 minutes. Finality is usually achieved with the necessary votes after two epochs (that is, 64 slots, although up to 95 May be required).

conclusion

All roads lead to the end of centralized block production, decentralized untrusted block verification, and anti-censorship. Ethereum's roadmap targets this vision.

Ethereum aims to be the ultimate unified DA and settlement layer -- massively decentralized and secure at the bottom and computationally scalable at the top. This condenses cryptographic assumptions into a robust layer. The unified modular base layer that includes execution also captures the highest value in L1 design -- achieving monetary premium and economic security, as I recently described (and now open source here).

I want you to get a better sense of how ethereum research is interwoven. We have so many moving pieces, very cutting edge, and a very big picture that goes around in your head.

Fundamentally, it all comes back to that single vision. Ethereum offers a compelling path to massive scalability while cherishing the values that we care so much about in this space.

Special thanks to Dankrad Feist for his comments and insights.

The original link

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia