Avail: Polygon's scalable data availability layer

DeFi之道

Read this article in 16 Minutes

Avail is a general-purpose, scalable, data-availability-focused blockchain designed for standalone chains, sidechains, and off-chain scaling solutions.

Original Title: "Understanding Polygon's New Product Avail, a Scalable Data Availability Layer"

Original Author: Polygon

Original Source: DeFi Pulse

Avail—an essential part of how the future of blockchains will operate. Avail is a generic, scalable, data availability-centric blockchain designed for standalone chains, sidechains, and off-chain scaling solutions.

Avail provides a robust data availability layer by utilizing highly secure mathematical primitives—performing data availability checks using erasure coding with key innovations. We use Kate polynomial commitments to create a 2D data availability scheme, resistant to fraud proofs, without requiring an honest majority assumption, and not relying on honest full nodes for confidence in data availability.

Avail offers a generic data availability layer usable across various execution environments such as standalone chains, sidechains, and off-chain scaling solutions. Long-term, it will facilitate a range of experiments in execution environments and finality implementations without teams and projects needing to bootstrap their security. Chains built using the Polygon SDK, Cosmos SDK, or Substrate can benefit from Avail for this purpose.

Avail decouples transaction execution and validity from the consensus layer, where consensus is only responsible for a) ordering transactions and b) ensuring their data availability.

Main Objectives

· Enable standalone chains or sidechains with arbitrary execution environments to guide validator security by ensuring transaction data availability without needing to create and manage their validator set

· Layer 2 solutions like Validiums to provide increased scalability throughput by using Avail as an off-chain data availability layer

We have been researching Avail in secret since the end of 2020, and currently, it is in the Devnet phase. The testnet is under development. More detailed information about the problem, architecture, and solutions can be found in the reference documentation, including references to the codebase.

Background

In today's Ethereum-like ecosystem, there are primarily three types of nodes:

· Validation Node

· Full Node

· Light Client

A block is attached to the blockchain by a validator node that collects transactions from the mempool, executes them, generates the block, and propagates it through the network before including it. This block contains a small block header that includes a digest and metadata relevant to the transactions included in that block. The entire network's full nodes receive this block and validate its correctness by re-executing the transactions included in the block. Light clients only fetch block headers and transaction details from nearby full nodes as needed. The metadata in the block header allows light clients to validate the authenticity of the received transaction details.

While this architecture is highly secure and widely adopted, it has some significant practical limitations. Since light clients do not download the entire block, they may be tricked into accepting a block with underlying data unavailability. Block producers may include malicious transactions in a block without disclosing all its contents to the network. This is known as the data availability problem and poses a serious threat to light clients. Furthermore, data unavailability is an unattributable fault, making it impossible for us to add a fraud proof structure that allows full nodes to convincingly inform light clients of missing data.

Existing Blockchains vs. Polygon Avail

Contrastingly, Avail takes a different approach to address this issue—it does not validate the application state but instead focuses on ensuring the availability of the transaction data being published and ensuring transaction ordering. Only when the data behind that block is available is a consensual block considered valid. This is to prevent block producers from releasing block headers without releasing the data behind them, which would hinder clients from reading the transactions required to compute their application state.

Avail simplifies the block validation problem to data availability verification, which can be accomplished efficiently at a constant cost using data availability checks. Data availability checks leverage erasure coding, widely used in data redundancy schemes.

Data availability checks require each light client to randomly sample a very small portion of blocks from the chain. A set of light clients can collectively sample the entire blockchain in this manner. A good mental model is a system like a p2p file-sharing system such as Torrent, where different nodes typically only store some parts of a file.

Please note that these techniques will see heavy usage in systems such as Ethereum 2.0 and Celestia (formerly known as LazyLedger).

This has also led to an interesting outcome: the more non-consensus nodes present in the network, the larger block size (and throughput) you can safely have. This is a useful property because it means that non-consensus nodes can also contribute to the network's throughput and security.

KZG-Based Commitments Scheme

In the KZG-based commitment scheme used in Avail, there are three main features:

· Data redundancy making it hard for a block producer to hide any part of the block.

· Fraud-free guarantee of correct erasure code.

· Vector commitments, allowing full nodes to convince light nodes of transaction inclusion with succinct proofs.

In essence, all the data in a block is arranged in a two-dimensional matrix. Data redundancy is introduced by erasure coding each column of the matrix to double the size of the original column. Kate commitments are used to commit to each row, and the commitments are included in the block header. This scheme easily captures attempts at data hiding because any light client that only has access to the block header can query a random cell of the matrix and receive a concise proof that can be checked against the block header (thanks to Kate commitments). Data redundancy forces the block producer to hide most of the block, even if it only wishes to hide a single transaction, making it easily caught by random sampling. The need for fraud proofs is avoided because the binding nature of Kate commitments makes it highly impractical for a block producer to construct an incorrect commitment without being caught computationally. Furthermore, the homomorphic properties of the KZG commitment scheme can compute extended commitments of rows.

KZG Commitment Scheme

While we mentioned the key features of the Avail construct here, there are other features such as partial data availability and collaborative availability guarantees. We have omitted the details here and will revisit them in upcoming articles.

Now might be a good time to give an example and walk through a practical use case. Imagine a new application wanting to host a application-specific standalone chain. It launches a new PoS chain using the Polygon SDK or any other similar framework (such as the Cosmos SDK or Substrate) and embeds its business logic within it. However, it faces the bootstrapping issue of achieving sufficient security through validator staking.

To avoid this situation, it uses Avail for transaction ordering and data availability. Application users submit transactions to the Polygon SDK chain, which are automatically forwarded to Avail and maintained there for ordering. The ordered transactions are picked up by one (or more) operators and built into the final application state based on the business logic. Application users can be confident that ordered data is available and can reconstruct the application state themselves at any time, enabling them to interact with a chain that offers a strong security guarantee provided by Avail.

While the above example discusses a new standalone chain using Avail to ensure security, the platform is generic, and any existing chain can also use it to ensure data availability. In the next section, we will briefly mention how Avail helps existing Ethereum scaling solutions.

Explanation on Ethereum Layer 1 Extension Solution Data Availability

Various Ethereum Layer 2 solutions have been proposed, such as Optimistic Rollup, ZK Rollup, and Validiums. These solutions move execution off-chain while ensuring application validation and data availability on-chain. While off-chain execution architectures increase throughput, they are still bound by the data volume that main chains like Ethereum can handle. This is because even though execution is off-chain, validation or dispute resolution occurs strictly on-chain. Transaction data is submitted as calldata on Ethereum to ensure the data is available for future reconstruction. This is crucial.

In the case of Optimistic Rollup, an operator might submit invalid transactions and then withhold part of the block from the entire blockchain. This way, other nodes in the system wouldn't be able to verify if the submitted assertion is correct. Due to the lack of data, they wouldn't be able to generate any fraud proof/challenge to prove that the assertion is indeed invalid.

In the case of a zkRollup, ZKP robustness ensures that the received transactions are valid. However, even with such assurance, not revealing the data supporting the transactions can have serious side effects.

This can lead to other validators being unable to compute the system's current state, and users being excluded from the system and having their balances frozen as they lack access to the information needed to spend that balance (the witness).

We recognize that to achieve higher throughput, we not only need to move execution off-chain but also require a scalable data availability layer to ensure data availability.

This blockchain design needs to address the following components:

· Data Hosting and Ordering: This part will receive transaction data and order it without any execution. It will then store the data and ensure full data availability in a decentralized manner. This is crucial for Avail.

· Execution: The execution component should take ordered transactions from Avail and execute them. It should create a checkpoint/assertion/proof and submit it to the data validation layer. We refer to this as the execution layer.

· Validation/Dispute Resolution: This part represents the main chain the system is anchored to. The security of the design relies on the robustness and security properties of this part. Checkpoints/assertions/proofs submitted by the execution layer are handled by this layer to ensure only valid state transitions are accepted in the system (assuming data availability). We refer to this part as the data validation layer.