Original Title: "Using The Graph to Preserve Historical Data and Enable EIP-4444"

Original Authors: Seve 2, Eray 4, Lichu 3 from Semiotic Labs 2, and Tatu 3 from The Graph core development team

Translation by: Luccy, BlockBeats

Editor's note:

The EIP-4444 proposal sets HISTORY_PRUNE_EPOCHS to 82125 epochs (i.e. 1 year on the beacon chain) to prevent execution layer clients in PoS Ethereum from providing data for block headers, block bodies, and receipts that are more than a year old on the p2p network. Clients can locally prune this historical data.

Related reading: "Introducing EIP-4444: Limiting Historical Data for Execution Layer Clients"

The Graph is an indexing protocol that allows for querying on networks such as Ethereum and IPFS. Anyone can build and publish open APIs called subgraphs to make data more accessible, with the core component being indexers.

Indexers need access to historical data on the blockchain to perform their tasks, and Firehose is an open-source tool for extracting data from the blockchain. Currently, blockchain data is extracted from Firehose-enabled full nodes by performing a complete archive sync, but after EIP-4444 is implemented, The Graph's indexers will need to adjust their blockchain data extraction process.

In this article, Semiotic Labs shares how they used technology development on The Graph protocol to solve problems for a wider community. Initially, Semiotic Labs developed Firehose and Substreams to improve the performance and verifiability of The Graph protocol, and these tools and technologies happen to address the EIP-4444 issue. Based on this, a new indexing pipeline using this technology was created specifically for indexers.

In this pipeline, Firehose first extracts data from the blockchain and stores it in flat files, providing a gRPC interface for streaming blocks, which can be consumed by multiple endpoints in The Graph. The flat files, as shareable Ethereum archives, contain all necessary historical data. These files are shareable and verifiable.

In addition, this article compares the subtle differences and use cases between the flat file format and the era1/era format structure, and points out that in The Graph, the new indexer must perform a complete archive node sync when running Firehose to extract the required historical blockchain data for indexing. It also explains how to solve these problems through shared, verifiable flat files and The Graph's incentive market.

Semiotic Labs is one of the core developers of The Graph protocol. The author's team is dedicated to developing and implementing mechanisms to verify data from various indexed blockchains provided by The Graph. The work described in this article is the result of Semiotic Labs' continuous efforts. Severiano Sisneros published this article in the Ethereum Research Forum, and BlockBeats compiled the original article as follows:

The Graph Introduction

The Graph is a decentralized blockchain data indexing and querying protocol. The indexer is the core of The Graph, indexing the history of a specific event on the blockchain and making the data available for consumers to query. Indexers require access to historical blockchain data to complete their work. In the future, The Graph's indexers will use specialized tooling as part of the Firehose data extraction process to extract data from the blockchain.

Firehose is an open-source tool created by StreamingFast, the core developers of The Graph. Firehose provides a flat file and streaming-first approach for processing blockchain data. Semiotic Labs is another core development team of The Graph, focusing on research and development in the fields of artificial intelligence and cryptography.

Currently, blockchain data is extracted from Firehose-enabled full nodes by executing complete archival synchronization, which relies on peers in the blockchain's P2P network providing complete historical data. However, after the implementation of EIP-4444, full nodes will no longer provide historical data beyond one year. The Graph's indexer will need to adjust its blockchain data extraction process to ensure continued access to the required data.

This article will introduce how The Graph indexer uses Firehose to extract data from the blockchain and store it in Google protobuf serialized object storage called flat files. When Firehose synchronizes from the genesis block, these flat files store all the information needed for the complete history of stored data in the blockchain (specifically, block bodies and receipts). We recommend using these flat files as an archival data format, similar to era and era1 files, and demonstrate how to verify that the data stored in the flat files belongs to the authoritative blockchain.

For The Graph, once EIP-4444 is implemented, indexers can share these flat files to obtain the necessary historical data for indexing and querying. In the case of Ethereum, The Graph's incentivized indexer network collectively ensures the preservation and availability of historical data, providing a verifiable, decentralized, and independent solution for EIP-4444.

Interestingly, our initial goal was not to design a solution for EIP-4444. The tools and techniques discussed here were developed to improve the performance and verifiability of The Graph protocol. However, in the process, we unexpectedly discovered a solution for EIP-4444. Our vision is for this solution to coexist with other EIP-4444 solutions, much like the various implementations of Ethereum execution clients, without causing heterogeneous distribution of block data.

The Graph Firehose Pipeline Overview

The Graph is a constantly evolving dynamic protocol designed to meet the demand for fast access to blockchain data. To this end, new technologies called Firehose and Substreams have been developed, and a new indexing pipeline using these technologies has been created specifically for indexers.

In this pipeline, Firehose is the first processing step. It extracts data from the blockchain and stores it in flat files (which will be detailed in the next section). Then, Firehose provides a gRPC interface for streaming the blocks stored in the flat files. In The Graph, this stream can be consumed by multiple endpoints. For example, the Substreams service can filter specific events of interest and perform any specified transformations on the filtered data. The processed results will be routed to an endpoint aggregator and provided to consumers.

Figure 1 provides a pipeline view of Firehose and Substreams, illustrating how data is extracted from the blockchain, processed, and made available to consumers through various endpoints.

图 1：Firehose + Substreams

Figure 1: Firehose + Substreams

Against the background of EIP-4444, we are particularly concerned about how to extract and store blockchain data in flat files. We observe that flat files contain all the data necessary to retain blockchain history, and these data can be verified as truly included in the canonical history of the blockchain through simple technical means. As a decentralized blockchain data indexing and querying protocol, The Graph has a natural advantage in ensuring the availability of the entire blockchain history.

Plain files as shareable Ethereum archives

Firehose extracts data from the blockchain by using a dedicated full node 2. Ethereum Firehose uses a streamlined version of the official Geth 1 implementation, and it should be noted that Firehose supports multiple chains, including non-EVM chains, and the full node used will vary depending on the chain. Specifically, when Firehose performs a full archival sync, it reads data from the tracking API of the full node. Please refer to Figure 2 for a detailed description of these steps.

Figure 2: Extracting Firehose Data

The output content of the tracking API is stored in a flat file. It should be noted that the tracking API can be configured as needed to provide different levels of EVM execution introspection. For example, it can be configured as a callTracer to trace the call frames executed during the transaction process (including top-level transaction calls and calls within transactions), or the tracking API can be configured as a structure/opcode recorder to output opcodes and execution contexts at each step of transaction execution. In order to verify the feasibility of Firehose flat files as Ethereum historical archives, we focus on preserving top-level transaction calls and receipts, as these are the data submitted to the blockchain and can be verified on the blockchain. In this case, Firehose stores all transaction data (block bodies) and receipts related to transaction execution in the blockchain.

Although Firehose flat files were not specifically designed for exporting historical Ethereum archive data, they contain all the necessary historical data. These files are shareable and, as we have seen, verifiable, making them a strong candidate for archival formats.

Verifiable Flat Files

The flat file stores all top-level transaction data (block body) and receipts related to transaction execution. For example, as shown in Figure 3, the flat file of 15 million blocks decoded using Semiotic's flat-files-decoder (in JSON format) displays the stored block header information and the transaction trace of top-level transactions, such as the transaction 0x7ecdee...e3222 shown below. Ethereum blockchain data extracted through Firehose is stored in this pattern in the flat file.

Figure 3: Decoded blockchain data of block 15,000,000 from a flat file

The flat file stores all the necessary information for reconstructing transaction and receipt Merkle trees. These Merkle trees are submitted to each block of the canonical blockchain. To verify that this data belongs to the canonical blockchain, we only need to ensure that the reconstructed Merkle tree root matches the root stored in the canonical block header. After merging, we can leverage the information provided by the consensus layer. For example, Ethereum provides a sync committee sub-protocol for verifying block headers from the canonical blockchain. To verify that the data stored in the merged flat file is correct, we only need to reconstruct the receipt and transaction Merkle trees for the blocks in the flat file, and then compare these roots with the block headers that have been validly signed by the sync committee. We can obtain valid block headers from the consensus light client.

For examples on how to implement this operation, please refer to our proof-of-concept code. According to our benchmark tests, extracting data from a block and constructing a Merkle tree on an M1 Mac takes approximately 5 milliseconds.

Before merging, we need to complete more work. Fortunately, we are not alone. The Ethereum community has been seeking solutions to this problem. Specifically, the portal network uses Header Accumulator to verify whether the block header is included in the canonical chain. It is worth noting that these accumulators are agnostic to the storage format of archival data, so we can stay consistent with this approach and calculate the same Header Accumulator value from data stored in flat files. When indexers share flat files containing pre-merge blocks, they also need to provide proof for block headers that include the Merkle tree root of receipts and transactions.

Flat files, Era files

The archive data format and validation mechanism proposed for The Graph's Firehose flat files are compatible with existing data format methods, such as the era/era1 data format described here. One key difference is which blockchain data is stored. Specifically, the era file format requires each era file to contain a snapshot of the Beacon chain state, in order to obtain state data when processing the chain from that file. However, Firehose flat files intentionally do not include any Beacon chain state data, as their purpose is solely to centrally store historical transaction data and receipts from the execution layer.

Solution 1: If the flat file is corrupted during the synchronization process (as we experienced two months later), Firehose synchronization can be completed by downloading flat files from other indexers and bootstrapping Firehose to use them. Using the above tools, the indexer can verify that the flat files downloaded from peers belong to the canonical blockchain. The indexer will provide seeds for Firehose using the flat files downloaded from peers and verified against the canonical blockchain, rather than performing a full archive sync. The bottleneck becomes the download speed, for example, downloading the entire archive from the genesis block of about 2TB takes a few days instead of several months required for a full archive sync.

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia