Partnered With:

Filecoin and Arweave: From Storage to Computing, the revival of decentralized storage

Read this article in 24 Minutes

There are many Arweav data storage protocols, the most notable of which are Filecoin and Arweave.

原文标题：《 Filecoin 与 Arweave：从存储到计算，去中心化存储复兴》

原文作者： Leo, IOSG Ventures

Summary:

The Arweave protocol is designed to technically enable permanent storage and is ideal for high-value digital assets such as NFT metadata.

Data storage, more need to calculate. With the introduction of smart contracts and programmability, the development of decentralized data storage networks has entered a new phase of "not just storage".

Filecoin uses financial incentives to store data redundancies, and Arweave uses protocol design.

FVM brings storage finance to Filecoin, commoditizing storage space and time, allowing users to lock in costs ahead of time, and storage vendors to cash in ahead of time and manage inventory, hardware, and operations based on future needs.

The mainstream of computer network development is nothing more than data calculation, transmission, storage three directions.Today, the development of decentralized data storage protocols is obvious.

Filecoin officially released the EVM-compatible Filecoin Virtual Machine (FVM) on Mainnet on March 14, 2023 at the height of 2,683,348 Epoch, bringing smart contracts and programmability to the Filecoin network, The development of decentralized data storage protocols has entered a new stage of "not just storage".

There are many decentralized data storage protocols, but the most notable ones are Filecoin and Arweave. We discuss each of the new features that have been released to Filecoin for FVM in this article.

Permanent storage Perpetual Storage

Permanent storage has special significance and needs in Web3. High value digital assets such as NFT Metadata need to be stored permanently.

Filecoin

With the release of FVM, Filecoin emphasized Permanent Storage. We understand Filecoin permanent storage as an economical way to theoretically enable long-term storage, but there are not many changes at the level of the Filecoin protocol design.

In Filecoin's current design, storage orders are recorded in a chain match between storage providers and storage customers. The order contains information such as data size, storage time, order price, and collateral. If the stored data is to continue to be saved after the storage time defined in the order expires, the storage demander has to manually submit the update order.

However, after FVM is published, orders can be automatically renewed on the chain.

Lighthouse

Lighthouse is a project to enable permanent file storage on Filecoin, where you pay once and your files are "forever." Lighthouse offers a smart contract-based endowment pool to continuously pay for file storage. When a user creates an order and pays, a portion of the amount goes to the storage provider and the rest goes to the donation pool. The donation-pool smart contract will be automatically renewed at the end of the order and paid for by the donation pool, thus enabling "permanent storage". The feasibility of this design is that the donor pool will increase the value of the assets in the pool through pledging, farming, etc., which over time will more than cover the cost of storage.

This is similar to the assumption in the Arweave Yellow Book about storage costs, which is that storage rates will continue to decline and that the incremental value paid by storage consumers will be enough to cover the cost of permanent storage.

Arweave

"Over the past 50 years, storage rates have declined by an average annual rate of 30.57%".

Arweave Yellow Paper: Cost per 1GB of Data Held for 1 Hour Since 1980 (log scale)

Arweave introduces a common blockchain data structure called Blockweave, which essentially allows data to be stored permanently through a protocol design.

In Blockware, each block in the chain (with the exception of the newly confirmed block and the candidate block being mined) is linked to three other blocks: the preorder block, the post-order block, and the recall block.

For blocks of a certain height, the recall block can be any historical block before that height block in Blockweave. When a miner digs a new block, the selection of the recall block is randomly determined by the presequencing block height and hash.

Recall blocks play an important role in Arweave's consensus mechanism, the Succinct Proof of Random Access -- SPoRA.

In Arweave, miners do not have to save all historical blocks (that is, it is not necessary to save all historical blocks), but miners' local saving of randomly selected recall blocks is a ticket for miners to participate in candidate new blocks. The recall block function is similar to randomly checking whether miners have saved the content of a historical block, so as to realize the permanent storage of historical data.

Arweav e's permanent storage is technically guaranteed according to the protocol design and is a more robust solution than Filecoin's permanent storage. That's why Web2 giants Meta, Instagram and Mirror choose Arweave as their NFT and content storage solution.

Decentralized Computation

The data is saved and, more importantly, used. Filecoin and Arweave's vision goes far beyond "decentralized web disks," which is where most storage users currently use them, to a blockchain protocol that combines low-cost storage and high-throughput computing.

On top of the data storage, Web3 Dapps also need to compute.

Filecoin

Filecoin and IPFS distribute content-addressed data sets among storage vendors around the world to increase data redundancy and resilience. This decentralized distribution of data brings cost, availability, and reliability advantages, but comes with the problem of parts of a single data set being stored at geographically distant storage providers. A widely distributed data set is not conducive to performing computations or querying indexes on top of the data. But regrouping highly dispersed data into a single center for computation is expensive, wasteful, underperforming, and goes against the principle of decentralized storage.

The EVM-compatible FVM on Filecoin has come up with edge computation combined with on-chain coordinated computation.

Contracts in FVM can broker computing resources, incentivize computing execution, distribute workloads among available storage vendors, and prove the validity of computing results to earn rewards.

Storage vendors can participate in decentralized computing networks through FVM contract registration. The computing client publishes the computing task to the contract. The contract defines a mechanism that assigns computing tasks to storage providers registered to participate in the computing network, and when the computation is complete, the storage provider will issue certificates for payment.

Arweave

Decentralized computing on Arweave is enabled through the SmartWeave smart contract protocol, with the ability to directly process rich data. The main difference between SmartWeave and other blockchain smart contract protocols is "Lazy Evaluation," which shifts the burden of smart contract execution calculations from network nodes to smart contract users. The benefits of lazy computing are obvious. By decoupling storage and computing, nodes do not need to keep growing global state.

The smart contract can be calculated and verified by the user only when it is used, rather than requiring every node involved in the consensus on the chain to calculate and verify. By delegating computing to users, blockchain scalability is also enhanced.

Warp

Warp has developed a Warp SDK based on the initial version of SmartWeave, which improves performance and modularity over the native version and allows for different execution environments.

Warp recently released its 2023 roadmap, which includes the following goals:

1) Layer1 synchronizer: realize efficient synchronization between Warp contract and the underlying Arweave layer;

2) The Layer2 collator, instead of sending data directly to the main Arweave network (which may wait 2-3 minutes for the data to be packaged into the next mined block), the data transaction is directed to the Warp collator and cleared immediately over the Bundlr network. Provide users with immediate access to data and near-instant final certainty;

3) contract enhancements, Warp contract is designed to provide Web3 Dapps with a fully featured technology stack to compete with Web2 services;

4) Develop Delegated Resolution Environment and aggregate nodes. The delegated resolution environment allows delegated delegated to highly interactive and/or insecure contracts, and the aggregate node provides delegated contract status monitoring and insight.

Storage redundancy Storage Redundancy

Decentralized data storage network avoids single point of failure, but how to ensure that each node/storage provider truly and effectively saves the uploaded data of storage users? And have multiple nodes/storage vendors store uploaded data separately to achieve storage redundancy and reliability?

Filecoin and Arweave take different approaches, Filecoin using financial incentives and Arweave using protocol design.

Filecoin

The Replication Worker was introduced in the FVM release highlights, Repair Worker (data repair worker)

Prior to the release of FVM, if a storage demander wanted to backup their data in a network node to maximize the chance that the data would be preserved in the event of a storage provider failure, the demander had to rigorously match N orders down the chain with the supplier, perform N on-chain transactions, and transfer data N times with intensive resources.

After the release of FVM, data replicators will act as intermediaries for data redundancy for a small fee, saving time and overhead for storage users. The replicator automatically matches and generates storage orders across the Filecoin network based on the number of backups selected by the demander, storage geographic area, delay requirements, price range, and so on. The data maintenance worker can act as the agent of the demander, monitor whether the stored data is lost or expired, and automatically copy and back up the data below the redundancy threshold to more storage suppliers according to the Settings of the demander, and update expired or terminated storage orders on behalf of the demander.

Arweave

Arweave's storage redundancy is achieved naturally through protocol design. Arweave uses the recall blocks as part of the input of the Random Access Concise Proof of Work (SPoRA) algorithm to ensure that miners who dig new blocks actually have all the data on the recalled blocks. The SPoRA Consensus mechanism encourages miners to save as much historical block and Blockweave data as possible within their storage space capabilities. However, if the storage capacity of miners is not enough to save all the historical blocks and complete Blockweave data, miners will preferentially save blocks that have been less saved by other miners, because when a recall block that has been saved by more miners is selected, more miners will compete for a new block at the same time; When a recall block is selected that is saved by fewer miners, there is less competition.

In addition, the selection of recall blocks is highly random, and the probability of any recall block being selected is discrete and evenly distributed. Therefore, in the case of limited storage capacity, rational miners should give priority to save the blocks that have been less saved so as to improve the probability of digging new blocks to obtain block rewards. Arweave's agreement ensures the reliability and data redundancy of Arweave's centralised storage network through smart design and financial incentives that allow blockweave and all historical blocks to be backed up to the maximum extent possible within the storage capacity of all miners across the network.

Data Retrieval Data Retrieval

When the data is saved, how to retrieve the data efficiently, accurately and quickly is another problem.

The data retrieval service in Filecoin is a separate financial incentive system. Retrieval Providers are responsible for providing quick access to storage data for those who need it. Retrieval providers focus on quick access to data rather than long-term storage. Most storage providers are also retrieval providers. The demander pays the retrieval provider for the data. Projects such as retrieval.market and the Saturn Network already exist in the Filecoin ecosystem for fast data retrieval and content distribution.

In addition to the aforementioned benefits of permanent storage and storage redundancy, Arweave's SPoRA consensus mechanism also improves data retrieval and access speed with one stone. Before the SPoRA upgrade, Arweave's old proof-of-acces-POa consensus mechanism solved the problem of how to incentivize miners to store as much data as possible, but did not incentivize miners to retrieve stored data quickly.

In fact, during the PoA period, miners pooled storage resources to create a storage pool, which held historical blocks. When a recall block is selected, the storage pool sends the contents of the recall block to the miner when requested by the miner. This is not conducive to network decentralization. The Arweave network statistics show that the number of nodes decreased while the hashing power of the entire network increased, indirectly proving the existence of storage pools. To solve this problem, and to encourage miners to store data locally, Arweave upgraded PoA to SPoRA. After the SPoRA upgrade, miners who choose not to store history block data locally request and transfer recall blocks to the storage pool (requiring extremely many times), which greatly increases the cost and time of transferring data, while miners who store history block data locally have a higher chance of mining new areas faster. The mechanism design eliminates the existence of storage pool. Miners nodes all over the world store historical block data locally, which also improves the retrieval and access speed of data storage users.

Storage Finance Financialization

With the release of FVM, many Web3 applications including DeFi can be introduced on Filecoin, such as pledge protocols, insurance protocols, stored derivatives, etc.

Storage providers in Filecoin need to pledge a certain amount of FIL as collateral to provide storage services. In the past, storage providers either raised money to purchase FIL or relied on an off-chain loan contract to borrow FIL. However, with the pledge agreement built on FVM, FIL token holders can deposit idle FIL into the agreement and set rules and terms under which storage providers of any size can obtain FIL on the chain to raise sufficient amount of FIL as collateral for storage services.

Storage derivatives are another interesting application scenario, where dynamic storage overhead poses budgetary challenges for both storage users and storage providers. By commoditizing storage space and time, storage demanders can lock in storage costs early and storage providers can capitalize early and plan to manage inventory, hardware, operations and finances based on long-term needs.

Project positioning and status quo

Filecoin currently has 3,678 nodes providing 19.544 eiBs of storage space, while Arweave has 112 nodes storing 125.62 tiBs of data.

Filecoin networks are larger in terms of scale, but Filecoin and Arweave are both decentralized data storage protocols, but they are positioned differently and cannot be compared simply by the number of network nodes or size.

Protocol Labs positioned Filecoin as Storage Marketplace and Incentive layer, and built a complete storage market, retrieval market and financial products around Filecoin. It aims to become the largest and most important decentralized data storage, distribution, and computing protocol through economic incentive design to achieve rich product functions (such as permanent storage, storage replication and repair, etc.).

Arweave's most important position is to store data permanently, and to develop a smart contract protocol based on Arweave to compute data. All of the mechanics are designed to serve this most important goal, and from the introduction of the previous features, it's easy to understand that Arweave is designed to be elegant and unified.

prospect

Compared to the Ethereum ecosystem and the Ethereum virtual machine, the development of decentralized data storage networks has been somewhat lackluster over the past few years.

There are many good projects and entrepreneurs in the Filecoin and Arweave ecosystems, but Filecoin and/or Arweave aren't being adopted on a large scale by Web3 Dapps, and many Dapps still rely on Web2. Computing on a blockchain that solves storage is a novel approach, and both FVM and SmartWeave have the potential to unlock unprecedented decentralized applications.

As a developer or user, it's not a matter of choosing which decentralized storage protocol to use, but one that should be based on the storage needs of your application and content.

Filecoin and Arweave have overlapping positions, but they are able to evolve in their respective areas to meet the evolving need for decentralized network storage, moving from a "decentralized web disk" to a decentralized server.