Partnered With:

Taihe observes the status quo and challenges of | distributed cloud storage racetrack

Read this article in 48 Minutes

This paper mainly introduces the typical application projects of blockchain in the field of distributed storage, as well as the conditions for successful de-centralization storage, the research status and challenges of relevant technologies

原文标题：《太和观察｜对于分布式云存储赛道的研究和思考》

原文来源： Calvin Zhang Ti Capital

The following parts are mainly explained by this article: Problem solved by distributed storage; The meaning of Distributed Storage; Typical applications of distributed storage in the field of BlockChain; Factors affect the success of Distributed Storage; The analysis of distributed storage related technical research and challenges.

Background to the rise of distributed storage technology & NBSP;

Web3.0 is the natural evolution of the contemporary Internet towards the next generation. The next generation of Internet must meet the need for machine intelligence and connectivity of everything. When people start to look forward to the new Web3.0 era, they will find that the contemporary data storage system (centralized cloud storage supplemented by terminal test storage) deviates from the requirements of Web3.0, and this is exactly the great opportunity of decentralized storage, which is bound to become the necessary support of the Web3.0 era.

According to IDC, by 2022, digital-driven economic output will account for 60 percent of global GDP, while China's digital economic output will account for 65 percent more than the global average. With the acceleration of the digitization process and the promotion of intelligent targets, the data generated by enterprises will be continuously monitored, showing a massive and diversified development trend. Multi-cloud and cloud-side-end deployment environments are more complex, and enterprises have higher and higher requirements on real-time and reliability of data. The current centralized data storage mode urgently needs to be reformed to adapt to the new Internet data structure and the ever-expanding demand.

Decentralized storage the biggest significance is to let a person with the machine, a large number of marginalized as a result of the interaction between machine and machine to store data in a distributed storage agreement, using block chain add trust writing layer, let each individual can participate in the governance of networks, each individual can participate in the distribution according to work in the network, can get incentives, it will be block chain technology give full play to the important field of utility.

This paper focuses on:

Problems encountered in centralized storage;

The significance of the existence of distributed storage;

The typical application projects of blockchain in distributed storage are listed.

The conditions for successful decentralized storage are analyzed.

The research status and challenges related to distributed storage are analyzed.

01. Problems and challenges of centralized storage

Our society is experiencing an unprecedented era of information explosion, computer, intelligent equipment, television, home security systems, wearable devices, cars, and even the robot all the time to produce and use the data, and these data are growing exponentially, coming of AI and Internet of things (IoT) times have been challenging the border of the current data storage.

According to the research of Gartner, by the end of this year, there will be more than 20 billion connected devices in the world. The huge group of devices will constantly generate massive data, which puts forward higher requirements for the current data storage system, including data storage, management and retrieval.

The current centralized cloud storage is the storage resources in the cloud for people to access a storage solution, and this kind of centralized cloud storage way more data centralized, related to the amount of data is bigger, huge amounts of data centralization storage highly vulnerable to attacks, for data privacy, security, and sustainability are the unprecedented huge risks.

1.1 Privacy and data ownership issues & NBSP;

For the current centralized storage, users' sensitive data are all uploaded, which not only makes users lose control of their data, but also puts the risk of data leakage on the side of cloud storage operators. There are two types of data is extremely sensitive to this problem: the first is their own sensitive information, such as enterprise strategic planning, financial information, investment and financing decisions, marketing strategy, customer data, business analysis, etc., this information is not known by the public, can bring economic benefits for the enterprise, is practical, including business information and technical information; The second type is the user's personal information, with the rapid development of the centralized cloud storage, production system has accumulated a large number of business contain the user name, identification number, address, phone number, account sensitive information such as data, if these data leaking, damaged, will not only to the leak itself bring certain problems, data centralization of centralized storage and even cause instability group events, which are not conducive to the current rapid development of the development and the arrival of the era of large data.

Growth rate of data leakage, from: Identity Theft Resource Center

1.2 Stability of data & NBSP;

The security of data includes two meanings. One is "to ensure the integrity of data". Another layer is "data privacy protection". Cloud storage market is very competitive now, due to the increasing of users, in order to ensure a good user experience, service costs rise, temporarily no good profit, so in recent years, news services run or stop the service, the user cannot have any constraints, and to the behavior of the service behavior of the claim. As a result, users tend to store their data in larger and more reliable service providers with increasingly high degree of data centralization, which also leads to widespread loss of data once lost.

Real-time data in the cloud keeps going high, source: IDC ;

1.3 Sustainability of data storage

Under the framework of the Web3.0 in the future, a lot of smart devices connected to the network, and can produce huge amounts of data in real time, the growth of the amount of data will be the index level, in this case the centralized data storage obviously can't satisfy the demand of the network storage, this point in the future unmanned and Internet of things (IoT) appeared in the field of will be particularly apparent, future data storage system is not only to do data storage, sharing, read, must be efficient and accurate data transmission and analysis, the centralized data storage structure put forward a great challenge.

These are the dilemmas faced by centralized cloud storage service providers today.

Therefore, the development of storage technology even today still faces great challenges. These problems are closely related to human factors and centralized operations and management. To solve these problems thoroughly, we must start from the perspective of decentralization. So the industry has turned its attention to blockchain technology. The decentralized storage scheme based on block chain technology emerges as The Times require.

02. The meaning of decentralized storage

The challenge of centralized storage is the opportunity of decentralized storage.

Decentralized storage will combine the best functions of blockchain technology to meet the needs of massive data storage. As the name implies, decentralized storage distributes data across multiple network nodes, similar to the distributed ledger of blockchain.

At present, single system and cloud storage are highly centralized, which makes them relatively vulnerable to hackers. Even leaving aside hacking, there are many obvious disadvantages, such as power outages. Decentralized storage does not encounter these problems as opposed to centralized storage. Because it makes use of geographically distributed regional or global nodes.

As we rethink applying blockchain thinking and structure to decentralized storage solutions, we open up a better path for today's data storage.

2.1 & have spent Adapt to unstructured data, edge data storage

With the proliferation of images and videos, the concept of Unstructured Data is everywhere. Many people simply understand that the content stored in the traditional relational database is structured data, while the image, audio, video, document and other data stored in the form of ordinary files is unstructured data. According to IDC, 75% of future data growth will be in unstructured marginal data.

Different from centralized cloud storage service, distributed storage can move data storage from remote cloud server to edge storage device or edge data center, which has lower network communication overhead, interaction delay and bandwidth cost, and higher adaptive capacity and scalability.

This storage mode is also more suitable for current and future data storage structures.

2.2 Ownership of the data return the data producer

With the development of the Internet, each of us leaving marks on the Internet, to produce a large amount of data, when people are no longer able to control their own data, such as data are collected, how to use and problems arise, especially in some cases, the user is not knowledge, will be a few enterprises to profit user data, or require the user to provide beyond the requirements of the more personal information.

Blockchain technology has been successful in the field of cryptocurrency, and proved that it is possible to achieve a democratic way of data management and return ownership to the hands of users through its security, efficiency and non-centralized control, which is a distributed storage technology based on blockchain technology.

2.3 It conforms to the sustainable development trend of data big explosion

We have previously analyzed that the modern society has entered the era of big data explosion, and massive data is expanding in an exponential way. With the maturity of the Internet of Things, artificial intelligence, cloud computing and other technologies, this trend will be further strengthened.

Big data is a massive, high-growth, diversified information asset that requires new processing models to gain greater decision-making, insight, and process optimization capabilities. Big data needs to deal with storage with large quantity and fast storage speed, which requires that the underlying hardware architecture and file system can flexibly expand storage capacity.

Distributed storage, which is essentially a decentralized distributed ledger, is more suitable for such a rapidly expanding data structure. Block chain technology, as an increasingly linked data structure, participates in the calculation and recording of data through multiple nodes in the network, and verifies the validity of its information. From this perspective, blockchain technology is also a specific database technology. Due to the security and convenience of decentralized database, many people in the industry are optimistic about its development, considering it as an upgrade and supplement of existing Internet technology.

The storage form of distributed storage is more in line with the development trajectory of data, source: Zhongxing Research Journal

The data structure of the next generation Internet must be dominated by unstructured data and marginalized data, under which only decentralized storage protocol can be selected.

Decentralized storage is obviously an important support for the next generation of Internet. Only with decentralized storage protocol can we truly realize the "cloud Internet" in the future. In the future, all apps and applications and even games will be built on the distributed cloud network, and end-test devices will only collect data, no longer assume the role of computing and storage. Local storage will continue to weaken, and peripheral decentralized storage will gradually replace end-measured data storage, which can truly constitute the future of decentralized Internet in my imagination.

When people dream up Web3.0, they can either remember everyone's data or forget it. It is a secure Internet that cannot be attacked, and the ownership of data can belong to the users who generated it. Under the distributed storage protocol, it could be either an online mall or a public speech square, giving back the entire Internet to all Internet users rather than being controlled by a few Internet giants.

03. A brief analysis of decentralized storage project cases

3.1 & have spent Representative item

Many of the decentralized storage projects that have been exposed so far, both in terms of technology and implementation, have overlapped with each other, each with its own characteristics, without a very clear division.

To roughly divided as implementation logic, BitTorrent, Filecoin, Arweave, Crust can be represented by the project is divided into based on content addressing the file sharing network layer, and represented by Sia, Storj, MaidSafe project is more favored by encouraging card share their hard drive space, including whether you can build in their own independent chain for secondary breakdown, such as Filecoin, Arweave is based on their own chain, And Crust is based on the Polkadot common chain.

BitTorrent was the earliest decentralized storage project, but it lacked a profitable business model until Tron acquired it in 2019 and launched BTT-inspired tokens. But it now appears that THE TOn-10 based BTTS will ultimately be more named than real. Filecoin is the largest of the decentralized projects, with a cumulative financing amount of $257 million. Filecoin encourages point-to-point storage through block rewards, fees and service fees, and matches the supply and demand of storage through an optimized Kademelia algorithm. Boca Also fits with various storage protocols including IPFS; Storj and Sia prefer to provide electronic network disk, which does not require actual storage. As long as storage space is provided, mining can be done, which has no actual use value. Sia provides a space for negotiation and communication between different nodes in a P2P storage network by means of built-in smart contracts.

Here is a summary and comparison of existing distributed storage projects:

3.2. BitTorrent -- a prototype for a decentralized storage route

BitTorrent, or BT, is an open source content distribution protocol that Mr Bram Cohen developed himself in 2003. It USES efficient software distribution systems and peer-to-peer technologies to share large files (such as a movie or TV show) and allows each user to upload them as a network reallocation node. Common applications include BitTorrent, Torrent, etc.

BitTorrent works in the same way as the general decentralized storage protocol, slicingfiles, and then dividing each Piece into blocks of 64-512KB in size. Each block generates a hash string, which is then encrypted and distributed to each node in the sha-1 Shared network (Swarm) of the file or filesets. Peer nodes with full files are seed nodes, others are non-seed nodes. Then, transfer the file sharing all the nodes in the network resources, make more nodes seed node, and through Opitimistic Unchoked algorithm to select the two Shared network node, to expand the Shared network.

Although BitTorrent is the earliest decentralized storage project, it lacks a complete incentive mechanism, so it can only be called the prototype of decentralized storage mode. BTT tokens based on TRON wavelength only borrowed the story of BitTorrent, but did not really enable BitTorrent.

3.3 Filecoin (IPFS official Protocol Incentive Layer) -- the leading project of decentralized storage project

Filecoin is an incentive mechanism and a public chain System based on the IPFS(InterPlanetary File System) protocol. The IPFS protocol defines how files can be stored, retrieved and transmitted in distributed systems. It can save and share files permanently and decentrally, which is a content addressable, point-to-point distributed protocol.

IPFS wants to build on BitTorrent to create a truly peer-to-peer, decentralized file storage system. In IPFS, all files are centralized, there is a common language, and all users are Shared across the system, allowing them to find and transfer files to each other.

Filecoin is an incentive token created by IPFS Official Protocol Lab to encourage the Filecoin network to store and retrieve market roles. The technical difficulties of Filecoin are data holding proof, prevention of cheating and attack, and zero-knowledge proof.

IPFS contains several innovations in communication protocols and distributed systems, and the combination of them produces a different file system. Therefore, to understand the breadth and depth of what IPFS is trying to achieve, it is important to understand the technological breakthroughs that make it possible and all the problems it is trying to solve.

More than 100 scenarios have been applied to IPFS. Not only jingdong and huawei layout IPFS, but also Microsoft, Google and firefox have joined IPFS application, which shows the rapid development of IPFS from this aspect. In the future, all kinds of content that users want to store, whether text, pictures, or video, will be possible via IPFS.

3.4 Crust -- IPFS compatible distributed storage project based on the Boca Ecosystem

Crust Network is an incentive layer for an applied public chain of decentralized cloud services, similar to Filecoin as a distributed storage incentive layer for IPFS, and for ease of understanding Crust can be thought of in part as a Filecoin on a Boca Network. Crust is an incentive layer protocol based on a boka parallel chain architecture for distributed clouds, with multiple storage protocols, including IPFS, and the current phase is focused on addressing storage issues similar to Filecoin's vision.

'Crust' as a distributed storage track has been a focus of Boca's efforts from the encryption community, investment organizations and Boca Ecology. As a cornerstone of Web3.0, the development of Crust was highly concerned by the management levels of The Parity and Web3 Foundation, which was not only a service partner of Substrate Builders Program and Web3 Foundation Grant, but also a core member of The Web3.0 Bootcamp established by the Wanxiangblockchain Web3 Foundation.

From the Poca Ecosystem, Crust is distinct from other distributed storage projects in that it pioneered the hardware solution known as TEE (trusted Execution Environment) technology as the core solution for quantifying and verifying actual node workloads within local CPU hardware. Its advantages are fast speed, mature technology and high versatility. It is one of the mainstream wireless communication chips in the Internet of Things, but it relies more on hardware manufacturers for security.

Based on TEE's Meaningful Proof of Work environment, Crust came up with an MPoW (Meaningful Proof of Work) mechanism to measure the storage capacity of the nodes and report it back to the chain. Meanwhile, Crust created a PoS consensus algorithm based on the defined amount of resources. It is called GPoS (Guaranteed Proof of Stake) workload report, which is recorded along with other transactions and packaged into the block to calculate a Staking amount, and then make the PoS consensus based on that amount.

3.5 Storj -- Distributed Cloud storage project based on Ethereum

Storj is an Etherean-based distributed cloud storage protocol developed by for-profit company Stroj Labs. Storj's core technology is an executable, point-to-point storage contract in which two people (or computers) agree to use a fixed amount of storage to reap the benefits without knowing each other.

Storj's core technology is executable, point-to-point storage contracts. It is a way in which two people (or computers) agree to exchange money for a certain amount of storage space without knowing each other. We call the computer that sells space "Farmer" and the computer that buys space "renter." Renter meets with Farmer to negotiate a contract to move the data from Renter to Farmer for safekeeping.

The for-profit side of Storj Labs is that it rents out its network to thousands of users and charges them to use it. It's a slightly more focused model to compete with Dropbox and Google Drive. They have also formed a partnership with Microsoft Azure to deploy some of their development tools.

3.6 & have spent Swarm -- a storage service over the Ethereum network

Swarm is also based on Ethereum by providing distributed storage platforms and content distribution services where participants can effectively pool storage and bandwidth resources in order to provide services to all participants in the network and in return receive a portion of their Ethereum rewards.

Swarm is a distributed storage platform and content distribution service and a local base layer service for the Ethereum Web3 stack. Swarm's primary goal is to provide public records for Ethereum with sufficient decentralized and redundant storage, in particular the storage and distribution of DApp's code and data, as well as blockchain data. From an economic perspective, it allows participants to effectively pool their storage capacity and bandwidth resources to provide these services to all participants in the network, while being motivated by Ethereum.

From a terminal point of view, Swarm is not much different from the Internet except that the upload action does not occur on a particular server.

3.7 Sia - Distributed storage project based on independent public chain

Siacoin is based on a separate Sia common chain used to resolve agreements between storage tenants and vendors.

Sia is a decentralized cloud storage platform, based on the block chain the Nebulous company is responsible for the development and operation. Sia splits, encrypts, and then distributes the split and encrypted data over its decentralized network.

By allowing users to "rent" their unused hard drive space, Sia can significantly reduce the overhead cost of cloud storage for users, leading many to call it the Airbnb for hard drives. Sia is completely private and data files cannot be viewed without a private key.

3.8 Arweave -- a non-IPFS official incentive layer featuring permanent data storage

Unlike IPFS, which features decentralised storage, and Crust, which features a decentralised cloud service, Arweave will feature permanent storage. Arwaeve focused on the limited freedom of expression, excessive censorship and easy tampering on the Internet, creating a one-time payment and access to permanent file storage. A storage solution called Permaweb Permanent Network is provided that takes advantage of the blockchain's non-tamper-resistant nature to write content directly to the block for storage.

Arweave's main web was launched in June 2018. The project USES Proof of Access (POA) to motivate miners to store their history forever and share it on demand, rewarding them with new blocks as well as random old blocks in the storage chain. Arweave mining USES RadomX algorithm, and introduces the block integrity rate parameter on this basis.

At the same time, a number of community developers have pointed out that Arweave has a narrow app scenario and that the most it stores is a screenshot of liberal speech on Twitter. At the same time, Arweave's feature is that it can never be tampered with, making it more difficult to develop programs.

04. The core elements of the success of the "Storage Track" project

Throughout the current blockchain ecology, the bottom layer of blockchain is widely used for trust aggregation, but it often lacks the support of a large number of storage resources on the way to realize decentralized application. Therefore, we believe that decentralized storage still belongs to the new stage and is a very potential development direction.

We see that a successful decentralized storage project should have the following characteristics:

1) Solid technical foundation: The bottom layer of technology determines the superstructure of the project and plays a decisive role in the consensus and future development of the project;

2) Innovative incentive mechanism: Based on the technical feasibility of decentralized storage, the design of incentive model is the core for the smooth start and even sustainable development of this project;

3) Clear scene cutting and user positioning: this track is big to store, and focusing on a specific segment is conducive to focus on consensus;

4) Excellent team: We can see that most of the teams in successful decentralized storage projects have the ability to build decentralized storage and design the economic model

4.1 Solid technical foundation

In terms of file sharing, decentralized storage system file sharing mode is completely different from centralized storage. After uploading large files of centralized storage system, the files are stored in a whole or slice form on a single or distributed network or server, which requires its efficient development and operation team to maintain its operation. Decentralized storage, however, you must use the distributed storage technology, the initial seed node (originally had full file resource nodes) after biopsy large file processing, make its produce multiple Pieces, each Piece separately stored in different nodes, each node in commonly downloaded a single Piece and uploaded to the decentralized storage network to become the seeds of this Piece after download other node, in the process of sharing multiple nodes to complete each Piece, realize the Piece in except the initial seed node node sharing, And the number of nodes in the file sharing network is constantly expanded. So, other things being equal at the same time, as more people download the same content, the faster they download it. Therefore, the decentralized storage system makes up for the slow transmission speed of the centralized storage system, overcomes the single point of failure and guarantees the data security.

IPFS, the pioneer of centralized storage, has been growing as freely as BT since its launch in 2014, and has been storing large amounts of data. But for IPFS to become a commercially available storage system, rather than a casual data-sharing platform, quality of service assurance must be provided. This is the problem Filecoin addresses, the economic incentive layer of IPFS.

The Filecoin protocol builds two markets: the data storage market and the data extraction market. Users with storage needs go to the data storage market to declare their needs: I want to store XX size of data, require XX copies, storage for XX days. The storage service provider in the market (the storage miner) quotes this storage requirement, and the user accepts the quote and signs a contract with the miner to pay the fee. When users need to use the data, to the data extraction market demand; Then the extracting miners give a quotation to meet the demand of data access.

This may not seem like a complicated process, but there are several difficulties in implementing it:

Miners are required to provide non-falsifiable cryptographic certificates that store user data; During the term of the contract, the agreement is to continue to check that the miners keep the data as agreed. If they do not, the miners will be fined; To encourage miners to store data, there is an additional incentive to make stored capacity more profitable than idle capacity. At the same time, it is necessary to prevent miners from flooding garbage data to cheat additional incentives.

Filecoin designed Replication proof (PoRe) to solve the first problem, using PoTS (PoTS) and pledge mechanism to solve problem 2. The third problem is solved by fine-tuning the economic model and introducing authentication for real users.

At the same time, as an encryption asset, Filecoin's price is highly correlated with the overall cryptographic market, that is, its volatility is high. If Filecoin prices collapse, miners could walk away and lose user data. In addition, large price fluctuations also increase the implied option cost of miners pledging Filecoin.

And all of these complicated games involve the design of the incentive model that we're going to talk about in the next video.

4.2 Innovative incentive mechanism

On the basis of the basic technical feasibility of decentralized storage, the design of incentive mechanism for decentralized storage has become the key to the success of the project. Trust consensus design is the core of decentralized storage, and the economic model behind the consensus is the soul and essence of the project.

Because it needs to have storage hardware server to participate in, so decentralized storage miners have a thorough knowledge of the industry attribute of the project, and the currency of the miners, not only with financial attributes, not just dig a coin, they have a lot of the demand of data storage, and how to use incentive access card to activate the ecology economic model of initial project team is more design capabilities.

The essence of "storage mining design" is to solve the problem of incremental system pass "to whom?" "And" How much?" "And" By what?" These problems provide incentives for ecological actors and maintainers to move the system into a self-running distributed system.

On the other hand, tokens initially have no value. Through mining, the value is finally anchored and captured, making tokens become scarce and "cost of acquisition". For example, under the framework of POW mechanism, miners invest in computing power and operation and maintenance in exchange for block rewards and participate in transactions in the secondary market, thus giving rise to the concept of "shutdown price".

Thanks to the support of industrial hardware, the storage track passes through the "action" of storage, and miners invest in storage equipment and storage services in exchange for block awards, realizing the value anchor and capture with the real commercial society.

For example, an economic model for Crust ties the upper limit of what a miner can store to a CRU held by a miner (or a CRU offered by a guarantor), which will determine the upper limit of what a miner can earn from digging. A consensus of GPoS (Guaranteed Proof of Stake) has evolved from the original PoS model.

References:

1. https://filecoin.io/zh-cn/2020-engineering-filecoins-economy-zh-cn.pdf

2. Labs, P. A Guide to Filecoin Storage Mining. Filecoin Available at:

https://filecoin.io/blog/filecoin-guide-to-storage-mining/.

3. https://pcpartpicker.com/user/tperson/saved/H2BskL

4. Venturo, B. The economics of Ethereum's Casper. Medium (2018). Available at:

https://medium.com/@brianventuro/the-economics-of-ethereums-casper-6c145f7247a2.

5. https://www.reddit.com/r/CryptoCurrency/comments/982x9l/top100cryptocurrenciesrankedbyannualized/

6. http://app.czce.com.cn/cms/cmsface/option/Calculator/utCal.jsp

7. Project, T. A. Decentralised storage: Incentives vs Contracts. Medium (2019). Available at:

https://blog.goodaudience.com/decentralised-storage-incentives-vs-contracts-b74ee0b7eff1.

8. https://viewblock.io/arweave/stats

9. Bram Cohen. Incentives build robustness in bittorrent. In Workshop on Economics of Peer-to-Peer systems, volume 6, pages 68{72, 2003. [19] Matt Corallo. Compact block relay. bip 152, 2017.