Foreword

Since the birth of GPT-3, generative AI has brought explosive turning points to the field of artificial intelligence with its amazing performance and broad application scenarios, and technology giants have begun to jump into the AI race. However, problems have also arisen. The operation of training and inference of large language models (LLMs) requires a lot of computing power. With the iterative upgrade of the model, the demand for computing power and cost increases exponentially. Taking GPT-2 and GPT-3 as examples, the parameter difference between GPT-2 and GPT-3 is 1166 times (GPT-2 has 150 million parameters, and GPT-3 has 175 billion parameters). The cost of one training of GPT-3 is calculated based on the price model of public GPU cloud at that time, which can reach up to 12 million US dollars, 200 times that of GPT-2. In actual use, each user's question requires inference calculation. Based on the situation of 13 million independent user visits at the beginning of this year, the corresponding chip demand is more than 30,000 A100 GPUs. Therefore, the initial investment cost will reach an astonishing 800 million US dollars, and the daily model inference cost is estimated to be 700,000 US dollars.

Insufficient computing power and high costs have become challenges facing the entire AI industry, and it seems that the same problem will also plague the blockchain industry. On the one hand, with the fourth halving of Bitcoin and the upcoming ETF approval, as future prices rise, the demand for computing hardware by miners will inevitably increase significantly. On the other hand, Zero-Knowledge Proof (ZKP) technology is booming, and Vitalik has repeatedly emphasized that the impact of ZK on the blockchain field in the next decade will be as important as the blockchain itself. Although this technology is highly anticipated by the blockchain industry, ZK, like AI, also requires a lot of computing power and time to generate proofs due to its complex calculation process.

In the foreseeable future, computing power shortage will become inevitable. Will a decentralized computing power market be a good business?

Decentralized Computing Power Market Definition

The decentralized computing power market is essentially equivalent to the decentralized cloud computing track, but compared to decentralized cloud computing, I personally think that this term is more appropriate to describe the new projects that will be discussed later. The decentralized computing power market should belong to the subset of DePIN (decentralized physical infrastructure network), which aims to create an open computing power market, where anyone with idle computing resources can provide their resources on this market through token incentives, mainly serving B-end users and developer communities. In terms of more well-known projects, networks such as Render Network, which is based on decentralized GPU rendering solutions, and Akash Network, a distributed peer-to-peer market for cloud computing, both belong to this track.

And the following text will start with basic concepts, and then discuss three emerging markets in this field: AGI computing power market, Bitcoin computing power market, and AGI computing power market in the ZK hardware acceleration market. The latter two will be discussed in "Prospects for Decentralized Computing Power Market (Part II)".

Computing Power Overview

The origin of the concept of computing power can be traced back to the invention of computers. Initially, computers were operated by mechanical devices to perform computing tasks, and computing power referred to the computing ability of these mechanical devices. With the development of computer technology, the concept of computing power has also evolved. Nowadays, computing power usually refers to the ability of computer hardware (such as CPU, GPU, FPGA, etc.) and software (such as operating systems, compilers, applications, etc.) to work together.

Definition

Computing Power refers to the amount of data that a computer or other computing device can process or the number of computing tasks it can complete within a certain period of time. Computing power is often used to describe the performance of a computer or other computing device, and it is an important indicator for measuring the processing capability of a computing device.

Measurement Standards

Computing power can be measured in various ways, such as computing speed, energy consumption, computing accuracy, and parallelism. In the field of computer science, commonly used metrics for measuring computing power include FLOPS (floating-point operations per second), IPS (instructions per second), TPS (transactions per second), and so on.

FLOPS (Floating Point Operations Per Second) refers to the ability of a computer to process floating point operations (mathematical operations involving numbers with decimal points, which require consideration of precision and rounding errors). It measures how many floating point operations a computer can complete per second. FLOPS is a metric used to measure the high-performance computing capabilities of computers, typically used to measure the computing power of supercomputers, high-performance computing servers, and graphics processing units (GPUs). For example, if a computer system has a FLOPS of 1 TFLOPS (1 trillion floating point operations per second), it means that it can complete 1 trillion floating point operations per second.

IPS (Instructions Per Second) refers to the speed at which a computer processes instructions. It measures how many instructions a computer can execute per second. IPS is a metric used to measure the single-instruction performance of a computer, typically used to measure the performance of central processing units (CPUs). For example, if a CPU has an IPS of 3 GHz (3 billion instructions per second), it means it can execute 3 billion instructions per second.

TPS (Transactions Per Second) refers to the ability of a computer to process transactions, measuring how many transactions a computer can complete per second. It is commonly used to measure the performance of database servers. For example, if a database server has a TPS of 1000, it means it can process 1000 database transactions per second.

In addition, there are also some computing performance indicators for specific application scenarios, such as inference speed, image processing speed, and speech recognition accuracy.

Types of Computing Power

GPU computing power refers to the computing ability of Graphics Processing Units. Unlike Central Processing Units (CPUs), GPUs are specifically designed to handle graphic data such as images and videos. They have a large number of processing units and efficient parallel computing capabilities, allowing for simultaneous processing of a large number of floating-point operations. Since GPUs were originally designed for gaming graphics processing, they typically have higher clock frequencies and larger memory bandwidths than CPUs to support complex graphic operations.

CPU and GPU Differences

Architecture: The computing architecture of CPU and GPU is different. CPU usually adopts one or more cores, each of which is a general-purpose processor capable of executing various operations. GPU, on the other hand, has a large number of stream processors and shaders, which are specialized processors used for performing image processing-related operations.

Parallel Computing: GPUs typically have higher parallel computing capabilities. The number of cores in a CPU is limited, and each core can only execute one instruction, but a GPU can have thousands of stream processors that can execute multiple instructions and operations simultaneously. Therefore, GPUs are usually better suited for parallel computing tasks, such as machine learning and deep learning, which require a large amount of parallel computing.

Programming: The programming of GPUs is more complex compared to CPUs and requires the use of specific programming languages (such as CUDA or OpenCL) and techniques to utilize the parallel computing capabilities of GPUs. In contrast, programming for CPUs is simpler and can use general programming languages and tools.

Importance of Computing Power

In the era of the Industrial Revolution, oil was the world's blood, permeating into various industries. Computing power is in the blockchain, and in the upcoming AI era, computing power will be the "digital oil" of the world. From the frenzy of major companies for AI chips, to Nvidia's stock breaking through trillions, to the recent US blockade of high-end chips in China, detailed to the size of computing power, chip area, and even plans to ban GPU cloud, its importance is self-evident. Computing power will be the commodity of the next era.

Overview of Artificial General Intelligence

Artificial Intelligence (AI) is a new technology science that studies, develops theories, methods, techniques, and application systems for simulating, extending, and expanding human intelligence. It originated in the 1950s and 1960s and has undergone more than half a century of evolution, experiencing three waves of symbolic, connectionist, and behavioral development. As an emerging general-purpose technology, it is driving tremendous changes in social life and various industries. The current commonly seen generative AI is specifically defined as Artificial General Intelligence (AGI), an AI system with broad understanding ability that can demonstrate intelligence similar to or surpassing that of humans in various tasks and fields. AGI basically requires three elements: deep learning (DL), big data, and large-scale computing power.

Deep Learning

Deep learning is a subfield of machine learning (ML), and deep learning algorithms are modeled after the neural networks of the human brain. For example, the human brain contains millions of interconnected neurons that work together to learn and process information. Similarly, deep learning neural networks (or artificial neural networks) are composed of multiple layers of artificial neurons that work together inside a computer. An artificial neuron is a software module called a node that uses mathematical calculations to process data. Deep learning algorithms use these nodes to solve complex problems.

According to the most authoritative calculation, the "2022-2023 Global Computing Power Index Evaluation Report" jointly compiled by International Data Corporation (IDC), Inspur Information, and the Global Industry Research Institute of Tsinghua University shows that the global AI computing market size will increase from 19.5 billion US dollars in 2022 to 34.66 billion US dollars in 2026. The market size of generative AI computing will increase from 820 million US dollars in 2022 to 10.99 billion US dollars in 2026. The proportion of generative AI computing in the overall AI computing market will increase from 4.2% to 31.7%.

Computing Power Monopoly

So do we have to accept this centralized authority and control, and pay 72% of the profit fee for computing resources? Will the giants who monopolize Web2 also monopolize the next era?

(In the context of deep learning, this problem will become more complex as models and frameworks shift from static graph construction to dynamic construction and execution.)

Solution: Although attempts to create a decentralized AGI computing power market are still in the early stages, there are two projects that have made initial progress in solving the consensus design of decentralized networks and the implementation of decentralized computing power networks for model training and inference. This article will analyze the design and issues of the decentralized AGI computing power market using Gensyn and Together as examples.

Gensyn

Gensyn is an AGI computing power market that is still in the construction phase. It aims to solve various challenges of decentralized deep learning computing and reduce the cost of current deep learning. Essentially, Gensyn is a first-layer proof-of-stake protocol based on the Polkadot network. It directly rewards solvers for their idle GPU devices used for computation and executing machine learning tasks through smart contracts.

So, returning to the previous question, building a truly trustless computing network core relies on verifying completed machine learning tasks. This is a highly complex problem that requires finding a balance between complexity theory, game theory, cryptography, and optimization.

Truebit-style incentive games: using collateral and slashing to build incentive games, ensuring that every economically rational participant will act honestly and perform their expected tasks.

The participant system consists of submitters, solvers, validators, and reporters.

Submitter(s):

The solver is the main worker of the system, responsible for executing model training and generating proofs that are verified by the verifier.

Verifiers:

Validators are the key to linking non-deterministic training processes with deterministic linear calculations, copying part of the solver's proof and comparing distances with expected thresholds;

Whistleblowers:

System Operation

A model binary file (or basic architecture);

Profiling: The profiling process establishes a baseline distance threshold for learning verification. The verifier will periodically capture profiling tasks and generate mutation thresholds for comparison with learning proofs. To generate the threshold, the verifier will deterministically run and rerun a portion of the training using different random seeds, generating and checking their own proofs. During this process, the verifier will establish an overall expected distance threshold for non-deterministic work that can be used as a verification solution.

Training: After analysis, the task enters the public task pool (similar to Ethereum's Mempool). Choose a solver to perform the task and remove it from the task pool. The solver executes the task based on the metadata submitted by the submitter, as well as the provided model and training data. When performing the training task, the solver also generates a learning proof by regularly checkpointing and storing metadata (including parameters) during the training process, so that the verifier can replicate the optimization steps as accurately as possible.

Proof generation: The solver periodically stores model weights or updates and corresponding indices to the training dataset to identify the samples used for generating weight updates. Checkpoint frequency can be adjusted to provide stronger guarantees or save storage space. Proofs can be "stacked," meaning they can start from a random distribution used to initialize weights or from pre-trained weights generated by their own proof generation. This allows the protocol to establish a set of pre-trained base models (i.e., base models) that have already been proven, which can be fine-tuned for more specific tasks.

Verification of proof: After completing the task, the solver registers the completion of the task with the chain and displays their learning proof in a publicly accessible location for verifiers to access. Verifiers extract verification tasks from the public task pool and perform computational work to rerun a part of the proof and perform distance calculations. The chain (along with the threshold calculated during the analysis phase) then uses the resulting distance to determine whether the verification matches the proof.

Graph-based pinpoint challenge: After the verification learning proof, the reporter can copy the verifier's work to check whether the verification work itself is executed correctly. If the reporter believes that the verification has been incorrectly executed (maliciously or non-maliciously), they can initiate a challenge to the contract arbitration to obtain a reward. This reward can come from the deposits of the solver and verifier (in the case of a true positive), or from the lottery pool bonus (in the case of a false positive), and the arbitration is executed using the chain itself. Reporters (in their case, verifiers) will only verify and subsequently challenge the work if they expect to receive appropriate compensation. In fact, this means that reporters are expected to join and leave the network based on the number of other reporters (i.e., those with real-time deposits and challenges) engaged in other activities. Therefore, the default strategy for any reporter is to join the network when there are fewer other reporters, post a deposit, randomly select an active task, and begin their verification process. After the first task is completed, they will grab another random active task and repeat until the number of reporters exceeds their predetermined payment threshold, at which point they will leave the network (or more likely, switch to another role in the network based on their hardware capabilities - verifier or solver) until the situation reverses again.

Contract arbitration: When a validator is challenged by the accused, they enter a process with the chain to identify the disputed operation or input location, and ultimately the chain executes the final basic operation and determines whether the challenge is valid. To maintain the honesty and credibility of the accuser and overcome the validator's dilemma, periodic mandatory errors and prize payments are introduced here.

Settlement: During the settlement process, participants are paid based on the conclusions of probability and determinism checks. Different scenarios will have different payments based on previously verified and challenged results. If the work is deemed to have been executed correctly and all checks have passed, the solution provider and validator will be rewarded based on the executed operation.

Project Review

Gensyn has designed an exciting game system on the verification and incentive layers, which can quickly locate errors by identifying the divergence points in the network. However, there are still many details missing in the current system. For example, how to set parameters to ensure reasonable rewards and punishments without setting the threshold too high? Have the extreme situations and differences in solver computing power been considered in the game process? There is also no detailed explanation of heterogeneous parallel operation in the current version of the white paper. It seems that the implementation of Gensyn still faces many obstacles.