Original Title: "IOSG Weekly Brief | From Compute Power to Intelligence: Reinforcement Learning-Driven Decentralized AI Investment Map"

Original Author: Jacob Zhao, IOSG Ventures

Artificial Intelligence is moving from "pattern fitting" -based statistical learning to a core capability system based on "structured inference," with the importance of Post-training quickly rising. The emergence of DeepSeek-R1 marks a paradigm-level transformation of reinforcement learning in the era of large models. An industry consensus is forming: pre-training builds the general-purpose capability base of models, and reinforcement learning is no longer just a value alignment tool but has been shown to systematically improve the quality of the reasoning chain and the ability to make complex decisions, gradually evolving into a technical path to continuously enhance intelligence.

Meanwhile, Web3 is restructuring AI's production relations through a decentralized compute network and a crypto incentive system, and reinforcement learning's structural requirements for rollout sampling, reward signals, and verifiable training naturally align with blockchain's collaborative compute, incentive distribution, and verifiable execution. This research report will systematically dissect the AI training paradigm and the principles of reinforcement learning, demonstrate the structural advantages of reinforcement learning × Web3, and analyze projects such as Prime Intellect, Gensyn, Nous Research, Gradient, Grail, and Fraction AI.

Three Stages of AI Training: Pre-training, Fine-tuning, and Post-training Alignment

The full life cycle of training a modern large language model (LLM) is usually divided into three core stages: pre-training, supervised fine-tuning (SFT), and post-training/reinforcement learning (RL). Each of them takes on the functions of "building a world model," "injecting task capabilities," and "shaping reasoning and values," with their computational structure, data requirements, and validation difficulties determining the degree of decentralization.

· Pre-training uses large-scale self-supervised learning to build the model's language statistical structure and cross-modal world model, forming the foundation of LLM's capabilities. This stage requires training on a trillion-level corpus in a globally synchronized manner, relying on thousands to tens of thousands of H100 homogeneous clusters, with costs accounting for 80–95%, extremely sensitive to bandwidth and data rights, and therefore must be completed in a highly centralized environment.

· Supervised Fine-tuning is used to inject task capabilities and instruction formats, with a small data volume accounting for about 5–15% of the cost. Fine-tuning can be done through full-model training or using the Parameter-Efficient Fine-Tuning (PEFT) method, where LoRA, Q-LoRA, and Adapter are industry standards. However, gradient synchronization is still needed to limit its decentralized potential.

· Post-training consists of multiple iterative stages that determine the model's inference capabilities, values, and safety boundaries. The methods include Reinforcement Learning Hierarchical Framework (RLHF, RLAIF, GRPO), preference optimization methods without RL (DPO), and Process Reward Modeling (PRM), among others. This stage has lower data volume and costs (5–10%) and is mainly focused on Rollout and policy updates. It inherently supports asynchronous and distributed execution, where nodes do not need to hold full weights. By combining verifiable computation and on-chain incentives, an open decentralized training network can be formed, making it the most suitable training phase for Web3.

Reinforcement Learning Landscape: Architecture, Frameworks, and Applications

Reinforcement Learning System Architecture and Core Components

Reinforcement Learning (RL) drives model self-improvement in decision-making through "environment interaction—reward feedback—policy update." Its core structure can be seen as a feedback loop consisting of states, actions, rewards, and policies. A complete RL system typically includes three types of components: Policy (policy network), Rollout (experience sampling), and Learner (policy updater). The policy interacts with the environment to generate trajectories, and the Learner updates the policy based on reward signals, forming a continuous iteration and continuous optimization learning process:

1. Policy Network (Policy): Generates actions from the environmental state, serving as the decision-making core of the system. During training, centralized backpropagation is required for consistency maintenance; during inference, it can be distributed to different nodes for parallel operation.

2. Experience Sampling (Rollout): Nodes interact with the environment according to a policy, generating state-action-reward trajectories. This process is highly parallelized, has very low communication, and is insensitive to hardware differences, making it the most suitable stage to scale in a decentralized manner.

3. Learner: Aggregates all Rollout trajectories and performs policy gradient updates. It is the module with the highest requirements for computing power and bandwidth, so it is usually deployed in a centralized or lightly centralized manner to ensure convergence stability.

Reinforcement Learning Stage Framework (RLHF → RLAIF → PRM → GRPO)

Reinforcement learning is typically divided into five stages, as described below:

Data Generation Stage (Policy Exploration)

Under given input prompts, the policy model πθ generates multiple candidate reasoning chains or complete trajectories, providing a sample basis for subsequent preference evaluation and reward modeling, determining the breadth of policy exploration.

Preference Feedback Stage (RLHF / RLAIF)

· RLHF (Reinforcement Learning from Human Feedback) trains the model with multiple candidate responses, human preference annotations, trains a reward model (RM), and uses PPO to optimize the policy, making the model's output more aligned with human values. This is a key component for GPT-3.5 → GPT-4.

· RLAIF (Reinforcement Learning from AI Feedback) replaces human annotations with an AI Judge or constitutional-style rules to achieve automated preference acquisition, significantly reducing costs and enabling scalability. It has become the mainstream alignment paradigm for companies like Anthropic, OpenAI, DeepMind, and others.

Reward Modeling Stage

Preferences feed into a reward model, teaching the model to map outputs to rewards. The RM teaches the model "what the correct answer is," while the PRM teaches the model "how to reason correctly."

· RM (Reward Model) is used to evaluate the quality of the final answer, scoring only the output:

· Process Reward Model PRM no longer evaluates only the final answer, but scores each step of reasoning, each token, each logical segment, also a key technology in OpenAI o1 and DeepSeek-R1, essentially "teaching the model how to think."

Reward Verifiability Phase (RLVR / Reward Verifiability)

Introducing "verifiable constraints" in the process of reward signal generation and usage, so that the reward comes as much as possible from reproducible rules, facts, or consensus, thereby reducing reward hacking and bias risks, and enhancing auditability and scalability in an open environment.

Policy Optimization Phase

Updating policy parameters θ under the guidance of the reward model signal to obtain a policy πθ′ with stronger reasoning ability, higher security, and more stable behavioral patterns. Mainstream optimization methods include:

· PPO (Proximal Policy Optimization): a traditional optimizer in RLHF, known for its stability, but often faces limitations such as slow convergence and insufficient stability in complex reasoning tasks.

· GRPO (Group Relative Policy Optimization): a core innovation of DeepSeek-R1, estimating the expected value by modeling the advantage distribution within the candidate answer group, rather than simple ranking. This method retains reward magnitude information, is more suitable for reasoning chain optimization, and has a more stable training process. It is considered an important reinforcement learning optimization framework for deep reasoning scenarios after PPO.

· DPO (Direct Preference Optimization): a post-training method of non-reinforcement learning: does not generate trajectories, does not build a reward model, but directly optimizes preferences. It is cost-effective and stable in effect, hence widely used in alignment of open-source models like Llama and Gemma, but does not enhance reasoning ability.

New Policy Deployment

The optimized model exhibits: stronger System-2 Reasoning capabilities, behaviors more aligned with human or AI preferences, lower illusion rates, and higher security. The model continuously learns preferences, optimizes processes, enhances decision quality through iterative refinement, forming a closed loop.

The Five Major Industrial Applications of Reinforcement Learning

Reinforcement Learning has evolved from early game intelligence to a cross-industry autonomous decision-making core framework. Its application scenarios can be categorized into five major groups based on technological maturity and industrial implementation level, each driving key breakthroughs in their respective domains.

· Game & Strategy: This was the earliest direction where RL was validated. In environments like AlphaGo, AlphaZero, AlphaStar, OpenAI Five, with "perfect information + explicit reward," RL has demonstrated decision intelligence on par with or surpassing human experts, laying the foundation for modern RL algorithms.

· Robotics & Embodied AI: Through continuous control, dynamic modeling, and environment interaction, RL enables robots to learn manipulation, motion control, and cross-modal tasks (e.g., RT-2, RT-X). This area is rapidly moving towards industrialization and is a key technical route for real-world robot deployment.

· Digital Reasoning / LLM System-2: RL + PRM is driving large models from "language imitation" towards "structured reasoning." Representative achievements include DeepSeek-R1, OpenAI o1/o3, Anthropic Claude, and AlphaGeometry. This fundamentally focuses on reward optimization at the level of reasoning chains, rather than merely evaluating final answers.

· Automated Scientific Discovery & Mathematical Optimization: RL searches for optimal structures or strategies in unlabeled data, complex reward systems, and vast search spaces. Breakthroughs like AlphaTensor, AlphaDev, Fusion RL have demonstrated an exploratory capability beyond human intuition.

· Economic Decision-making & Trading: RL is used for strategy optimization, high-dimensional risk control, and adaptive trading system generation, and compared to traditional quantitative models, it can continuously learn in uncertain environments. It is an important part of intelligent finance.

Reinforcement Learning and the Natural Match with Web3

Reinforcement Learning (RL) and Web3 are highly compatible, stemming from the fact that both are fundamentally "incentive-driven systems." RL relies on reward signals to optimize strategies, while blockchain relies on economic incentives to coordinate participant behavior, making the two naturally aligned at the mechanism level. The core requirements of RL—large-scale heterogeneous Rollout, reward distribution, and verifiability—are precisely where Web3's structural advantages lie.

Decoupling Inference and Training

The training process of reinforcement learning can be explicitly divided into two stages:

· Rollout (Exploration Sampling): The model generates a large amount of data based on the current policy, a computationally intensive but communication-sparse task. It does not require frequent communication between nodes and is suitable for parallel generation on globally distributed consumer-grade GPUs.

· Update (Parameter Update): The model's weights are updated based on the collected data, requiring high-bandwidth centralized nodes to complete.

The "Inference-Training Decoupling" naturally fits the decentralized heterogeneous computational power structure: Rollout can be outsourced to an open network, settled based on contribution through a token mechanism, while model updates remain centralized to ensure stability.

Verifiability

ZK and Proof-of-Learning provide means to verify whether nodes are genuinely performing inference, addressing honesty issues in open networks. In deterministic tasks such as code and mathematical reasoning, validators only need to check the answers to confirm the workload, significantly enhancing the credibility of decentralized RL systems.

Incentive Layer, Feedback Production Mechanism Based on Tokenomics

Web3's token mechanism can directly reward contributors of RLHF/RLAIF preference feedback, making preference data generation transparent, settlement-ready, and permissionless in terms of incentive structures; staking and slashing further constrain feedback quality, creating a more efficient and aligned feedback market than traditional crowdsourcing.

Multi-Agent Reinforcement Learning (MARL) Potential

Blockchain is essentially an open, transparent, and continuously evolving multi-agent environment, where accounts, contracts, and agents continuously adjust their strategies under incentives, naturally possessing the potential to build a large-scale MARL experimental field. Despite being still in its early stages, its characteristics of transparent state, verifiable execution, and programmable incentives provide a foundational advantage for the future development of MARL.

Classic Web3 + Reinforcement Learning Project Analysis

Based on the above theoretical framework, we will briefly analyze the most representative projects in the current ecosystem:

Prime Intellect: Asynchronous Reinforcement Learning Paradigm prime-rl

Prime Intellect is committed to building a global open compute market, lowering the training barrier, driving collaborative decentralized training, and developing a complete open-source superintelligent technology stack. Its system includes: Prime Compute (unified cloud/distributed compute environment), the INTELLECT model family (10B–100B+), the Open Reinforcement Learning Environment Hub, and large-scale synthetic data engines (SYNTHETIC-1/2).

The core infrastructure component of Prime Intellect, the prime-rl framework, is specifically designed for asynchronous distributed environments and highly relevant to reinforcement learning, with additional components including the OpenDiLoCo communication protocol that breaks through bandwidth bottlenecks and the TopLoc validation mechanism ensuring computational integrity.

Overview of Prime Intellect Core Infrastructure Components

Technical Cornerstone: prime-rl Asynchronous Reinforcement Learning Framework

prime-rl is the core training engine of Prime Intellect, designed for large-scale asynchronous decentralized environments, achieving high-throughput inference and stable updates through complete decoupling of Actor-Learner. Rollout Workers and Trainers are no longer synchronously blocked, allowing nodes to join or exit at any time, merely needing to continuously pull the latest policy and upload generated data:

· Executor (Rollout Workers): Responsible for model inference and data generation. Prime Intellect innovatively integrates the vLLM inference engine on the Executor side. The vLLM's PagedAttention technology and Continuous Batching capability enable the Executor to generate inference trajectories at a very high throughput.

· Learner (Trainer): Responsible for policy optimization. The Learner asynchronously pulls data from a shared experience replay buffer for gradient updates, without waiting for all Executors to complete the current batch.

· Orchestrator: Responsible for scheduling model weights and data flow.

Key Innovations of prime-rl

· Full Asynchrony: prime-rl abandons the traditional synchronous paradigm of PPO, does not wait for slow nodes, and does not require batch alignment, allowing any number and performance of GPUs to join at any time, laying the foundation for decentralized RL.

· Deep Integration of FSDP2 and MoE: Through FSDP2 parameter slicing and MoE sparse activation, prime-rl enables efficient training of billion-scale models in a distributed environment. Executors only run active experts, significantly reducing GPU memory usage and inference costs.

· GRPO+ (Group Relative Policy Optimization): GRPO eliminates the Critic network, significantly reducing computational and memory overheads, naturally adapting to an asynchronous environment. prime-rl's GRPO+ further ensures reliable convergence under high-latency conditions through stabilization mechanisms.

INTELLECT Model Family: A Sign of Decentralized RL Technology Maturity

INTELLECT-1 (10B, October 2024) first proved that OpenDiLoCo can efficiently train in a heterogeneous network spanning three continents (communication ratio <2%, compute utilization 98%), breaking the physical boundaries of cross-continental training;

INTELLECT-2 (32B, April 2025) serves as the first Permissionless RL model, verifying the stable convergence capabilities of prime-rl and GRPO+ in a multi-step delayed, asynchronous environment, achieving decentralized RL with global open compute participation;

INTELLECT-3 (106B MoE, November 2025) adopts a sparse architecture activating only 12B parameters, trained on 512×H200 to achieve flagship inference performance (AIME 90.8%, GPQA 74.4%, MMLU-Pro 81.9%, etc.), with overall performance nearing or even surpassing significantly larger-scale centralized proprietary models.

Prime Intellect has also built several supportive infrastructures: OpenDiLoCo reduces cross-continental training communication by hundreds of times through time-sparsity communication and quantized weight differentials, maintaining 98% utilization efficiency for INTELLECT-1 across a three-continent network; TopLoc + Verifiers form a decentralized trustful execution layer to activate fingerprint and sandbox verification to ensure the authenticity of inference and reward data; the SYNTHETIC data engine produces a large-scale high-quality inference chain and efficiently runs the 671B model on consumer-grade GPU clusters through pipelined parallelism. These components provide a key engineering foundation for data generation, validation, and inference throughput of decentralized RL. The INTELLECT series demonstrates that this technology stack can produce state-of-the-art global models, marking the transition of decentralized training systems from the conceptual stage to the practical stage.

Gensyn: Reinforcement Learning Core Stack RL Swarm and SAPO

The goal of Gensyn is to aggregate global idle compute power into an open, trustless, and infinitely scalable AI training infrastructure. Its core includes a cross-device standardized execution layer, peer-to-peer coordination network with a trustless task verification system, and automatic task and reward allocation through smart contracts. Around the characteristics of reinforcement learning, Gensyn introduces core mechanisms such as RL Swarm, SAPO, and SkipPipe, decoupling the three stages of generation, evaluation, and updating, utilizing a globally heterogeneous GPU-composed "swarm" to achieve collective evolution. What it ultimately delivers is not just raw compute power but Verifiable Intelligence.

Reinforcement Learning Application of the Gensyn Stack

RL Swarm: Decentralized Collaborative Reinforcement Learning Engine

RL Swarm demonstrates a novel collaborative pattern. It is no longer a simple task distribution but a decentralized "generate-assess-update" loop that simulates human social learning, analogous to a cooperative learning process, in an infinite loop:

· Solvers (Actors): Responsible for local model inference and Rollout generation, node-heterogeneity friendly. Gensyn integrates a high-throughput inference engine locally (such as CodeZero), capable of outputting complete trajectories instead of just answers.

· Proposers (Question Setters): Dynamically generate tasks (math problems, code challenges, etc.), supporting task diversity and Curriculum Learning-like adaptive difficulty.

· Evaluators: Use a frozen "referee model" or rules to evaluate local Rollouts, generating local reward signals. The evaluation process is auditable, reducing the space for malfeasance.

Together, they form a P2P RL organizational structure, capable of achieving large-scale collaborative learning without centralized scheduling.

SAPO: Policy Optimization Algorithm for Decentralized Refactoring

SAPO (Swarm Sampling Policy Optimization) takes "sharing Rollouts and filtering gradient-less samples instead of sharing gradients" as its core, achieving stable convergence in a decentralized environment with significant node latency differences through large-scale decentralized Rollout sampling, treating received Rollouts as locally generated. Compared to PPO, which relies on a Critic network and has high computational costs, or GRPO, which is based on intra-group advantage estimation, SAPO enables even consumer-grade GPUs to efficiently participate in large-scale reinforcement learning optimization with extremely low bandwidth.

Through RL Swarm and SAPO, Gensyn has demonstrated that reinforcement learning (especially the RLVR phase) inherently fits a decentralized architecture because it relies more on large-scale, diverse exploration (Rollout) rather than high-frequency parameter synchronization. By combining PoL and Verde's validation system, Gensyn offers an alternative path for training trillion-parameter models that no longer relies on a single tech giant: a self-evolving super-intelligent network composed of millions of heterogeneous GPUs worldwide.

Nous Research: Verifiable Reinforcement Learning Environment Atropos

Nous Research is building a set of decentralized, self-evolving cognitive infrastructure. Its core components — Hermes, Atropos, DisTrO, Psyche, and World Sim — are organized into a continuous closed-loop intelligent evolutionary system. Unlike the traditional "pre-training—fine-tuning—inference" linear workflow, Nous leverages DPO, GRPO, rejection sampling, and other reinforcement learning technologies to unify data generation, validation, learning, and inference into a continuous feedback loop, creating a self-improving closed-loop AI ecosystem.

Nous Research Component Overview

Model Layer: Hermes and the Evolution of Inference Capability

The Hermes series is the primary model interface of Nous Research for users, and its evolution clearly demonstrates the industry's transition from traditional SFT/DPO alignment to reasoning reinforcement learning (Reasoning RL):

· Hermes 1–3: Instruction alignment and early agent capability: Hermes 1–3 rely on low-cost DPO to achieve robust instruction alignment and, in Hermes 3, leverage synthetic data and the first introduction of the Atropos validation mechanism.

· Hermes 4 / DeepHermes: Embedding System-2-style slow thinking into weights through a thought chain, enhancing mathematical and code performance through Test-Time Scaling, and relying on "rejection sampling + Atropos validation" to build high-purity inference data.

· DeepHermes further adopts GRPO to replace PPO, which is difficult to deploy in a distributed manner, enabling inference RL to run on the Psyche decentralized GPU network and laying the engineering foundation for the scalability of open-source inference RL.

Atropos: Verifiable Reward-Driven Reinforcement Learning Environment

Atropos is the true hub of the Nous RL system. It encapsulates prompts, tool calls, code execution, and multi-round interactions into a standardized RL environment that can directly validate the correctness of outputs, providing deterministic reward signals to replace expensive and unscalable human labeling. More importantly, in the decentralized training network Psyche, Atropos acts as a "judge" to validate whether nodes genuinely improve policies, supporting auditable Proof-of-Learning and fundamentally addressing the reward trustworthiness issue in distributed RL.

DisTrO and Psyche: Decentralized Reinforcement Learning Optimizer Layer

Traditional RLF (RLHF/RLAIF) training relies on centralized high-bandwidth clusters, which is a core barrier to open-source replicability. DisTrO reduces the communication cost of RL by several orders of magnitude through momentum decoupling and gradient compression, enabling training to run on internet-scale bandwidth; Psyche then deploys this training mechanism on-chain, allowing nodes to perform inference, validation, reward assessment, and weight updates locally, forming a complete RL loop.

In the Nous architecture, Atropos verifies the thought chain; DisTrO compresses training communication; Psyche runs the RL loop; World Sim provides a complex environment; Forge collects real inference; Hermes records all learning into weights. Reinforcement learning is not just a training phase, but a core protocol in the Nous framework connecting data, environment, model, and infrastructure, allowing Hermes to be a living system that can continuously self-improve on an open-source compute network.

Gradient Network: Reinforcement Learning Architecture Echo

The core vision of Gradient Network is to refactor the AI computing paradigm through the "Open Intelligence Stack." Gradient's tech stack consists of a set of independently evolving, yet heterogeneously collaborative core protocols. The system includes, from bottom-level communication to upper-level intelligent collaboration: Parallax (distributed inference), Echo (decentralized RL training), Lattica (P2P network), SEDM / Massgen / Symphony / CUAHarm (memory, collaboration, security), VeriLLM (trust verification), Mirage (high-fidelity simulation), collectively forming a continuously evolving decentralized intelligent infrastructure.

Echo—Reinforcement Learning Training Architecture

Echo is Gradient's reinforcement learning framework, whose core design principle is to decouple the training, inference, and data (reward) paths in reinforcement learning, allowing Rollout generation, policy optimization, and reward evaluation to independently scale and schedule in a heterogeneous environment. It operates collaboratively in a heterogeneous network composed of inference-side and training-side nodes, maintaining training stability in a wide-area heterogeneous environment through a lightweight synchronization mechanism, effectively mitigating the SPMD failures and GPU utilization bottlenecks caused by the mixing of inference and training in traditional DeepSpeed RLHF / VERL.

Echo adopts the "Inference-Training Dual Swarm Architecture" to maximize compute utilization, with each swarm running independently without blocking each other:

· Maximizing Sampling Throughput: Inference Swarm consists of consumer-grade GPUs and edge devices, leveraging Parallax to build a high-throughput sampler through pipeline-parallelism, focusing on trajectory generation;

· Maximizing Gradient Compute: Training Swarm comprises consumer-grade GPUs that can run in centralized clusters or distributed globally, responsible for gradient updates, parameter synchronization, and LoRA fine-tuning, focusing on the learning process.

To maintain policy and data consistency, Echo provides two lightweight synchronization protocols: Sequential and Asynchronous, achieving bidirectional consistency management of policy weights and trajectories:

· Sequential Pull Mode | Accuracy First: The training side forces the inference node to refresh the model version before pulling new trajectories, ensuring trajectory freshness, suitable for tasks highly sensitive to outdated policies;

· Asynchronous Push-Pull Mode | Efficiency First: The inference side continuously generates version-tagged trajectories, while the training side consumes at its own pace, with the coordinator monitoring version deviation and triggering weight refresh to maximize device utilization.

At its core, Echo is built on Parallax (heterogeneous inference in low-bandwidth environments) and lightweight distributed training components (such as VERL), relying on LoRA to reduce cross-node synchronization costs, enabling reinforcement learning to operate stably across a global heterogeneous network.

Grail: Reinforcement Learning in the Bittensor Ecosystem

Through its unique Yuma consensus mechanism, Bittensor has built a vast, sparse, non-stationary reward function network.

Within the Bittensor ecosystem, Covenant AI has constructed a vertical integrated pipeline from pre-training to RL post-training through SN3 Templar, SN39 Basilica, and SN81 Grail. Specifically, SN3 Templar handles the pre-training of base models, SN39 Basilica provides a distributed compute marketplace, and SN81 Grail serves as the "Verifiable Inference Layer" for RL post-training, supporting the core processes of RLHF/RLAIF, completing the closed-loop optimization from base models to policy alignment.

The goal of GRAIL is to cryptographically prove the authenticity of each reinforcement learning rollout and bind it to the model's identity, ensuring that RLHF can be securely executed in a trustless environment. The protocol establishes a trusted chain through a three-layer mechanism:

1. Deterministic Challenge Generation: Using a drand random beacon and block hashes to generate an unpredictable yet reproducible challenge task (e.g., SAT, GSM8K), eliminating precomputation cheating ;

2. PRF-Indexed Sampling and Sketch Commitments: Validators sample token-level logprob and inference chain with minimal cost through PRF indexing, confirming that the rollout was indeed generated by the stated model ;

3. Model Identity Binding: Binding the inference process to the model's weight fingerprint and a structural signature of the token distribution to ensure that replacing the model or replaying results will be immediately identified. Thus, providing a foundation of authenticity for RL inference trajectories (rollouts).

Building on this mechanism, the Grail subnet has implemented a GRPO-style verifiable post-training process: miners generate multiple inference paths for the same prompt, and validators score based on correctness, inference chain quality, and SAT satisfaction, then normalize the results on-chain as TAO weights. Public experiments have shown that this framework has increased the accuracy of Qwen2.5-1.5B in MATH from 12.7% to 47.6%, demonstrating its ability to prevent cheating and significantly enhance model capabilities. Within the Covenant AI training stack, Grail serves as the cornerstone of decentralized RLVR/RLAIF trust and execution, but has not yet officially launched on the mainnet.

Fraction AI: Competition-Based Reinforcement Learning RLFC

The architecture of Fraction AI is explicitly built around Competition-Based Reinforcement Learning (RLFC) and gamified data labeling, replacing the static rewards and manual annotations of traditional RLHF with an open, dynamic competitive environment. Agents compete in different Spaces, where their relative rankings and AI Judge scores together form real-time rewards, transforming the alignment process into an ongoing online multi-agent adversarial system.

Core Difference between Traditional RLHF and Fraction AI's RLFC:

Core Value of RLFC lies in rewards no longer coming from a single model, but from an ever-evolving set of opponents and evaluators, preventing the model from being exploited and avoiding ecosystem lock-in to local optima through policy diversity. The structure of Spaces determines the nature of the game (zero-sum or non-zero-sum), driving emergent complex behaviors in adversarial and cooperative settings.

In terms of system architecture, Fraction AI breaks down the training process into four key components:

· Agents: Lightweight policy units based on open-source LLM, extended with differential weighting through QLoRA for low-cost updates;

· Spaces: Isolated task domain environments where agents pay to enter and receive rewards based on outcomes;

· AI Judges: Instant reward layer built on RLAIF, providing scalable, decentralized evaluation;

· Proof-of-Learning: Binding policy updates to specific competitive outcomes to ensure a verifiable, cheat-resistant training process.

The essence of Fraction AI is the construction of a human-machine collaborative evolutionary engine. Users act as "Meta-optimizers" at the policy layer, guiding exploration direction through Prompt Engineering and hyperparameter configuration, while agents generate massive quantities of high-quality preference pairs through micro-scale competition. This pattern enables data labeling to achieve a commercial closed loop through "Trustless Fine-tuning."

Comparison of Reinforcement Learning Web3 Project Architectures

Summary and Outlook: Pathways and Opportunities for Reinforcement Learning × Web3

Based on the deconstruction analysis of the cutting-edge projects mentioned above, we observe that: despite each team's unique entry points (algorithmic, engineering, or market), when reinforcement learning (RL) is combined with Web3, the underlying architectural logic converges into a highly consistent "Decoupling-Verification-Incentive" paradigm. This is not just a technical coincidence but rather an inevitable outcome of decentralized networks adapting to the unique properties of reinforcement learning.

Reinforcement Learning General Architecture Features: Addressing Core Physical Constraints and Trust Issues

1. Rollout-Learning Physics Decoupling (Decoupling of Rollouts & Learning)——Default Compute Topology

Communication-sparse, parallelizable Rollouts outsourced to global consumer-grade GPUs, high-bandwidth parameter updates focused on a few training nodes, from Prime Intellect's asynchronous Actor-Learner to Gradient Echo's dual-cluster architecture.

2. Verification-Driven Trust Layer (Verification-Driven Trust)——Infrastructure

In a permissionless network, computational authenticity must be enforced through mathematical and mechanism design guarantees, representing implementations such as PoL of Gensyn, TOPLOC of Prime Intellect, and cryptographic verification of Grail.

3. Tokenized Incentive Loop (Tokenized Incentive Loop)——Market Self-Regulation

Compute supply, data generation, validation ordering, and reward distribution form a closed loop, driving participation through rewards, suppressing cheating through Slash, enabling the network to remain stable and continually evolve in an open environment.

Differentiated Technology Paths: Different "Breakthrough Points" Under a Consistent Architecture

Despite architectural convergence, each project has chosen a different technological moat based on its own DNA:

· Algorithm Breakthrough Faction (Nous Research): Aims to address the fundamental contradiction of distributed training from the mathematical foundation (bandwidth bottleneck). Its DisTrO optimizer aims to compress gradient communication by thousands of times, targeting to enable large model training even on home broadband, presenting a "dimensionality reduction strike" against physical constraints.

· Systems Engineering Faction (Prime Intellect, Gensyn, Gradient): Focuses on building the next-generation "AI runtime system." Prime Intellect's ShardCast and Gradient's Parallax are both designed to squeeze out the highest efficiency of heterogeneous clusters through ultimate engineering means under existing network conditions.

· Market Game Theorist (Bittensor, Fraction AI): Focuses on the design of the Reward Function. By designing a clever scoring mechanism, it guides miners to spontaneously seek the optimal strategy to accelerate the emergence of intelligence.

Advantages, Challenges, and Endgame Outlook

In the paradigm of combining reinforcement learning with Web3, the systemic advantage is first reflected in the cost structure and governance structure rewrite.

· Cost Reshaping: The demand for sampling (Rollout) after RL training is infinite, and Web3 can mobilize global long-tail computing power at extremely low cost, a cost advantage that centralized cloud providers find hard to match.

· Sovereign Alignment: Breaking the monopoly of Big Teach AI values alignment, the community can use Token voting to determine what the model's "good answer" is, achieving democratized AI governance.

At the same time, this system also faces two major structural constraints.

· Bandwidth Wall: Despite innovations such as DisTrO, physical latency still limits the full-scale training of super large parameter models (70B+), with current Web3 AI more focused on fine-tuning and inference.

· Reward Hacking: In a highly incentivized network, miners are extremely prone to "overfitting" reward rules (gaming the system) rather than enhancing real intelligence. Designing cheat-resistant robust reward functions is an eternal game.

· Malicious Byzantine Worker Attacks: Actively manipulating and poisoning training signals to disrupt model convergence. The key is not just to continuously design cheat-resistant reward functions but to build mechanisms with adversarial robustness.

The integration of reinforcement learning with Web3 is fundamentally about rewriting the mechanism of "how intelligence is produced, aligned, and value distributed." Its evolutionary path can be summarized into three complementary directions:

1. Decentralized Inference Network: From compute mining rigs to policy networks, outsourcing parallel and verifiable Rollout to the global long-tail GPU, focusing in the short term on a verifiable inference market, and evolving in the medium term into reinforcement learning subnets clustered by task;

2. Preference and Reward Assetization: From Annotated Labor to Data Equity. Achieving the assetization of preference and reward transforms high-quality feedback in a Reward Model into a governable, distributable data asset, elevating from "annotated labor" to "data equity".

3. 'Small is Beautiful' Evolution in Verticals: Nurturing small yet powerful specialized RL Agents in verifiable outcome, quantifiable yield verticals, such as DeFi strategy execution, code generation, aligning strategy improvement with direct value capture and potentially outperforming general-purpose closed-source models.

Overall, the true opportunity of Reinforcement Learning × Web3 lies not in replicating a decentralized version of OpenAI, but in rewriting the "intelligent production relations": making training execution an open compute market, turning rewards and preferences into governable on-chain assets, shifting the value brought by intelligence away from platform centralization and towards a reallocation among trainers, aligners, and users.

Original Article Link

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia