header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

Breakthrough: Trillion-Parameter Large-Scale Model Reinforcement Learning Threshold Crossed: Open Source Prime-RL Enables Training on 28 Servers with 131k Contexts

According to Perspicacity Beating monitoring, Prime Intellect has released version 0.6.0 of the distributed reinforcement learning training framework prime-rl, overcoming the reinforcement learning training threshold for a trillion-parameter mixed-expert model MoE in a super-long-context agent task.

It is not uncommon for large models to read through 256k of super-long text. However, in reinforcement learning training, to allow the model to engage in reasoning exercises through trial and error, the GPU must hold the massive intermediate activation values of 131k length throughout, leading to a skyrocketing memory consumption. Previously, this required a massive cluster of thousands of GPUs, while prime-rl 0.6.0 successfully completed 131k context reinforcement learning training for GLM-5 using only 28 H200 servers, keeping the single-step time below 5 minutes.

To address the issue of a few highly time-consuming tail tasks blocking the overall pace, causing GPU resources to remain idle for extended periods during complex code generation and other trial tasks, the framework has broken away from the traditional synchronous waiting mechanism and adopted a fully decoupled asynchronous RL architecture. The background trainer, upon computing new weights, no longer waits for ongoing trial tasks to finish but instead directly issues updates in real-time during text generation by the model. Distributed tasks continue to use the old policy to maintain speed, while new tasks inject KV-cache salt to forcefully rebuild caches. To tackle the problem of model logic confusion resulting from inconsistencies between training and inference steps in asynchronous updates, the framework has introduced Routing Replay (R3) technology to handle expert data distribution at the lowest level, bypassing system latency from data conversion and reducing the discrepancy between the two ends to one-tenth, significantly stabilizing asynchronous training.

For maximum resource utilization, the framework has thoroughly addressed the pain point of GPU memory overflow due to lengthy text by employing a meticulously designed architecture. The inference end utilizes a read-and-write separated computation design to prevent large models from getting stuck in text generation following the reading of extensive context. It also leverages multiple GPUs to share expert knowledge and utilizes Mooncake technology to combine idle memory and disk space across multiple servers into a shared cache pool. For parallel computation of super-long texts, tailored to the unique DSA sparse attention mechanism of GLM-5, the framework has devised a customized parallel solution, ensuring the model's comprehensive view while reducing data communication overhead between GPU layers to just once. On the training end, combining DeepGEMM with DeepSeek V3's proposed block scaling FP8 training enhances training and inference ends to utilize the same precision and computing kernels, fundamentally eliminating training crashes caused by precision deviations.

Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish