Over 70 Individuals Utilize Consumer GPUs to Accomplish Distributed Training of a 72 Billion Parameter Large Model, Huang Renxun: "Open Source and Closed Source Are Not A or B, but A Plus B"

According to 1M AI News monitoring, the decentralized AI network Bittensor's Subnet 3 (Templar) recently completed the largest distributed large-scale model pretraining to date, codenamed Covenant-72B, with 720 billion parameters and training data of around 1.1 trillion tokens. This was achieved by over 70 contributors collaborating across the network using ordinary consumer-grade hardware, without the need for centralized compute infrastructure, and demonstrated performance comparable to LLaMA-2-70B in benchmark tests.

Following the mention of this achievement in the All-In podcast, NVIDIA CEO Jensen Huang made a clear statement on the relationship between open source and closed source: "This is not A or B, but A plus B, without a doubt." He pointed out that general models (such as ChatGPT, Claude, Gemini) are suitable for most users, while "industry-specific domain knowledge can only be encapsulated and controlled through open source models, which is the fundamental value of open source." He also revealed that every startup currently invested in by NVIDIA "prioritizes adopting an open-source model first, and then gradually integrates closed-source models."