header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

FreeToken Open Source: A 5090 running DeepSeek-V4-Flash, achieving up to 25 Token/s

Perceive Beating AI News Flash: Researchers from Berkeley, MIT, and others have open-sourced FreeToken. This is a MoE (Mixture of Experts) inference engine designed specifically for edge devices, aiming to address the issue of oversized models that cannot fit in GPU memory. It offloads part of the model weights to system memory and enables CPU and GPU joint inference.


FreeToken places a significant portion of the model's expert weights in system memory, with only a fraction cached in GPU memory. When encountering an expert not present in GPU memory, it dynamically decides, based on PCIe and CPU memory bandwidth, whether to move the weights to the GPU or directly involve the CPU in computation. This allows CPU, GPU, system memory, and PCIe to collaborate in inference.


In empirical tests, a gaming desktop equipped with an RTX 5090 running the 284B-parameter DeepSeek-V4-Flash model achieved 22–25 Tokens/s. Since this model only activates about 13B parameters per generated Token, it is particularly suitable for CPU-GPU hybrid inference.


FreeToken also ran a 35B model on an 8GB GPU memory RTX 4060 laptop at 39.3 Tokens/s; while a single 96GB RTX PRO 6000 could run the 753B-parameter GLM-5.2 model at approximately twice the speed of llama.cpp. The project currently supports over 20 MoE models and is open-sourced under the Apache 2.0 license.

举报 Correction/Report
RelatedRecommended
The road ahead no longer allows it to continue walking with its eyes closed.
Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish