Perceive Beating AI News Flash: Researchers from Berkeley, MIT, and others have open-sourced FreeToken. This is a MoE (Mixture of Experts) inference engine designed specifically for edge devices, aiming to address the issue of oversized models that cannot fit in GPU memory. It offloads part of the model weights to system memory and enables CPU and GPU joint inference.
FreeToken places a significant portion of the model's expert weights in system memory, with only a fraction cached in GPU memory. When encountering an expert not present in GPU memory, it dynamically decides, based on PCIe and CPU memory bandwidth, whether to move the weights to the GPU or directly involve the CPU in computation. This allows CPU, GPU, system memory, and PCIe to collaborate in inference.
In empirical tests, a gaming desktop equipped with an RTX 5090 running the 284B-parameter DeepSeek-V4-Flash model achieved 22–25 Tokens/s. Since this model only activates about 13B parameters per generated Token, it is particularly suitable for CPU-GPU hybrid inference.
FreeToken also ran a 35B model on an 8GB GPU memory RTX 4060 laptop at 39.3 Tokens/s; while a single 96GB RTX PRO 6000 could run the 753B-parameter GLM-5.2 model at approximately twice the speed of llama.cpp. The project currently supports over 20 MoE models and is open-sourced under the Apache 2.0 license.
