1.5TB to 214GB: Tencent Provides Ultra-Lite Version for Hy4 Preview

动察 Beating AI News Flash: Hy4 preview has just been open-sourced, and Tencent Hyperform released an ultra-limited edition. The original model's weight was close to 1.5TB, while the new GGUF version is only about 214GB, significantly reducing the local deployment threshold of this 770B MoE model.

It did not compress the entire model to 1.25 bits. Tencent quantized the precision differently for each layer based on sensitivity, with the least sensitive parts quantized to about 1.31 bits, while the sensitive parts retained 2 bits or even higher precision. The overall file now averages around 2.38 bpw. In the four evaluations provided by Tencent, the new version only drops by 0.2 to 1.6 points compared to the original BF16.

After compression, Tencent also tested heterogeneous device joint inference with prima.cpp. By using an RTX 4090 laptop and a four-card A4000 server, with a total of only 80GB VRAM and 64GB RAM, they were able to run the model at 1.02 tokens/s, about 6 times faster than single-machine offload on the laptop. Different configurations of devices can collaborate to share the model inference workload.