Tencent Hy3 Compresses Hydrated Tensor to 1 Bit, Allowing Single Card to Handle 295 Billion Parameters

According to Dongcha Beating monitoring, Tencent has released compressed 1-bit and 4-bit versions of the Hy3 model, significantly reducing the deployment threshold of this 295 billion parameter model. The original Hy3 model weighs close to 600GB. The 1-bit version has been reduced to 85.5GiB, which can be run on a single 96GB NVIDIA H20.

When running on a single card, users need to disable some acceleration features and shorten the length of text processed by the model at once. When using two H20s, acceleration can be enabled. The 4-bit version has a size of 169.9GiB and requires at least two H20s. It consumes more VRAM, but the effect is closer to the original model.

Tencent stated that the 4-bit version is close to the original model in terms of code, tool invocation, long-text understanding, and Agent tasks. The 1-bit version also only experiences a slight decline. After enabling generation acceleration, the speed of the 1-bit version increases by about 50%, while the 4-bit version increases by close to 60%.

Tencent also provides a 4-bit version suitable for server deployment, which can be turned into an API for multiple users or applications to call.