header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

Tencent Mega-Essence Hy4 Reworks the Underlying Architecture: DeepSeek Sparse Attention + Genie IndexCache Together

Dynamic Beating AI News Update, Tencent Hy4 Preview This time, in addition to scaling up the model, the underlying architecture has also undergone a major overhaul. The previous generation, Hy3, used traditional full attention, while Hy4 has switched to Gated DSA, which is the gated version of DeepSeek Sparse Attention. When the model faces extremely long text, it no longer recalculates all the previous content but instead first identifies the most relevant parts to focus on.


Hy4 has also added IndexCache to DSA. Each layer of the model originally had to reevaluate "which content is most important," but now some layers can directly reuse the previously selected results, reducing a lot of redundant computation.


Tencent has explicitly stated that this design was inspired by DeepSeek and GLM.


Another change is in the residual structure. The traditional Transformer can be roughly understood as having only one information backbone, but Hy4 has expanded it into 4 using iHC, allowing the model to retain more information simultaneously in a very deep network. The number of MoE (Mixture of Experts) specialists has also increased from 192 in Hy3 to 256 routing specialists + 1 shared specialist in Hy4. Each token selects 8 routing specialists and always passes through the shared specialist.

RelatedRecommended
So we strove to paddle forward, sailing against the current, only to be constantly pushed back to the past.
Around 2022, Tencent's blockchain suddenly collapsed in competition with the Alibaba group and disappeared from the domestic blockchain landscape. Today, Tencent has fully retreated in the new Web3 and virtual asset fields, while Tencent employees who hav
In fact, Tencent Cloud’s cooperation with the TON Foundation is to build nodes for the TON public chain in order to achieve further development.
Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish