Dynamic Beating AI News Update, Tencent Hy4 Preview This time, in addition to scaling up the model, the underlying architecture has also undergone a major overhaul. The previous generation, Hy3, used traditional full attention, while Hy4 has switched to Gated DSA, which is the gated version of DeepSeek Sparse Attention. When the model faces extremely long text, it no longer recalculates all the previous content but instead first identifies the most relevant parts to focus on.
Hy4 has also added IndexCache to DSA. Each layer of the model originally had to reevaluate "which content is most important," but now some layers can directly reuse the previously selected results, reducing a lot of redundant computation.
Tencent has explicitly stated that this design was inspired by DeepSeek and GLM.
Another change is in the residual structure. The traditional Transformer can be roughly understood as having only one information backbone, but Hy4 has expanded it into 4 using iHC, allowing the model to retain more information simultaneously in a very deep network. The number of MoE (Mixture of Experts) specialists has also increased from 192 in Hy3 to 256 routing specialists + 1 shared specialist in Hy4. Each token selects 8 routing specialists and always passes through the shared specialist.
