SmartDec's Tang Jie Reviews Scaling Law: The Trillion-Parameter Model Was a Detour Taken by the Entire Industry

The Pulse of AI Advancement Newsflash: Tang Jie, Co-Founder of IntellectMap, posted a detailed analysis of the Scaling Law. In the past few years, whenever a large model was released, the most common question was "How many parameters?" However, Tang Jie believes that parameters alone can no longer explain a model's capabilities. The amount of data, where the compute power is utilized, and the model's future applications are equally crucial.

The industry has already experienced a detour in this regard. In 2020, Kaplan et al.'s Scaling Law suggested that after an increase in compute power, the parameters should grow faster than the training data. GPT-3, Gopher, MT-NLG, and others have been scaling up in this direction, propelling the industry towards models with trillions of parameters.

In 2022, Chinchilla conducted new experiments and found that the problem lied precisely here. Many large models became oversized, yet they were not fed with sufficient data. As compute power continued to increase, parameters and data should have grown in parallel. Looking back, Tang Jie realized that the race for trillion-parameter models was a detour that the entire industry took together, only to backtrack later.

However, Chinchilla's calculations only consider the training aspect. Once the model goes into production, it will be repeatedly invoked, making long-term inference costs increasingly important. Taking this into account, the optimal solution will change again. Models could be smaller and trained more extensively with additional data. Subsequently, models like Llama-2-7B and Gemma-2-9B have been moving in this direction.

Further on, Mixture of Experts (MoE) complicated the matter. More total parameters usually indicate a capacity for more knowledge, whereas the parameters and compute depth involved in actual calculations significantly impact inference capabilities. These two aspects should not be conflated. By 2025, new research has revealed that different tasks require different scaling approaches. Memory-intensive tasks benefit from more parameters, while data-intensive tasks benefit from more data. Within MoE, simply adding more total parameters may even hamper inference, as increasing the actual experts involved in computations proves more effective.

Vulnerability analysis is a straightforward example. Discovering new vulnerabilities requires more than memorizing additional CVEs. The real challenge lies in executing a twenty-step reasoning process without interruption. Having more parameters does not necessarily equate to deeper thinking.

GLM-5.3 serves as Tang Jie's latest validation. It shares the same base and architecture as GLM-5.2, with no change in total or activation parameters. The team continued training for just one month, focusing on expanding long-term task environments and reinforcement learning, resulting in a significant improvement in performance. This time, the enhancement was not in parameters but in post-training.

Tang Jie's final conclusion is clear. Scaling is far from over. There is still room to increase the model size, pre-training data, compute allocation for each computation, and the depth of post-training. However, each generation of models faces different bottlenecks, and the most critical areas for enhancement will also evolve.

In the next round of large model competition, the focus may shift from who has the most parameters to who understands better where the compute power should be allocated.