header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

The truth is, even the most powerful tools cannot read minds. Meituan has released a new benchmark for AI systems, VitaBench 2.0.

According to Perceiving Dynamics Beating monitoring, Meituan's LongCat team, in collaboration with the National University of Singapore, University of Science and Technology of China, Beijing University of Posts and Telecommunications, and Zhejiang University, has released the VitaBench 2.0 benchmark for intelligent agents. The evaluation system has expanded large-scale model assessment from single-task to long-term multi-session lifestyle services, assessing the performance of intelligent agents in personalized modeling and proactive interaction.

The evaluation simulated long-term human-machine interaction, including 56 user profiles, over 2,000 dynamic preference annotations, and 819 service requests in areas such as food delivery, in-store consumption, and online travel. The test found that as the tool's invocation error rate significantly decreased, capturing and applying user preferences has become a core bottleneck limiting the deployment of intelligent agents.

In the evaluation of multiple cutting-edge large-scale models, the flaws of large models in long-term personalized services were prominently exposed. Even in the most ideal full-context mode that can read all history, the best-performing Claude-Opus-4.6 running independently four times had an average performance (Avg@4) of only 0.503, while the strongest non-contemplative model, DeepSeek-V4-Pro, scored only 0.456.

Most models did not exhibit stable performance improvements over the full-context baseline after introducing external memory modules. Intelligent agents' autonomously controlled memory often lost information during updates, and retrieval-assisted generation (RAG) was easily disturbed by noise with similar representations. At the same time, enabling contemplation mode did not show a systematic advantage in personalized modeling, demonstrating that noise filtering and proactive questioning cannot be solved solely through logical deduction.

The test also revealed the major weaknesses of large models in proactive questioning and preference application. When information was insufficient, models often made blind guesses and preempted decisions, resulting in a sharp decline in performance for the Claude intelligent agent in proactive interaction tasks, from 46.0 to 27.4, DeepSeek from 44.1 to 27.8, and GLM from 36.4 to 19.3. Even when the most accurate standard preference data was directly input into memory, DeepSeek and Claude only scored 52.7 and 51.2, respectively, indicating that in the actual decision-making process of large models, the difficulty of reasonable ranking, integration, and execution of user preferences far exceeds preference extraction itself.

An analysis of the reasons for errors showed that as the capabilities of large models evolved, tool misuse-class errors significantly decreased, while errors due to bias in preference understanding began to dominate. The future core barrier for intelligent agents will no longer be just tool invocation but truly understanding and remembering the unspoken wishes of users.

举报 Correction/Report
RelatedRecommended
Companies such as Bytedance are also discussing possible partnerships with Banks.
The Meituan ecosystem, Sequoia Capital, and institutions under the control of Liang Wengen have realized significant gains, while DJI missed out on substantial returns due to not completing an early investment.
Most Internet celebrities don't have that much of Web3. Helping them enter Web3 with the lowest threshold and realize deeper interaction with fans is a core value of Slash
Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish