According to Perceiving Dynamics Beating monitoring, Meituan's LongCat team, in collaboration with the National University of Singapore, University of Science and Technology of China, Beijing University of Posts and Telecommunications, and Zhejiang University, has released the VitaBench 2.0 benchmark for intelligent agents. The evaluation system has expanded large-scale model assessment from single-task to long-term multi-session lifestyle services, assessing the performance of intelligent agents in personalized modeling and proactive interaction.
The evaluation simulated long-term human-machine interaction, including 56 user profiles, over 2,000 dynamic preference annotations, and 819 service requests in areas such as food delivery, in-store consumption, and online travel. The test found that as the tool's invocation error rate significantly decreased, capturing and applying user preferences has become a core bottleneck limiting the deployment of intelligent agents.
In the evaluation of multiple cutting-edge large-scale models, the flaws of large models in long-term personalized services were prominently exposed. Even in the most ideal full-context mode that can read all history, the best-performing Claude-Opus-4.6 running independently four times had an average performance (Avg@4) of only 0.503, while the strongest non-contemplative model, DeepSeek-V4-Pro, scored only 0.456.
Most models did not exhibit stable performance improvements over the full-context baseline after introducing external memory modules. Intelligent agents' autonomously controlled memory often lost information during updates, and retrieval-assisted generation (RAG) was easily disturbed by noise with similar representations. At the same time, enabling contemplation mode did not show a systematic advantage in personalized modeling, demonstrating that noise filtering and proactive questioning cannot be solved solely through logical deduction.
The test also revealed the major weaknesses of large models in proactive questioning and preference application. When information was insufficient, models often made blind guesses and preempted decisions, resulting in a sharp decline in performance for the Claude intelligent agent in proactive interaction tasks, from 46.0 to 27.4, DeepSeek from 44.1 to 27.8, and GLM from 36.4 to 19.3. Even when the most accurate standard preference data was directly input into memory, DeepSeek and Claude only scored 52.7 and 51.2, respectively, indicating that in the actual decision-making process of large models, the difficulty of reasonable ranking, integration, and execution of user preferences far exceeds preference extraction itself.
An analysis of the reasons for errors showed that as the capabilities of large models evolved, tool misuse-class errors significantly decreased, while errors due to bias in preference understanding began to dominate. The future core barrier for intelligent agents will no longer be just tool invocation but truly understanding and remembering the unspoken wishes of users.
