header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

Databricks Runs Real Codebase Tests, GLM 5.2 Moves to the Forefront

According to Dynamic Beating monitoring, Databricks has released an internal AI Programming Agent evaluation, using real PRs from its own multimillion-line-of-code repository for testing.

This evaluation did not directly use SWE-Bench. Databricks believes that publicly available benchmarks may have been trained on, and are not close enough to real-world engineering.

The testing tasks were drawn from recently merged PRs, covering languages such as Python, Go, TypeScript, and Scala. The team filtered out bot submissions, AI-generated code, and auto-generated code, retaining only tasks that could validate the results.

The results show that the Programming Agent has been roughly divided into three tiers. The top tier includes not only OpenAI and Anthropic models but also open-source models. GLM 5.2 has entered the top tier, with quality close to Opus 4.8, at a single-task cost of $1.28, lower than Opus 4.8's $1.94.

The evaluation also found that the token price cannot directly represent the real cost. Sonnet 5 is cheaper per token, but it runs longer and reads more in this set of tasks, resulting in a higher single-task cost.

The choice of tool is also crucial. Switching the same model to different Agent frameworks can result in task costs differing by over 2 times, but with little change in completion quality.

举报 Correction/Report
RelatedRecommended
The Metaverse Agent is leading a revolution, bringing a new twist to Web3 gaming with AI agents and tokenization
In less than a month, the total market capitalization of the AI Agent Racetrack project reached a staggering ten billion dollars.
Can the Narrative of AI Agent and DeSci Carry Through the New Year?
Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish