According to Dynamic Beating monitoring, Databricks has released an internal AI Programming Agent evaluation, using real PRs from its own multimillion-line-of-code repository for testing.
This evaluation did not directly use SWE-Bench. Databricks believes that publicly available benchmarks may have been trained on, and are not close enough to real-world engineering.
The testing tasks were drawn from recently merged PRs, covering languages such as Python, Go, TypeScript, and Scala. The team filtered out bot submissions, AI-generated code, and auto-generated code, retaining only tasks that could validate the results.
The results show that the Programming Agent has been roughly divided into three tiers. The top tier includes not only OpenAI and Anthropic models but also open-source models. GLM 5.2 has entered the top tier, with quality close to Opus 4.8, at a single-task cost of $1.28, lower than Opus 4.8's $1.94.
The evaluation also found that the token price cannot directly represent the real cost. Sonnet 5 is cheaper per token, but it runs longer and reads more in this set of tasks, resulting in a higher single-task cost.
The choice of tool is also crucial. Switching the same model to different Agent frameworks can result in task costs differing by over 2 times, but with little change in completion quality.
