header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

Cognition Launches FrontierCode: The First AI Benchmark to Measure Code Merge Quality

According to the Beatify Monitoring, Cognition has launched a new smart code evaluation set called FrontierCode, focusing on assessing the "Mergeability" of AI-generated code rather than just correctness. The evaluation set was collaboratively developed by 36 open-source project maintainers including Celery, Budibase, Uppy, and Mattermost, with each task requiring over 40 hours of developer effort to examine code test quality, design consistency, modification scope, and style adherence.

FrontierCode is divided into three levels: Extended (150 tasks), Main (100 tasks), and Diamond (50 tasks of highest difficulty). In the extremely challenging Diamond task set, there is still significant room for improvement in current large models' performance: the top-performing Claude Opus 4.8 only achieved a 13.4% score, GPT-5.5 scored 6.3%, Gemini 3.1 Pro scored 4.7%, and the best-performing open-source model Kimi K2.6 scored only 3.8%.

To prevent large models from passing the test through cheating or gaming the system, the evaluation set has introduced multiple assessment mechanisms, including requiring tests written by the AI to fail on the original defective code in reverse testing and constraining the range of modified files and lines. The development team also designed the mutagent tool based on large models, dynamically adjusting the test environment to reasonably accommodate differential implementations, reducing the evaluation misjudgment rate to one-fifth of SWE-Bench Pro.

举报 Correction/Report
RelatedRecommended
The new generation of entrepreneurs has taken the stage. Before them lies the far more challenging question than any Olympiad: How will AI reshape the world?
Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish