According to the Beatify Monitoring, Cognition has launched a new smart code evaluation set called FrontierCode, focusing on assessing the "Mergeability" of AI-generated code rather than just correctness. The evaluation set was collaboratively developed by 36 open-source project maintainers including Celery, Budibase, Uppy, and Mattermost, with each task requiring over 40 hours of developer effort to examine code test quality, design consistency, modification scope, and style adherence.
FrontierCode is divided into three levels: Extended (150 tasks), Main (100 tasks), and Diamond (50 tasks of highest difficulty). In the extremely challenging Diamond task set, there is still significant room for improvement in current large models' performance: the top-performing Claude Opus 4.8 only achieved a 13.4% score, GPT-5.5 scored 6.3%, Gemini 3.1 Pro scored 4.7%, and the best-performing open-source model Kimi K2.6 scored only 3.8%.
To prevent large models from passing the test through cheating or gaming the system, the evaluation set has introduced multiple assessment mechanisms, including requiring tests written by the AI to fail on the original defective code in reverse testing and constraining the range of modified files and lines. The development team also designed the mutagent tool based on large models, dynamically adjusting the test environment to reasonably accommodate differential implementations, reducing the evaluation misjudgment rate to one-fifth of SWE-Bench Pro.
