Far from truly replacing programmers: Agents entering enterprise private code achieve a maximum pass rate of only 40%.

Beating AI News Flash: YC-incubated AI data company Specific Labs has launched a coding benchmark called Real-SWE, specifically designed to test Coding Agents using real companies' private code. The tasks come directly from enterprise production environments, including tax calculation, billing migration, API billing, and customer data migration. Neither the code nor the answers have been published online, so the Agent has to figure out the company's business rules and code structure on its own.

As a result, Fable 5.1 ranked first with a pass rate of only 38.8%; GPT-6 Astra scored 33.8%, and Gemini 3.8 Flash scored 31.2%. GLM 5.3 ranked fourth at 28.8%, higher than Grok 4.6, Kimi K3, and GPT-5.6 Sol. Among the 10 currently public tasks, 6 have an overall pass rate below 15%, and one task was failed by all models.

The most surprising result was GLM 5.3. Real-SWE places greater emphasis on an Agent's ability to continuously read code, identify rules, and complete multi-step modifications in unfamiliar projects. Zhipu researcher Xiaopu Peng responded that they have been training long-horizon tasks since GLM-5.1, and that Real-SWE is closer to this type of capability than public SWE leaderboards, which partly explains why GLM 5.3 performed prominently on this benchmark.