According to Dynamic Beating monitoring, the fintech unicorn Ramp has released a private benchmark, Ramp SWE-Bench, for cutting-edge AI code intelligence agents. Ramp SWE-Bench consists of 80 backend development tasks derived from Ramp's actual production environment, aimed at addressing data leakage and metric saturation issues caused by model pretraining on public benchmark datasets.
The test tasks are all extracted from real pull requests (PRs) successfully deployed to production by Ramp's internal AI code assistant, Inspect. Each task refactors the base codebase before the PR submission, keeping the merged code and tests as a gold standard, and extracting engineers' original intent from Inspect dialogues as prompts. Evaluation is done in the mini-swe-agent sandbox environment, with a pass@1 standard of successfully passing all tests in a single run without breaking existing functionality.
Based on the published results of the 14 models' cross-type evaluations, Anthropic's latest release, Claude Fable 5, ranks first with a resolution rate of 87.5%. Claude Opus 4.7 and GPT-5.5 are tied for second place, both with a resolution rate of 83.75%. Although Claude Opus 4.8 has a resolution rate of 77.5%, its average single-task runtime has been shortened to 8 minutes 30 seconds, with a single-run cost of only $1.09, 40% lower than the runtime cost of Fable 5, demonstrating excellent cost-effectiveness.
The test data also revealed the trade-off between price and performance among different models. The domestic model Kimi K2.6 and GLM 5.1 have similar resolution rates of 72.5% and 71.25%, respectively, but Kimi K2.6 has an average cost of $0.69, approximately 34% cheaper than GLM 5.1. In the lightweight model category, GPT-5.4 Mini leads Claude Haiku 4.5 by about 10 percentage points in resolution rate at 58.75%, with both average cost and number of steps halved. Ramp points out that as models move towards deployment, the trade-off between performance, speed, and cost is becoming a key consideration for engineering practicality.
