AI Programming Assignments Scores Are Inaccurate Again, OpenAI Finds Issues with SWE-Bench Pro

According to Dynamic Probe Beating monitoring, OpenAI has released a programming assessment audit report, stating that approximately 30% of the questions in SWE-Bench Pro are not suitable for effective assessment. SWE-Bench Pro was originally used to test the real programming capabilities of AI agents and was also recommended by OpenAI as an alternative evaluation set to SWE-bench Verified.

OpenAI audited 731 publicly available questions in SWE-Bench Pro. Automated processes flagged 200 questions, accounting for 27.4%; manually labeled processes found 249 questions, accounting for 34.1%.

The main issues include: incomplete question descriptions, overly strict testing requirements, hidden tests checking for additional criteria, and insufficient test coverage. This can lead to model scoring inaccuracies: a failure does not necessarily mean the model is incapable, and a pass does not necessarily mean the problem has been truly fixed.

OpenAI stated that the pass rate of state-of-the-art models on this set of public questions increased from 23.3% to 80.3% within 8 months. However, in situations where noise exists in the questions themselves, such score improvements need to be interpreted more cautiously.

OpenAI has withdrawn its previous recommendation to use SWE-Bench Pro and is calling on the community to rebuild a more reliable programming assessment.