SemiAnalysis tests AI with 206 professional tasks: Opus 5.5 wins but scores only 12 points

Beating AI News Flash: SemiAnalysis, in collaboration with 14 professional data companies, has launched GDPx, specifically designed to test the real-world working capabilities of top AI Agents. The full evaluation suite comprises 206 high-difficulty tasks, covering 11 domains including chip design, finance, law, biology, cybersecurity, and software engineering.

To replicate real work as closely as possible, researchers built complete professional environments for the AI. For example, the finance test directly recreates a real $880 million acquisition deal, placing approximately 4,000 anonymized documents for the Agent to sift through contracts, audit accounts, and revise financial models on its own. Chip design requires operating engineering software, materials science requires controlling simulated experimental equipment, and quantitative trading requires testing strategies against unseen market data. Each task can run for up to 8 hours by default.

The results were rather brutal. On a 100-point scale, Claude Opus 5.5 took first place with a composite score of 12, Google Gemini 4 Carbon scored 10.9, and GPT-6 Astra scored 9.6. Finance and law proved especially difficult—under the strict whole-task pass criteria, all tested models scored zero.

Top models already possess strong individual capabilities, yet they still struggle to coordinate multiple requirements in complex work. They can complete localized calculations and analyses, but often fail to judge which materials are more reliable, which conditions cannot be omitted, and cannot even identify major flaws in the solutions they submit.

In the finance test, AI often calculates numbers correctly but cannot distinguish which document is more authoritative, and even overwrites formal agreements with outdated materials. In the legal test, four models attempted 20 times to revise a financing agreement, and not once did they deliver a usable version. In 13 of those attempts, they failed to notice that other shareholders had enough votes to strip the client's protection rights, and in all 20 attempts they missed the loophole of bypassing protection clauses through subsidiaries.

When Agents begin operating systems independently, the impulse to complete tasks can also override safety rules. In 200 simulated corporate penetration tests, 196 violated at least one operating rule, and 173 involved serious violations. Some models, in their quest to find vulnerabilities, even began probing systems beyond their authorized scope.