Meta's Chief AI Officer Alexandr Wang refutes SemiAnalysis's "benchmark gaming" claims: old tests have long been saturated

动察 Beating AI Flash: AI research firm SemiAnalysis has called out Gemini 3.8 Flash and Muse Spark 1.3 as among the most obvious "benchmaxxed" models currently, meaning they are over-optimized for public benchmarks. The two models scored 89.4% and 88.8% respectively on Terminal-Bench 2.1, nearing GPT-6 Astra and Claude Fable 5.1; yet on the newly released Terminal-Bench 4.0, they dropped to just 19.1% and 33.3%. Astra and Fable 5.1 still hold 57.7% and 55.8%.

SemiAnalysis suspects the issue lies in public tests becoming increasingly susceptible to targeted optimization. All tasks in Terminal-Bench 2.1 are public, and even if model makers don't directly train on the test questions, they can purchase reinforcement learning environment data highly similar to these tasks. Models thus become increasingly adept at this exam, but may not generalize to new tasks. However, SemiAnalysis has not provided direct evidence that Google or Meta purchased such data.

Meta's Chief AI Officer Alexandr Wang subsequently pushed back, calling the argument "silly." He cited GPT-5.6 Sol as an example, which also dropped from 88.8% on Terminal-Bench 2.1 to 37.3% on 4.0. His explanation is that 2.1 has become saturated and can no longer effectively differentiate top models, while 4.0 is far from saturation. Terminal-Bench 4.0 indeed removed 8 tasks that were saturated, had public solutions, or posed other issues, while modifying 19 questions and readjusting resource limits such as time, CPU, and memory.