OpenAI Accused of Quietly Adjusting GPT-6 Astra Evaluation Data, with Some Metrics Making Competitors Look Worse

动察 Beating AI Flash: Since OpenAI released GPT-6 Astra on September 3, multiple model benchmark data points have been continuously adjusted, with some modifications making Astra's performance appear stronger while scores for certain rival models declined, sparking external questions about AI "benchmark gaming" and evaluation transparency.

Among the changes, Astra's hallucination rate was revised down from 4.2% to 2%, while GPT-5.6 Sol saw its rate drop from 12.2% to 9.4%, before both were restored to 4.2% and 12.2%, respectively. In math evaluations, Anthropic's Fable 5.1 score was also cut from 87.8% to 78%, and has since rebounded to 83%; GPT-5.6 Sol dropped from 83% to 80.5% before recovering to 83%.

Additionally, Astra's score on the ARC-AGI-3 benchmark rose from 98.6% in a pre-release draft to 99.99% on the final page, while its coding evaluation score was slightly adjusted upward from 57.7% to 57.9%. OpenAI stated that evaluation results can be affected by factors such as model versions, tool configurations, reasoning levels, and test runs, and that these adjustments were made to ensure the data more accurately reflects the model's optimal performance.

However, researchers at Stanford University argue that frequently rerunning evaluations may involve so-called "Benchmaxxing," a practice of tweaking test conditions to maximize benchmark scores. Industry insiders note that as competition among AI models intensifies, benchmark data has become a key tool for measuring model capabilities and vying for market share, and improving the transparency and reproducibility of benchmark testing is drawing increasing attention.