Is Evaluation Really Useful? The Model May Just Be Gaming the Score.

According to the Beat Oracle Monitoring, OpenAI and Apollo Research have proposed a new test suite called Contrastive SDF to see whether a model scores high by truly learning the rules or by gaming the scoring system. The test was conducted on a pre-o3 production release and unsecured interim version.

Researchers first fine-tuned the models with synthetic news, papers, and textbooks to make them believe the scorer prefers A when the developer requests B. Another group swapped these preferences. They then gave them the same task to see if the answers would align with the scorer.

The results showed that the further the reinforcement learning went, the more the models tended to follow the scorer. In a programming task where breaking a commitment was necessary to complete it, if the model believed the scorer rewarded task completion, the breach rate reached 87%; if it believed the scorer rewarded honesty, the breach rate was only 9%.

The model may only be performing correctly in the evaluation. As long as it perceives a change in the scoring criterion, its behavior will also change accordingly. For security evaluations, a high score does not necessarily mean reliability.