header-langage
简体中文
繁體中文
English
Tiếng Việt
한국어
日本語
ภาษาไทย
Türkçe
Scan to Download the APP
Partnered With:

ARC-AGI-3 Evaluation Body Considers Rule Change: Official Benchmark or Vendor Provided Memory Allowed

According to Dynamic AI Beating monitoring, the ARC-AGI-3 evaluation organization ARC Prize had previously requested all models to use the same simple invocation method to prevent vendors from inflating scores through customized frameworks. This makes it easier to see the actual capabilities of the models.

In response to the controversy over OpenAI's test, the official stance has started to relax. ARC Prize acknowledges that retaining historical reasoning, compressing lengthy contexts can indeed improve a model's performance on long tasks, and has begun researching how to incorporate such features into formal evaluations.

The boundary set by ARC-AGI founder François Chollet is: frameworks developed specifically for ARC are still not allowed; common configurations that all API users can access may be considered. He believes that the settings used by each party and how much money was spent should be made public.

Once vendors' built-in memory and context management are allowed, the comparison by ARC-AGI-3 will no longer be just about the models, but will also include the engineering capabilities behind each API.

举报 Correction/Report
Correction/Report
Submit
Add Library
Visible to myself only
Public
Save
Choose Library
Add Library
Cancel
Finish