One-Touch Rule Resets the Score: Zapier Launches AI Automation New Evaluation, Top Model Score Also Halved

According to Dynamic Insights monitoring, the workflow automation platform Zapier has partnered with the evaluation agency Artificial Analysis to launch the standalone SaaS Workflow Automation Evaluation Benchmark, AutomationBench-AA.

The benchmark is based on real anonymized data on the Zapier platform and has designed a private test set containing 657 multi-step tasks and 40 simulated SaaS software environments (including Gmail, Slack, Salesforce, Jira, etc.) for unbiased third-party evaluation.

An essential aspect of the test is the introduction of security and compliance guardrails. The large models must complete operations without violating any predefined corporate business rules. If any compliance violations are triggered during the execution, the final score of the task will be directly zeroed. In the initial evaluation, large models generally saw a steep decline in scores due to compliance guardrail penalties, with none achieving a score above fifty percent.

Notably, Claude Fable 5, which introduced the "High Difficulty Task Rollback to Opus 4.8 (approx. 18% share)" security downgrade mechanism, achieved 73% of its objectives in pure task steps but only received a comprehensive score of 48.6%. Gemini 3.5 Flash and GPT-5.5 also accomplished nearly seventy percent (68% and 67%, respectively) of their objectives but saw their final scores drop to 42.6% and 42.1%, respectively. The open-source leader GLM-5.2 only scored 27.8%.

The tests indicate that how models adhere to security guardrails in automatic operations has become a critical challenge for practical implementation.