Alibaba's GUI Agent achieves 3rd generation: Qwen can now switch between mobile, desktop, and command line interfaces

Perceive Beating AI News Flash: Alibaba has launched a new GUI-based Intelligent Agent model called Qwen-UI-Agent. This agent can directly view the screen, click buttons, input text, operate mobile phones, computers, and web pages. When encountering tasks suitable for command-line processing, it can also run CLI directly, without needing to rely on the mouse for everything.

Alibaba has been working on GUI Agents for over two years. The Mobile-Agent in 2024 mainly allowed the model to view mobile screenshots and then control clicks and swipes. Subsequent models like GUI-Owl and MAI-UI began to train GUI operations directly into the model and expanded to computers. The Qwen-UI-Agent has integrated mobile phones, computers, web pages, DeepSearch, and command-line functionalities into a single agent.

For example, when checking high-speed rail information, calculating the time to reach the office, and then scheduling a meeting on DingTalk, the agent can seamlessly navigate through multiple apps. During data analysis, it can search for information on the internet, process data using the command line, and finally generate Excel, PPT, and Word documents.

The core model is derived from Qwen3.5-27B. In Alibaba's in-house real-world evaluation MobileWorld-Real, it achieved a score of 92.2%, surpassing other models such as GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro evaluated in the same test.