Codex can now directly operate an iPhone, as developers have open-sourced the entire toolkit.

Beating AI News Flash: Zhong Erxin, former product manager of Doubao's desktop client, has open-sourced iPhone Use, allowing Codex to directly operate a real iPhone. Users only need to issue tasks in natural language, and Codex can open apps, read pages, tap, swipe, input Chinese, and even search and organize lists. The entire operation process can be viewed in real time in the Codex sidebar.

The project uses Appium's WebDriverAgent (WDA) under the hood, a mobile automation tool based on Apple's XCTest testing framework. Developers first install WDA on the iPhone via Xcode, then send Codex instructions to the phone through an MCP service on the Mac, where WDA executes actions such as taps and text input. No jailbreak is required, and there is no need to deploy an Appium Server separately.

To reduce the overhead of AI repeatedly viewing screenshots, iPhone Use prioritizes reading interface control information provided by the system, directly identifying button text, position, and state. When it encounters controls that cannot be recognized or are obscured, it falls back to screenshots and lets the model determine the tap coordinates. It also supports batch execution of consecutive actions to reduce the number of model calls. The real-time view in the sidebar is obtained through WDA's independent screen stream, eliminating the need for repeated screenshots.

At present, iPhone Use's official tutorial still requires connecting the phone via USB, and first-time use requires signing and installing WDA through Xcode. Apple supports wireless debugging, and WDA can also be accessed over the network, but the project has not yet provided a complete wireless connection configuration solution.