Alibaba has unveiled Qwen-UI-Agent, a new foundation model for GUI-based AI agents built to interact with real-world devices. It works across smartphones, PC desktops, web browsers, and deep-search environments, allowing AI to understand on-screen interfaces and perform digital tasks through interactions that closely resemble those of a human user.
Unlike traditional AI models that rely on APIs or predefined commands, Qwen-UI-Agent can see and understand what’s displayed on a screen and perform actions accordingly. It can navigate apps, operate software, and execute tasks across different device types, effectively serving as a general-purpose digital device operator.
Benchmark Performance: Top-Tier Results Across the Board
According to Alibaba, Qwen-UI-Agent has achieved state-of-the-art or near-state-of-the-art results across multiple industry benchmarks:
| Benchmark | Environment | Score | Significance |
|---|---|---|---|
| MobileWorld | Mobile | 82.1% | Strong performance in simulated mobile tasks |
| MobileWorld-Real | Real mobile devices | 92.2% | Excellent real-world device operation |
| AndroidDaily | Mobile | 97.5% | Near-perfect score on daily Android tasks |
| OSWorld-Verified | Desktop | 79.5% | Surpassed GPT-5.5, Gemini 3.1 Pro, and others |
| WebArena | Web browser | 73.6% | Ranked first among all compared models |
| ScreenSpot-Pro | Interface localization | 81.5% | Precise element identification across screens |
Alibaba states that the model has reached or exceeded the performance of leading international models—including GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro—while retaining the general-purpose and agent capabilities of the underlying Qwen foundation model.
Real-Device Training: Learning on Physical Hardware
A major differentiator for Qwen-UI-Agent is its emphasis on real-device training. Alibaba built a dedicated testing cluster containing:
-
100+ physical smartphones
-
150+ applications
This setup allows the model to move beyond simulated environments and learn how to perform tasks on actual hardware, making its behavior more reliable and practical for real-world deployment.
Built-In Security and User Control
Security is deeply integrated into the system. Qwen-UI-Agent can automatically reject high-risk requests, while sensitive actions—including payments, data deletion, and permission authorization—require explicit user confirmation before execution.
This dual-layer safety approach ensures that the agent remains helpful without becoming a liability.
CLI Integration and Batch Execution
The model supports a combination of GUI interactions and command-line interface (CLI) operations. It can also generate batches of actions simultaneously.
Alibaba notes that nearly half of all commands used during computer-based tasks can be handled through the command line, helping reduce unnecessary interaction steps and improve overall execution efficiency.
Continuous Learning and Scalable Training
Qwen-UI-Agent is trained through online reinforcement learning with trajectories exceeding 100 steps. More than 10,000 concurrent environments are used for training and iteration, while an AutoResearch-style AI-driven data pipeline continuously feeds new task information back into the development process.
This scalable training infrastructure allows the model to adapt and improve over time without manual intervention.
Cross-Device Coordination and DeepSearch Integration
Built on Alibaba’s Harness framework, Qwen-UI-Agent is designed to provide proactive assistance across devices. It can:
-
Coordinate tasks between smartphones and computers
-
Allow users to hand off an ongoing task between devices
-
Work together with DeepSearch, combining web research with direct GUI interaction
This cross-device capability makes it a truly versatile assistant for complex, multi-step workflows.
Solving the API Gap: The Broader Vision
The broader goal of Qwen-UI-Agent is to tackle one of the biggest limitations facing today’s AI agents: software that does not provide an open API.
Instead of requiring developers to modify an application or build a dedicated integration, the AI can simply interpret the screen and interact with the software through its existing interface.
If successful, this approach could significantly expand the range of tasks AI agents can handle—bringing automated interaction to a much wider range of traditional applications and digital environments, without requiring any changes to the software itself.
Bottom Line
Qwen-UI-Agent marks a meaningful advance toward general-purpose AI capable of operating any digital interface. Its combination of strong benchmark results, real-device training, integrated security, and cross-platform coordination directly addresses the core shortcomings of today’s AI agents. By working with applications that lack open APIs, it opens the door to automation, accessibility, and digital assistance across a wide range of industries.