UI-TARS-desktop: Control Computers with a Multimodal AI Agent

Overview

UI-TARS-desktop is a native GUI agent that translates natural-language instructions into computer interactions, using screenshots and vision-language models to interpret interfaces and control mouse and keyboard input. It addresses the gap between AI models and desktop workflows by letting an agent operate applications through their visual interfaces.

The repository also describes the wider Agent TARS stack, but UI-TARS Desktop is the desktop application for local and remote computer and browser operation. It is intended for people exploring GUI automation or integrating visual agents into practical workflows.

Key Features

• Accepts natural-language instructions for GUI tasks.
• Uses screenshots and visual recognition to understand interfaces.
• Controls mouse and keyboard actions.
• Provides local and remote computer operators, as well as browser operators.
• Supports Windows, macOS, and browser environments, as described by the project.
• Displays real-time feedback and status during operation.
• Uses UI-TARS and Seed vision-language model series.

Use Cases

• Automate repetitive desktop settings changes, such as adjusting an application preference, when a user wants to describe the task instead of navigating menus manually.
• Ask an agent to inspect a web page or browser-based service and carry out a multi-step interaction.
• Prototype GUI-agent workflows that combine visual understanding with mouse and keyboard control.
• Evaluate local versus remote computer operation for teams developing or assessing computer-use agents.

Getting Started

See the Quick Start guide for setup and local or remote operator instructions. For the broader Agent TARS CLI, the README documents a separate launch command: npx @agent-tars/cli@latest.

Alternatives

• OpenBot: OpenBot runs AI coworkers in isolated virtual computers, while UI-TARS-desktop operates a user's local or remote computer and browser.
• UFO: UFO focuses on Windows desktop automation and coordinating workflows across devices, while UI-TARS-desktop provides cross-platform GUI and browser operators.

Considerations

• GUI agents can make mistakes when interpreting changing or ambiguous interfaces, so review actions before relying on them for consequential tasks.
• Model access and operator setup may differ between local and remote workflows. Consult the quick-start documentation for requirements and configuration.
• The README covers both UI-TARS Desktop and the separate Agent TARS CLI and Web UI. Check that instructions apply to the desktop application you intend to use.
• The repository is licensed under Apache-2.0.