Agent-S: Automate Desktop Tasks Through a GUI Agent

Use at your own risk

OSRepos shares public repositories for knowledge and discovery only. Any installation, execution, configuration, or use of code from these repositories is the user's own responsibility. Always review the repository, source code, dependencies, licenses, and security implications before running or installing anything. OSRepos is not responsible for issues, damages, or losses resulting from third-party repositories.

Overview

Agent-S is a Python framework for computer-use agents. It turns a natural-language instruction into actions in a graphical interface, using the screen as input and mouse, keyboard, and scrolling actions to work across desktop and web applications.

It is aimed at developers and researchers who want to experiment with GUI automation without writing a separate integration for each application. The framework supports macOS, Windows, and Linux, and connects to external language and grounding models. It is a better fit for hands-on development and evaluation than for users seeking a ready-to-run hosted service.

Key Features

Interacts with applications through screenshots and GUI actions rather than app-specific APIs.
Provides a Python SDK and an agent_s command-line interface.
Supports macOS, Windows, and Linux.
Connects to model providers including OpenAI, Anthropic, Gemini, Open Router, and vLLM.
Uses a separate grounding model to translate intended actions into screen coordinates.
Offers an optional local coding environment for Python and Bash execution.
Includes deployment instructions for OSWorld.

Use Cases

GUI-agent researchers can test computer-use methods against desktop environments and benchmarks.
Developers can automate repetitive workflows in applications that lack suitable APIs.
Teams can prototype cross-application automation while retaining a human-readable task interface.
Users with data or file tasks can optionally combine GUI interaction with local code execution, in trusted environments.

Project Facts

Language: Python
License: Apache-2.0
Stars: 12.5k
Forks: 1.5k
Topics: agent-computer-interface, ai-agents, computer-automation, computer-use, computer-use-agent, cua, grounding, gui-agents, in-context-reinforcement-learning, memory, mllm, planning, retrieval-augmented-generation
Archived: No

Getting Started

Install the package with:

pip install gui-agents

The agent also needs configured model and grounding endpoints. See the README for setup, configuration, and usage details.

Alternatives

UI-TARS-desktop: UI-TARS-desktop provides local and remote computer and browser operators, while Agent-S is a Python framework for screenshot-based desktop interaction.
UFO: UFO focuses on Windows automation and coordinating agents across devices, while Agent-S provides a Python framework for desktop interaction.

Considerations

The framework requires API credentials or accessible endpoints for its main and grounding models; the README lists supported providers and configuration options.
The grounding model and its screen-coordinate dimensions must be configured, and Tesseract is an additional prerequisite noted by the project.
The project recommends single-monitor setups. GUI actions can have real effects on the machine being controlled.
Enabling the optional local coding environment allows generated Python and Bash code to run with the current user's permissions. Use it only with trusted inputs, preferably in a sandbox.
The repository is a framework, not a turnkey hosted product. Users should expect to configure models and their computer environment; consult the repository for current instructions.