simular-ai

Agent-S: Control Desktop GUIs with AI and UI-TARS

Agent S is a Python framework that uses an AI agent to operate desktop apps through screenshots, mouse and keyboard actions on macOS, Windows, and Linux. It recommends ByteDance’s UI-TARS-1.5-7B for grounding and supports multiple model providers for task planning.

ActivePythonApache-2.0AI AgentsAutomationFramework
Stars
12.6k
Forks
1.6k
Stars, last 30 days
+315
Last commit
9 days ago
Contributors
26

Our take

Activerecent releases

Health score

51/100

How it is scored
Activity
44
Community
59
Issues
53
Pull requests
46

+315 stars in the last 30 days

Agent S is an actively maintained research framework with broad platform support, though its configurable model stack and recent release history mean teams should validate setup and maintenance needs before relying on it in production.

Good fit if

  • You want to build or evaluate an agent that interacts with real desktop interfaces on macOS, Windows, or Linux.
  • You can configure a main model and a grounding endpoint, and want the option to use ByteDance UI-TARS-1.5-7B.
  • You need a Python SDK or CLI and can work under the Apache-2.0 license.

Look elsewhere if

  • You need a hosted, ready-to-run service rather than configuring model providers and a grounding service.
  • Your target setup requires multiple monitors, which the README does not recommend.
  • You cannot safely permit optional local execution of arbitrary Python and Bash code.
All health signals
Last commit2026-10-08 (9 days ago)
Commits, last 90 days6
Releases, last 12 months1 (latest v0.3.2, 2025-12-16)
Contributors26 (top contributor: 54% of commits)
Issues closed, last 90 days2
Pull requests merged, last 90 days2
Project age2 years

Checked on 2026-10-18 with the GitHub API.

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

13,017

Forks

1,522

Change since Nov 21, 2026: +4 stars·+1 forks

Overview

Agent S is a Python framework for computer-use agents: it takes a natural-language task, observes a real desktop GUI, and acts through mouse, keyboard, and scrolling rather than app-specific APIs. It runs on macOS, Windows, and Linux, and is intended for researchers and developers building or testing agents that work across ordinary desktop applications.

The current Agent S3 setup pairs a main generation model with a grounding model that maps screen content to actions. The README recommends ByteDance’s UI-TARS-1.5-7B for grounding, while allowing other providers and custom endpoints. This makes the framework useful for experiments and self-managed automation, but it requires configuring model access and a grounding service.

Key Features

  • Operates applications through screenshots and mouse, keyboard, and scroll actions.
  • Supports macOS, Windows, and Linux, with a single-monitor setup recommended.
  • Provides the AgentS3 Python SDK and an agent_s command-line interface.
  • Separates task generation from screen grounding, with configurable providers, models, endpoints, and screen dimensions.
  • Includes optional reflection and trajectory settings for agent runs.
  • Offers an optional local coding environment for Python and Bash execution alongside GUI actions.
  • Documents deployment with OSWorld and cites research papers for the Agent S generations.

Use Cases

  • Researchers can evaluate computer-use agents on desktop tasks and benchmarks such as OSWorld.
  • Developers can prototype cross-platform GUI workflows where applications lack suitable APIs.
  • Automation engineers can combine GUI interaction with local code for file, spreadsheet, or text-processing tasks.
  • Teams studying agent planning or grounding can swap model providers and endpoints while using the same framework.

What you need

Detected in the repository

  • Python (from setup.py)
  • A test suite and automated checks on GitHub Actions

License in plain words

Apache-2.0permissive

  • Commercial use: yes
  • Modify and redistribute: yes
  • You must keep: the license, the NOTICE file and a note of your changes
  • Share your changes: no
  • Includes an explicit patent grant from the contributors:

A summary, not legal advice: the LICENSE file is what applies.

Getting Started

Install the package:

pip install gui-agents

The CLI also requires a configured main model and grounding provider. See the README for setup instructions and examples.

Alternatives

  • open-codex-computer-use: It exposes desktop automation through an MCP service and uses accessibility-based controls, rather than Agent-S's screenshot-driven agent framework.
ProjectLanguageLicenseStarsStatus
Agent-SPythonApache-2.012.6kActive
open-codex-computer-useSwiftMIT2.3kNew

Considerations

  • The project is active, with recent commits and contributions from multiple contributors, but its latest listed release is from December 2025. Recent issue and pull-request closure activity is limited, so account for possible delays when assessing support needs.
  • This is a framework, not a turnkey agent endpoint. You must configure model credentials and a grounding model endpoint; the README recommends UI-TARS-1.5-7B and describes other supported providers.
  • The README recommends a single-monitor setup. It also warns that enabling the local coding environment allows generated Python and Bash to run with the current user's permissions.
  • The repository is licensed under Apache-2.0. Review that license and any separate terms for the models or hosted services you choose.

Found this useful?

Share it with someone who would like Agent-S.

Comparisons