Open-Interface: Control Your Computer with GPT-4o

Overview

Open Interface is a desktop computer-control agent built around OpenAI GPT-4o. It turns a natural-language request into a sequence of keyboard and mouse actions, then sends screenshots back to the model so it can assess progress and course-correct. The project also supports Google Gemini and custom LLMs with an OpenAI-style API.

It targets people who want an LLM to operate ordinary desktop applications rather than just answer questions or generate code. The project provides apps for macOS, Linux, and Windows, and can also be run as a Python script.

Key Features

• Converts user requests into actions for the computer through an LLM backend.
• Simulates keyboard and mouse input to carry out those actions.
• Uses screenshots to check progress and adjust the next steps.
• Supports OpenAI GPT-4o, Google Gemini, and configurable OpenAI-style custom backends.
• Provides downloadable desktop builds for macOS, Linux, and Windows.
• Includes a Stop button and a screen-corner cursor gesture to interrupt execution.

Use Cases

• A user can ask the agent to solve a Wordle puzzle by interacting with the game in a browser.
• Someone preparing a meal plan can ask it to create a document in Google Docs.
• A developer can ask it to create a web app through actions in desktop software.
• Users can delegate straightforward, multi-step tasks in familiar desktop applications when they are willing to supervise an LLM controlling their computer.

Getting Started

For a Python setup, install Python 3.12 or newer, then run:

git clone https://github.com/AmberSahdev/Open-Interface.git
cd Open-Interface
pip install -r requirements.txt
python app/app.py

The app needs an LLM API key. See the README for API setup, platform-specific permissions, and desktop downloads from Releases.

Alternatives

• open-codex-computer-use: It exposes accessibility-based desktop controls as an MCP service for compatible agents, rather than providing a standalone GPT-4o-driven desktop app.
• Agent-S: It is a Python agent framework that recommends UI-TARS for visual grounding, rather than a desktop app centered on GPT-4o.
• UFO: It focuses on Windows desktop automation and coordinating agents across devices, rather than offering a cross-platform desktop app.
• UI-TARS-desktop: It is a vision-language-model desktop agent with local and remote operators, rather than an app centered on GPT-4o and configurable LLM backends.

Considerations

• The README calls out inaccurate spatial reasoning, unreliable interaction with tabular layouts such as Excel and Google Sheets, and difficulty navigating complex GUI-heavy applications.
• The app relies on screenshots and simulated input, so it needs screen-recording and accessibility permissions on macOS. It can only see the primary display when multiple monitors are in use.
• Requests require an external LLM backend and API key. The README estimates a per-request cost and notes that complex tasks may make multiple backend calls.
• Maintenance is quiet: there have been no commits in the last 90 days and no release in over a year. Contribution activity is highly concentrated in one person.
• The GPL-3.0 license may impose obligations when distributing modified versions; review its terms before incorporating the project into a product.