OmniParser: Turn Screenshots into GUI Elements

Overview

OmniParser converts user interface screenshots into structured elements that a vision-based agent can interpret. By identifying screen regions and describing icons, it helps an agent connect a requested action to the relevant part of an interface.

It is intended for developers building GUI agents that rely on visual input rather than accessibility trees or application-specific integrations. The repository includes parsing models and a Gradio demo, while OmniTool provides a separate Windows VM workflow for testing computer-use agents.

Key Features

• Detects interactive regions in UI screenshots.
• Identifies and describes icons to make screen elements easier for vision models to interpret.
• Supports element-level interactability predictions in the V1.5 release.
• Provides V2 model checkpoints through Hugging Face.
• Includes a notebook with parsing examples and a Gradio demo.
• Can be paired with OmniTool to control a Windows 11 VM using supported vision models.

Use Cases

• GUI-agent developers can parse screenshots into actionable regions when building visual computer-use workflows.
• Researchers can experiment with screenshot understanding and visual grounding using the released models and examples.
• Teams evaluating agents across desktop interfaces can use the parser as a visual perception component.
• Developers exploring different vision models can pair OmniParser with OmniTool to test Windows 11 computer-use workflows.

Getting Started

Clone the repository, create the documented Python environment, and install dependencies:

conda create -n omni python==3.12
conda activate omni
pip install -r requirements.txt

Model weights are also required. See the README for current download instructions and demo details.

Alternatives

• UI-TARS-desktop: UI-TARS-desktop is a full desktop agent that interprets screens and executes tasks, while OmniParser focuses on parsing screenshots into actionable UI elements.

Considerations

• Setup requires downloading model weights in addition to installing Python dependencies. The README's V2 setup includes a detector weight from a Hugging Face pull request, so check the repository for updated instructions.
• The project parses screenshots, but it is not by itself a complete GUI agent. An agent or other controller must decide and execute actions.
• License information differs across components: the repository metadata lists CC-BY-4.0, while the README identifies separate licenses for model components and earlier detectors. Review the relevant terms before use.
• The project is actively maintained according to the supplied repository metadata, but its README describes some tooling and documentation as still in progress.