{"name":"OmniParser: A Vision-Based Tool for GUI Agent Screen Parsing","description":"OmniParser is a comprehensive tool developed by Microsoft for parsing user interface screenshots into structured, understandable elements. It significantly enhances the ability of vision-based models, such as GPT-4V, to generate accurate actions grounded in specific regions of a GUI. This project aims to advance pure vision-based GUI agents by providing robust screen parsing capabilities.","github":"https://github.com/microsoft/OmniParser","url":"https://osrepos.com/repo/microsoft-omniparser","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/microsoft-omniparser","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/microsoft-omniparser.md","json":"https://osrepos.com/repo/microsoft-omniparser.json","topics":["Jupyter Notebook","AI","Computer Vision","GUI Automation","LLM","Screen Parsing","Agent","Microsoft"],"keywords":["Jupyter Notebook","AI","Computer Vision","GUI Automation","LLM","Screen Parsing","Agent","Microsoft"],"stars":null,"summary":"OmniParser is a comprehensive tool developed by Microsoft for parsing user interface screenshots into structured, understandable elements. It significantly enhances the ability of vision-based models, such as GPT-4V, to generate accurate actions grounded in specific regions of a GUI. This project aims to advance pure vision-based GUI agents by providing robust screen parsing capabilities.","content":"## Introduction\nOmniParser, developed by Microsoft, is a powerful tool designed to parse user interface screenshots into structured and easily understandable elements. This capability significantly enhances the performance of vision-based models, including GPT-4V, by allowing them to generate actions that are precisely grounded in corresponding regions of a graphical user interface. OmniParser is a crucial step towards building pure vision-based GUI agents, enabling advanced automation and interaction with digital interfaces. It has seen continuous development, including the release of V2 and OmniTool, which allows control of Windows 11 VMs with various large language models.\n\n## Installation\nTo get started with OmniParser, follow these steps to clone the repository and set up your environment. Ensure you have `conda` installed for environment management.\n\nFirst, clone the repository:\n\nbash\ngit clone https://github.com/microsoft/OmniParser.git\ncd OmniParser\n\n\nNext, create and activate a new `conda` environment, then install the required dependencies:\n\nbash\nconda create -n \"omni\" python==3.12\nconda activate omni\npip install -r requirements.txt\n\n\nFinally, download the necessary V2 model checkpoints:\n\nbash\n# download the model checkpoints to local directory OmniParser/weights/\nfor f in icon_detect/{train_args.yaml,model.pt,model.yaml} icon_caption/{config.json,generation_config.json,model.safetensors}; do huggingface-cli download microsoft/OmniParser-v2.0 \"$f\" --local-dir weights; done\nmv weights/icon_caption weights/icon_caption_florence\n\n\n## Examples\nOmniParser provides several ways to explore its capabilities. You can find simple examples demonstrating its core functionalities within the `demo.ipynb` Jupyter Notebook.\n\nFor an interactive experience, a Gradio demo is available. To run it, simply execute the following command in your activated environment:\n\nbash\npython gradio_demo.py\n\n\nAdditionally, you can explore the official [HuggingFace Space demo](https://huggingface.co/spaces/microsoft/OmniParser-v2) for OmniParser V2 to see it in action.\n\n## Why Use OmniParser\nOmniParser stands out as a critical tool for anyone working with GUI automation, AI agents, or computer vision applications. Its ability to accurately parse screen elements into a structured format makes it invaluable for:\n\n*   **Enhancing AI Agents**: It empowers large language models and vision models to understand and interact with graphical user interfaces more effectively.\n*   **Pure Vision-Based Interaction**: It moves towards a future where AI agents can operate purely based on visual input, mimicking human interaction.\n*   **Robust Performance**: OmniParser has achieved state-of-the-art results on benchmarks like Screen Spot Pro and Windows Agent Arena.\n*   **Continuous Development**: With ongoing updates like OmniParser V2, OmniTool for Windows VM control, and support for various LLMs (OpenAI, DeepSeek, Qwen, Anthropic), it remains at the forefront of GUI parsing technology.\n*   **Detailed Element Detection**: Features like fine-grained icon detection and interactability prediction provide a rich understanding of the UI.\n\n## Links\nExplore OmniParser further through these official resources:\n\n*   [GitHub Repository](https://github.com/microsoft/OmniParser)\n*   [Project Page](https://microsoft.github.io/OmniParser/)\n*   [OmniParser V2 Blog Post](https://www.microsoft.com/en-us/research/articles/omniparser-v2-turning-any-llm-into-a-computer-use-agent/)\n*   [OmniParser V2.0 Models on HuggingFace](https://huggingface.co/microsoft/OmniParser-v2.0)\n*   [OmniParser V1.5 Models on HuggingFace](https://huggingface.co/microsoft/OmniParser)\n*   [HuggingFace Space Demo (V2)](https://huggingface.co/spaces/microsoft/OmniParser-v2)\n*   [arXiv Technical Report](https://arxiv.org/abs/2408.00203)","metrics":{"detailViews":17,"githubClicks":5},"dates":{"published":null,"modified":"2025-12-28T20:01:00.000Z"}}