microsoft/OmniParser

Health v2 · maintenance only

Adoption not part of the score

25310 stars · 2223 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

OmniParser is a screen parsing tool from Microsoft that converts UI screenshots into structured, understandable elements to ground vision-language models for GUI automation. It includes detection models for interactable regions and icons, enabling pure vision-based computer-use agents.

Use cases

• parse ui screenshots into structured elements
• build a computer use agent that controls a gui
• ground llm actions to screen regions
• detect clickable icons and elements in screenshots
• automate desktop apps with vision models
• create training data for gui agents

When to choose

• you need vision-based ui element detection and grounding for screen automation
• you want to drive a gui agent with models like GPT-4V, Qwen-VL, or Claude computer use
• you need state-of-the-art screen grounding benchmarks performance

When to avoid

• you need DOM/accessibility-tree based automation rather than pure vision
• you need a production-ready end-user automation product rather than models and notebooks
• you cannot run GPU inference for the detection models

Facets

library · maturity active

computer-vision machine-learning ocr agent-framework image-processing computer-vision artificial-intelligence large-language-models python cross-platform gui-parsing screen-understanding ui-element-detection vision-language-models computer-use-agents screenshot-parsing ai-agents gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/OmniParser")