microsoft/SoM

Health v2 · maintenance only

Adoption not part of the score

1557 stars · 111 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A research toolbox from Microsoft implementing Set-of-Mark (SoM) visual prompting, which overlays numbered spatial marks on images to improve visual grounding in GPT-4V and other large multimodal models. It includes mark-generation code using segmentation models, a GPT-4V demo, and an evaluation benchmark.

Use cases

• overlay numbered marks on images to prompt GPT-4V
• improve visual grounding of multimodal LLMs
• build GUI or web navigation agents with visual prompting
• evaluate GPT-4V on visual reasoning benchmarks
• generate set-of-mark prompts from segmentation models

When to choose

• you want to enhance GPT-4V or other LMMs with spatial grounding on images
• you are building agents that navigate UIs or detect anomalies via multimodal models
• you are reproducing or extending the SoM research

When to avoid

• you need a production-ready end-user application rather than research code
• you do not use large multimodal models or OpenAI APIs
• you need actively maintained tooling with long-term support

Facets

library · maturity maintenance

computer-vision image-processing prompt-engineering machine-learning computer-vision large-language-models artificial-intelligence python cross-platform visual-prompting gpt-4v multimodal-llm set-of-mark image-segmentation research

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/SoM")