ddupont808/GPT-4V-Act

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1058 stars · 100 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

GPT-4V-Act is a multimodal AI agent that combines GPT-4V(ision) with a Chromium browser, using Set-of-Mark prompting and a DOM auto-labeler to control a web UI via mouse and keyboard actions like a human operator. It is a JavaScript/Node.js demo application aimed at UI accessibility, workflow automation, and automated UI testing.

Use cases

• automate web tasks by describing them in natural language
• have an AI agent click and type in a browser like a human
• build accessibility tools that operate any UI
• automated UI testing with a vision-language model
• experiment with set-of-mark prompting on web screenshots

When to choose

• you want a GPT-4V-based browser automation agent demo
• you're researching multimodal web agents or set-of-mark prompting
• you need a starting point for accessibility or UI-testing agents

When to avoid

• you need a production-ready or maintained tool (no license, partial features, successor project exists)
• you need scrolling, special keycodes, or multi-step task memory
• you want a headless scraping or RPA solution without LLM costs

Facets

application · maturity experimental

agent-framework llm-inference workflow-automation browser-extension computer-vision artificial-intelligence web-development accessibility cross-platform gpt-4v set-of-marks web-agent ui-automation multimodal browser-agent ai-agents automation nodejs web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ddupont808/GPT-4V-Act")