# ddupont808/GPT-4V-Act

AI agent using GPT-4V(ision) capable of using a mouse/keyboard to interact with web UI

Repository: https://github.com/ddupont808/GPT-4V-Act
Canonical: https://ross.abutalabs.com/products/gpt-4v-act
Language: JavaScript
License Family: other
Last push: 2024-12-09T17:48:09+00:00

## Health v2 (maintenance only)
Score: 27/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 35, longevity 74
- inputs: {"age_days": 1047, "days_push": 632, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1058, forks 100 (observed 2026-08-28T04:03:25.093443+00:00)

## What it is
GPT-4V-Act is a multimodal AI agent that combines GPT-4V(ision) with a Chromium browser, using Set-of-Mark prompting and a DOM auto-labeler to control a web UI via mouse and keyboard actions like a human operator. It is a JavaScript/Node.js demo application aimed at UI accessibility, workflow automation, and automated UI testing.

## Use cases
- automate web tasks by describing them in natural language
- have an AI agent click and type in a browser like a human
- build accessibility tools that operate any UI
- automated UI testing with a vision-language model
- experiment with set-of-mark prompting on web screenshots

## When to choose
- you want a GPT-4V-based browser automation agent demo
- you're researching multimodal web agents or set-of-mark prompting
- you need a starting point for accessibility or UI-testing agents

## When to avoid
- you need a production-ready or maintained tool (no license, partial features, successor project exists)
- you need scrolling, special keycodes, or multi-step task memory
- you want a headless scraping or RPA solution without LLM costs

## Facets
- artifact type: application
- maturity: experimental
- function: agent-framework, llm-inference, workflow-automation, browser-extension, computer-vision
- domain: artificial-intelligence, web-development, accessibility
- platform: cross-platform
- tags: gpt-4v, set-of-marks, web-agent, ui-automation, multimodal, browser-agent, ai-agents, automation, nodejs, web-server

## Member repositories
- ddupont808/GPT-4V-Act (main) score 27

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:25.093443+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:57:45.907208+00:00, confidence not recorded.
  - readme: https://github.com/ddupont808/GPT-4V-Act (fetched 2026-08-28T04:03:25.093443+00:00, sha 82598705bbe0)
- Data as of 2026-08-30T08:39:29.467469+00:00.