web-infra-dev/midscene

Health v2 · maintenance only

Adoption not part of the score

14714 stars · 1134 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Midscene.js is an open-source TypeScript SDK that acts as a GUI agent for E2E testing and UI automation, using multimodal vision models to plan and operate interfaces from natural-language instructions. It works across web (Playwright/Puppeteer), Android, iOS, HarmonyOS, and desktop apps, reaching elements that DOM/accessibility-tree-based tools cannot, such as canvas and native controls.

Use cases

• write e2e tests with natural language instead of css selectors
• automate web forms in playwright or puppeteer using ai vision
• test android and ios apps on real devices with plain english commands
• automate desktop applications on macos windows and linux
• interact with canvas elements and unlabeled buttons that selectors cannot reach
• run yaml-scripted ui automation flows
• automate any interface from screenshots

When to choose

• your UI has fragile or missing selectors, canvas rendering, or cross-origin frames
• you want one natural-language automation API across web, mobile, and desktop
• you need AI-driven E2E tests integrated with Playwright or Puppeteer
• you want to automate native apps without accessibility-tree access

When to avoid

• you need fast, deterministic, low-cost tests and stable selectors already work
• you cannot send screenshots to external or self-hosted vision models
• your tests require strict reproducibility without AI variability
• you need offline automation with no LLM access

Facets

library · maturity active

e2e-testing testing computer-vision agent-framework workflow-automation testing developer-tools artificial-intelligence web-development cross-platform windows cli gui-agent vision-driven-automation ui-testing playwright puppeteer multimodal-llm yaml-automation mobile-automation ai-testing automation nodejs web-server android ios macos linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="web-infra-dev/midscene")