# web-infra-dev/midscene

GUI Agent for E2E Testing

Repository: https://github.com/web-infra-dev/midscene
Canonical: https://ross.abutalabs.com/products/midscene
Homepage: https://midscenejs.com
Language: TypeScript
License: MIT
License Family: permissive
Topics: testing, browser-use, computer-use, gpt-operator, javascript, phone-use, e2e-testing, gui-agent
Last push: 2026-08-26T15:34:32+00:00

## Health v2 (maintenance only)
Score: 86/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 99, release rhythm 87, longevity 55
- inputs: {"age_days": 771, "days_push": 7, "days_rel": 6, "gap_med": 2.0, "n_releases_24m": 187}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 14714, forks 1134 (observed 2026-08-28T04:11:07.526368+00:00)

## What it is
Midscene.js is an open-source TypeScript SDK that acts as a GUI agent for E2E testing and UI automation, using multimodal vision models to plan and operate interfaces from natural-language instructions. It works across web (Playwright/Puppeteer), Android, iOS, HarmonyOS, and desktop apps, reaching elements that DOM/accessibility-tree-based tools cannot, such as canvas and native controls.

## Use cases
- write e2e tests with natural language instead of css selectors
- automate web forms in playwright or puppeteer using ai vision
- test android and ios apps on real devices with plain english commands
- automate desktop applications on macos windows and linux
- interact with canvas elements and unlabeled buttons that selectors cannot reach
- run yaml-scripted ui automation flows
- automate any interface from screenshots

## When to choose
- your UI has fragile or missing selectors, canvas rendering, or cross-origin frames
- you want one natural-language automation API across web, mobile, and desktop
- you need AI-driven E2E tests integrated with Playwright or Puppeteer
- you want to automate native apps without accessibility-tree access

## When to avoid
- you need fast, deterministic, low-cost tests and stable selectors already work
- you cannot send screenshots to external or self-hosted vision models
- your tests require strict reproducibility without AI variability
- you need offline automation with no LLM access

## Facets
- artifact type: library
- maturity: active
- function: e2e-testing, testing, computer-vision, agent-framework, workflow-automation
- domain: testing, developer-tools, artificial-intelligence, web-development
- platform: cross-platform, windows, cli
- tags: gui-agent, vision-driven-automation, ui-testing, playwright, puppeteer, multimodal-llm, yaml-automation, mobile-automation, ai-testing, automation, nodejs, web-server, android, ios, macos, linux

## Member repositories
- web-infra-dev/midscene (main) score 86

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:07.526368+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:07:16.551052+00:00, confidence not recorded.
  - readme: https://github.com/web-infra-dev/midscene (fetched 2026-08-28T04:11:07.526368+00:00, sha c11345720af6)
  - homepage: https://midscenejs.com (fetched 2026-08-29T08:05:47.434927+00:00, sha fa1b5d504d44)
  - site_page: https://midscenejs.com/changelog (fetched 2026-08-29T08:05:47.468724+00:00, sha c9913b79e8b9)
  - site_page: https://midscenejs.com/introduction (fetched 2026-08-29T08:05:47.473728+00:00, sha e31c1a986874)
- Data as of 2026-08-30T08:39:29.467469+00:00.