oxylabs/ai-crawler-py
Some data may be outdated — see provenance labels.
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt. observed · 2026-09-24
Health v2 · maintenance only
10/100
- Activity 90
- Release rhythm 35
- Longevity 28
Flags: no_releases archived no_license
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 398
- days_rel: n/a
- days_push: 60
- n_releases_24m: 0
Adoption not part of the score
3264 stars · 12 forks observed · 2026-09-24
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
A Python client library for Oxylabs AI Studio's AI-Crawler, a commercial service that crawls websites from a starting URL, uses natural language prompts to find relevant pages, and extracts structured JSON or Markdown data. It wraps a paid API (with a free trial of 1,000 credits) rather than performing crawling locally.
Use cases
- crawl a website and extract data using a natural language prompt
- find relevant pages on a domain without writing custom scrapers
- extract structured JSON from websites using a schema
- convert website content to markdown for LLM pipelines
- feed crawled web data into AI agents or automation workflows
When to choose
- you want prompt-driven crawling without building or maintaining scrapers
- you need schema-based structured extraction from websites
- you already use Oxylabs AI Studio or want managed crawling infrastructure
When to avoid
- you need a fully free, self-hosted crawler with no API costs
- you want full control over crawl logic, politeness, or proxies
- your project cannot depend on a third-party commercial service
Facets
library · maturity active
web-scraping nlp etl data-generation crawlers artificial-intelligence python cloud ai-crawler web-crawler ai-scraping oxylabs api-client low-code commercial-service prompt-driven data-engineering automation web-server
4 sources
- readme: https://github.com/oxylabs/ai-crawler-py · fetched 2026-09-24 · 7c2c2a5e4c10
- homepage: https://aistudio.oxylabs.io/apps/crawl · fetched 2026-08-29 · 816b67fd1a58
- site_page: https://aistudio.oxylabs.io/pricing · fetched 2026-08-29 · 9226846bdd8d
- site_page: https://aistudio.oxylabs.io/releases · fetched 2026-08-29 · 2807647bbec5
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| oxylabs/ai-crawler-py | main | 10 |
For agents
markdown · JSON · MCP: product_card(name="oxylabs/ai-crawler-py")
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem