BuilderIO/gpt-crawler
Crawl a site to generate knowledge files to create your own custom GPT from a URL observed · 2026-08-28
Health v2 · maintenance only
31/100
- Activity 30
- Release rhythm 8
- Longevity 73
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 1024
- days_rel: 588
- days_push: 422
- n_releases_24m: 1
Adoption not part of the score
22393 stars · 2368 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
A Node.js/TypeScript crawler that scrapes one or more websites and generates knowledge files (JSON) for creating custom GPTs or OpenAI assistants. It can run locally, in Docker, or as an API.
Use cases
- create a custom GPT from a documentation site
- crawl a website to generate knowledge files for OpenAI assistants
- turn any site into a chatbot knowledge base
- scrape docs into JSON for RAG ingestion
- build a support bot from my website content
When to choose
- you want to build a custom GPT or assistant grounded in a specific site's content
- you need a quick, configurable site crawler that outputs structured JSON
When to avoid
- you need a full RAG pipeline with embeddings and vector storage
- you need large-scale, distributed crawling with scheduling
Facets
cli-tool · maturity active
web-scraping rag data-generation cli artificial-intelligence large-language-models crawlers developer-tools cli cross-platform custom-gpt openai knowledge-files crawler playwright nodejs docker
8 sources
- readme: https://github.com/BuilderIO/gpt-crawler · fetched 2026-08-28 · 0a5648a230f6
- homepage: https://www.builder.io/blog/custom-gpt · fetched 2026-08-29 · e3d75f06d401
- site_page: https://www.builder.io/c/docs/developers · fetched 2026-08-29 · 10bb2aa68f1d
- site_page: https://www.builder.io/m/about-us · fetched 2026-08-29 · 686c281d03b7
- site_page: https://www.builder.io/c/docs/get-started-fusion · fetched 2026-08-29 · e2133b056fc4
- site_page: https://www.builder.io/c/docs/get-started-publish · fetched 2026-08-29 · 736c3720b918
- site_page: https://www.builder.io/pricing · fetched 2026-08-29 · 0fb592ca72d9
- site_page: https://www.builder.io/integrations · fetched 2026-08-29 · e288e66e5b1e
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| BuilderIO/gpt-crawler | main | 31 |
For agents
markdown · JSON · MCP: product_card(name="BuilderIO/gpt-crawler")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem