CIRCL/AIL-framework

Health v2 · maintenance only

Adoption not part of the score

1378 stars · 288 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

AIL framework is an open-source Python platform for collecting, crawling, processing, and analyzing unstructured data from the clear web, Tor, I2P, chat platforms, files, and external feeds. Originally developed at CIRCL, it helps analysts extract, detect, correlate, and share intelligence such as leaked credentials and threat information.

Use cases

• monitor dark web marketplaces and Tor hidden services for leaked data
• detect leaked credentials and sensitive information mentions
• retro-hunt historical data with keywords, regex, and YARA rules
• crawl and analyze chat platform communications for threat intelligence
• correlate extracted indicators and export them to MISP
• process unstructured text with OCR, decoding, and tagging pipelines

When to choose

• you need continuous monitoring of leaks, pastes, chats, or hidden services for security intelligence
• you want an extensible Python framework with detection, correlation, and MISP integration
• you run a SOC or CERT and need automated retro-hunting with YARA and regex

When to avoid

• you need a simple one-off web scraper without analysis workflows
• you cannot self-host a multi-component Python framework with crawlers and feeders
• your focus is structured data analytics rather than unstructured threat intelligence

Facets

framework · maturity active

web-scraping nlp search-engine monitoring security data-science ocr etl security osint crawlers privacy python self-hosted threat-intelligence leak-analysis tor-crawler yara misp-integration dark-web-monitoring retro-hunting natural-language-processing linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="CIRCL/AIL-framework")