# wilpel/caveman-compression

Caveman Compression is a semantic compression method for LLM contexts. It removes predictable grammar while preserving the unpredictable, factual content that defines meaning.

Repository: https://github.com/wilpel/caveman-compression
Canonical: https://ross.abutalabs.com/products/caveman-compression
Language: Python
License Family: other
Last push: 2025-12-03T17:04:50+00:00

## Health v2 (maintenance only)
Score: 41/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 55, release rhythm 35, longevity 20
- inputs: {"age_days": 289, "days_push": 273, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1092, forks 67 (observed 2026-08-28T04:03:32.964752+00:00)

## What it is
A Python tool that compresses text for LLM contexts by stripping predictable grammar while preserving factual content, reducing token usage by 15-58%. It offers three modes: LLM-based (best compression via OpenAI API), NLP-based (free, offline, multilingual via spaCy), and MLM-based (free, offline, predictability-aware).

## Use cases
- reduce token costs when sending long context to an LLM
- compress prompt text to fit more content in a context window
- shrink RAG retrieved documents before passing to a model
- compress verbose documentation for LLM consumption
- offline text compression without an API
- compress multilingual text for LLM contexts

## When to choose
- you pay per token and want 15-58% context reduction
- you need a free, offline compression option via spaCy or masked language models
- you want to fit more retrieved documents into a fixed context window

## When to avoid
- you need byte-perfect lossless reconstruction of original text
- your content is mostly facts with little redundant grammar
- you cannot tolerate LLM-based decompression ambiguity in critical data

## Facets
- artifact type: library
- maturity: active
- function: nlp, llm-inference, compression, prompt-engineering
- domain: large-language-models
- platform: python, cli
- tags: semantic-compression, token-reduction, context-optimization, text-compression, natural-language-processing

## Member repositories
- wilpel/caveman-compression (main) score 41

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:32.964752+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:49:04.659798+00:00, confidence not recorded.
  - readme: https://github.com/wilpel/caveman-compression (fetched 2026-08-28T04:03:32.964752+00:00, sha f29d9d076842)
- Data as of 2026-08-30T08:39:29.467469+00:00.