# dleemiller/WordLlama

Things you can do with the token embeddings of an LLM

Repository: https://github.com/dleemiller/WordLlama
Canonical: https://ross.abutalabs.com/products/wordllama
Language: Python
License: MIT
License Family: permissive
Last push: 2025-12-01T21:45:32+00:00

## Health v2 (maintenance only)
Score: 49/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 55, release rhythm 35, longevity 58
- inputs: {"age_days": 813, "days_push": 275, "days_rel": 275, "gap_med": 164.5, "n_releases_24m": 3}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1451, forks 48 (observed 2026-08-28T04:04:46.094551+00:00)

## What it is
WordLlama is a fast, lightweight Python NLP toolkit built on LLM token embeddings for tasks like similarity computation, ranking, fuzzy deduplication, clustering, filtering, and semantic text splitting. It runs on CPU with minimal dependencies, making it suitable for resource-constrained environments.

## Use cases
- compute semantic similarity between two texts
- rank documents by relevance to a query
- deduplicate a large corpus of near-duplicate texts
- cluster documents by topic
- split long documents into semantically coherent chunks
- top-k retrieval of similar documents
- embed text cheaply on CPU without a GPU

## When to choose
- you need lightweight semantic text similarity on CPU without heavy inference dependencies
- you want fast fuzzy deduplication or clustering of text corpora
- you need semantic chunking/splitting for RAG pipelines on limited hardware

## When to avoid
- you need state-of-the-art full transformer embedding quality
- you require multilingual embeddings beyond the model's training
- you need GPU-accelerated or fine-tunable embedding models

## Facets
- artifact type: library
- maturity: active
- function: nlp, machine-learning, search-engine, data-science
- domain: machine-learning, developer-tools
- platform: python, cross-platform
- tags: embeddings, semantic-similarity, fuzzy-deduplication, clustering, semantic-splitting, cpu-optimized, token-embeddings, llm-derived, natural-language-processing, cpu

## Member repositories
- dleemiller/WordLlama (main) score 49

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:46.094551+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:35:50.921205+00:00, confidence not recorded.
  - readme: https://github.com/dleemiller/WordLlama (fetched 2026-08-28T04:04:46.094551+00:00, sha a3107ef215b8)
  - registry_pypi: https://pypi.org/pypi/wordllama/json (fetched 2026-08-29T11:45:13.734830+00:00, sha 600551cbe40c)
- Data as of 2026-08-30T08:39:29.467469+00:00.