# karpathy/minbpe

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Repository: https://github.com/karpathy/minbpe
Canonical: https://ross.abutalabs.com/products/minbpe
Language: Python
License: MIT
License Family: permissive
Last push: 2024-07-01T14:20:22+00:00

## Health v2 (maintenance only)
Score: 25/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 66
- inputs: {"age_days": 929, "days_push": 793, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 10691, forks 1092 (observed 2026-08-28T04:10:43.784508+00:00)

## What it is
A minimal, clean Python implementation of the byte-level Byte Pair Encoding (BPE) algorithm used for tokenization in modern LLMs like GPT-4 and Llama. It provides train, encode, and decode functionality with a GPT-4-compatible tokenizer that reproduces tiktoken's output.

## Use cases
- train a BPE tokenizer on custom text
- understand how LLM tokenization works
- reproduce GPT-4 tokenization in Python
- encode and decode text to tokens and back
- learn the byte pair encoding algorithm from clean code
- build a custom tokenizer for an LLM project

## When to choose
- you want readable, educational BPE code to study or modify
- you need to train a small custom tokenizer without heavy dependencies
- you want to exactly reproduce GPT-4 tokenization behavior

## When to avoid
- you need a fast production tokenizer for large-scale workloads (use tiktoken or HuggingFace tokenizers)
- you need tokenizer features like parallel encoding or vocabularies beyond BPE

## Facets
- artifact type: library
- maturity: stable
- function: nlp, parser, machine-learning, llm-training, developer-tools
- domain: large-language-models, machine-learning, education, developer-tools
- platform: python, cross-platform
- tags: tokenizer, byte-pair-encoding, bpe, llm, educational, minimal-implementation, natural-language-processing

## Member repositories
- karpathy/minbpe (main) score 25

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:43.784508+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:17:58.659744+00:00, confidence not recorded.
  - readme: https://github.com/karpathy/minbpe (fetched 2026-08-28T04:10:43.784508+00:00, sha 3257561ae230)
- Data as of 2026-08-30T08:39:29.467469+00:00.