karpathy/minbpe

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

10691 stars · 1092 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A minimal, clean Python implementation of the byte-level Byte Pair Encoding (BPE) algorithm used for tokenization in modern LLMs like GPT-4 and Llama. It provides train, encode, and decode functionality with a GPT-4-compatible tokenizer that reproduces tiktoken's output.

Use cases

• train a BPE tokenizer on custom text
• understand how LLM tokenization works
• reproduce GPT-4 tokenization in Python
• encode and decode text to tokens and back
• learn the byte pair encoding algorithm from clean code
• build a custom tokenizer for an LLM project

When to choose

• you want readable, educational BPE code to study or modify
• you need to train a small custom tokenizer without heavy dependencies
• you want to exactly reproduce GPT-4 tokenization behavior

When to avoid

• you need a fast production tokenizer for large-scale workloads (use tiktoken or HuggingFace tokenizers)
• you need tokenizer features like parallel encoding or vocabularies beyond BPE

Facets

library · maturity stable

nlp parser machine-learning llm-training developer-tools large-language-models machine-learning education developer-tools python cross-platform tokenizer byte-pair-encoding bpe llm educational minimal-implementation natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="karpathy/minbpe")