openai/tiktoken

Health v2 · maintenance only

Adoption not part of the score

19102 stars · 1598 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

tiktoken is a fast byte pair encoding (BPE) tokenizer library for Python built for OpenAI's language models, converting text into reversible, lossless token sequences and supporting encodings like o200k_base and cl100k_base. It includes an educational submodule for training simple BPE encodings and visualising how models like GPT-4 tokenize text.

Use cases

• count tokens for an OpenAI API request to fit the context window
• tokenize text the same way GPT-4 or GPT-4o does
• estimate OpenAI API costs by counting tokens in prompts
• encode and decode text reversibly with byte pair encoding
• split documents into token-sized chunks for LLM processing
• learn and visualise how BPE tokenization works
• look up the tokenizer corresponding to a specific OpenAI model

When to choose

• You need token counts that exactly match how OpenAI models see text, e.g. for context limits or cost estimation
• You need a fast tokenizer that outpaces comparable open-source tokenizers by 3-6x on large text volumes
• You want a reversible, lossless encoder that works on arbitrary text and can be extended with custom encodings
• You want an educational tool to understand and visualise the BPE procedure

When to avoid

• Your stack is not Python, since it is distributed as a pip package
• You need to train production-grade BPE vocabularies from scratch on your own corpus
• You only target non-OpenAI models and don't want to define or load custom encodings for their vocabularies

Facets

library · maturity stable

nlp large-language-models artificial-intelligence developer-tools python tokenizer bpe byte-pair-encoding token-counting openai gpt llm text-encoding context-window encoding-decoding natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="openai/tiktoken")