# openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Repository: https://github.com/openai/tiktoken
Canonical: https://ross.abutalabs.com/products/tiktoken
Language: Python
License: MIT
License Family: permissive
Last push: 2026-08-17T05:25:57+00:00

## Health v2 (maintenance only)
Score: 89/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 98, release rhythm 74, longevity 97
- inputs: {"age_days": 1371, "days_push": 16, "days_rel": 16, "gap_med": 133, "n_releases_24m": 6}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 19102, forks 1598 (observed 2026-08-28T04:11:27.409242+00:00)

## What it is
tiktoken is a fast byte pair encoding (BPE) tokenizer library for Python built for OpenAI's language models, converting text into reversible, lossless token sequences and supporting encodings like o200k_base and cl100k_base. It includes an educational submodule for training simple BPE encodings and visualising how models like GPT-4 tokenize text.

## Use cases
- count tokens for an OpenAI API request to fit the context window
- tokenize text the same way GPT-4 or GPT-4o does
- estimate OpenAI API costs by counting tokens in prompts
- encode and decode text reversibly with byte pair encoding
- split documents into token-sized chunks for LLM processing
- learn and visualise how BPE tokenization works
- look up the tokenizer corresponding to a specific OpenAI model

## When to choose
- You need token counts that exactly match how OpenAI models see text, e.g. for context limits or cost estimation
- You need a fast tokenizer that outpaces comparable open-source tokenizers by 3-6x on large text volumes
- You want a reversible, lossless encoder that works on arbitrary text and can be extended with custom encodings
- You want an educational tool to understand and visualise the BPE procedure

## When to avoid
- Your stack is not Python, since it is distributed as a pip package
- You need to train production-grade BPE vocabularies from scratch on your own corpus
- You only target non-OpenAI models and don't want to define or load custom encodings for their vocabularies

## Facets
- artifact type: library
- maturity: stable
- function: nlp
- domain: large-language-models, artificial-intelligence, developer-tools
- platform: python
- tags: tokenizer, bpe, byte-pair-encoding, token-counting, openai, gpt, llm, text-encoding, context-window, encoding-decoding, natural-language-processing

## Member repositories
- openai/tiktoken (main) score 89

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:27.409242+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:00:31.874545+00:00, confidence not recorded.
  - readme: https://github.com/openai/tiktoken (fetched 2026-08-28T04:11:27.409242+00:00, sha 0f4c9d7a6104)
- Data as of 2026-08-30T08:39:29.467469+00:00.