dqbd/tiktokenizer

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1678 stars · 185 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Tiktokenizer is a web-based playground for OpenAI's tiktoken tokenizers that calculates the exact number of tokens in a given prompt. It is a Next.js application built on the T3 stack, available as a hosted site and self-hostable source.

Use cases

• count tokens in a prompt before sending to the OpenAI API
• see how text is split into tokens by gpt-4o or other models
• estimate API costs by measuring token usage of a message
• compare tokenization across different OpenAI model encodings
• inspect token boundaries with whitespace visualization
• test system prompts and chat messages for token limits

When to choose

• you need a quick, visual way to count tokens for OpenAI models
• you want to check whether a prompt fits a model's context window
• you prefer a browser tool over writing tiktoken code yourself

When to avoid

• you need tokenizers for non-OpenAI models like Llama or Claude
• you need programmatic token counting in your own code (use tiktoken directly)
• you need offline tokenization without a web app

Facets

application · maturity stable

nlp developer-tools large-language-models developer-tools tiktoken tokenizer openai token-counting playground nextjs natural-language-processing web-server nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="dqbd/tiktokenizer")