# dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

Repository: https://github.com/dqbd/tiktokenizer
Canonical: https://ross.abutalabs.com/products/tiktokenizer
Homepage: https://tiktokenizer.vercel.app
Language: TypeScript
License: MIT
License Family: permissive
Topics: chatgpt, nextjs, openai, t3-stack, tiktoken, tokenizer
Last push: 2025-04-24T08:55:21+00:00

## Health v2 (maintenance only)
Score: 39/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 18, release rhythm 35, longevity 91
- inputs: {"age_days": 1280, "days_push": 496, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1678, forks 185 (observed 2026-08-28T04:05:20.945871+00:00)

## What it is
Tiktokenizer is a web-based playground for OpenAI's tiktoken tokenizers that calculates the exact number of tokens in a given prompt. It is a Next.js application built on the T3 stack, available as a hosted site and self-hostable source.

## Use cases
- count tokens in a prompt before sending to the OpenAI API
- see how text is split into tokens by gpt-4o or other models
- estimate API costs by measuring token usage of a message
- compare tokenization across different OpenAI model encodings
- inspect token boundaries with whitespace visualization
- test system prompts and chat messages for token limits

## When to choose
- you need a quick, visual way to count tokens for OpenAI models
- you want to check whether a prompt fits a model's context window
- you prefer a browser tool over writing tiktoken code yourself

## When to avoid
- you need tokenizers for non-OpenAI models like Llama or Claude
- you need programmatic token counting in your own code (use tiktoken directly)
- you need offline tokenization without a web app

## Facets
- artifact type: application
- maturity: stable
- function: nlp, developer-tools
- domain: large-language-models, developer-tools
- platform: -
- tags: tiktoken, tokenizer, openai, token-counting, playground, nextjs, natural-language-processing, web-server, nodejs

## Member repositories
- dqbd/tiktokenizer (main) score 39

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:20.945871+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:41:33.272755+00:00, confidence not recorded.
  - readme: https://github.com/dqbd/tiktokenizer (fetched 2026-08-28T04:05:20.945871+00:00, sha 8f298c4e8a10)
  - homepage: https://tiktokenizer.vercel.app (fetched 2026-08-29T11:15:09.039068+00:00, sha 8901a48e54c8)
- Data as of 2026-08-30T08:39:29.467469+00:00.