huggingface/tokenizers

Health v2 · maintenance only

Adoption not part of the score

10997 stars · 1177 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Hugging Face Tokenizers is a fast, Rust-based library implementing state-of-the-art tokenization algorithms (BPE, WordPiece, Unigram) with bindings for Python, Rust, and Node.js. It handles the full tokenization pipeline including normalization, pre-tokenization, training new vocabularies, and pre-processing like padding, truncation, and special token handling.

Use cases

• train a BPE tokenizer on my own corpus
• tokenize text for a transformer model
• convert text to token ids for BERT or GPT
• build a custom vocabulary for a language model
• fast tokenization of large text datasets
• map tokens back to original text offsets
• pad and truncate sequences for model input

When to choose

• you need extremely fast tokenization or vocabulary training (GBs of text in seconds)
• you are training or fine-tuning transformer models and need consistent tokenization
• you need alignment tracking to map tokens back to original text
• you want the standard tokenizer used by the Hugging Face Transformers ecosystem

When to avoid

• you need a general-purpose text segmentation tool unrelated to ML models
• you only need simple whitespace splitting or regex tokenization
• you need subword tokenization for a language/model family with incompatible requirements

Facets

library · maturity stable

nlp parser machine-learning sdk machine-learning large-language-models developer-tools rust python cross-platform tokenizer bpe wordpiece unigram hugging-face transformers text-preprocessing natural-language-processing nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/tokenizers")