google/sentencepiece

Health v2 · maintenance only

Adoption not part of the score

12043 stars · 1379 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

SentencePiece is a fast, lightweight unsupervised text tokenizer and detokenizer for neural network-based text generation systems, implementing BPE and unigram language model subword segmentation. It trains directly from raw sentences without language-specific pre- or post-processing, making it language-independent and end-to-end.

Use cases

• train a subword tokenizer for an LLM from raw text
• tokenize text into subword pieces for neural machine translation
• encode and decode text with a fixed vocabulary size
• segment words in languages without whitespace delimiters
• build a language-independent preprocessing pipeline for NLP models

When to choose

• you need a fixed-vocabulary tokenizer trained directly on raw sentences
• you want language-agnostic tokenization without pre-tokenization rules
• you need BPE or unigram subword models used by LLMs like LLaMA and T5

When to avoid

• you need a tokenizer for a pretrained model with a different tokenizer format (e.g., WordPiece)
• you only need simple whitespace or rule-based word splitting

Facets

library · maturity stable

nlp parser machine-learning llm-training machine-learning large-language-models cpp python cross-platform tokenizer subword-tokenization bpe unigram-language-model detokenizer neural-machine-translation natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="google/sentencepiece")