marcelroed/gigatoken

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

4061 stars · 217 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Gigatoken is a Rust-based tokenizer library for language modeling that tokenizes text at GB/s throughput, claiming ~1000x speedups over HuggingFace tokenizers. It offers a native fast API plus drop-in compatibility modes with HuggingFace Tokenizers and tiktoken, installable via pip.

Use cases

• tokenize large text corpora for LLM pretraining
• speed up tokenization bottleneck in training data pipelines
• drop-in replacement for HuggingFace tokenizers
• replace tiktoken with a faster tokenizer
• batch encode large text files at GB/s
• tokenize text for models like GPT-2, Llama 3, or Qwen

When to choose

• you need to tokenize gigabytes of training data quickly
• tokenization is the bottleneck in your LLM data pipeline
• you want exact-output compatibility with HF tokenizers or tiktoken but faster
• you have many CPU cores available for parallel tokenization

When to avoid

• you need tokenizers for exotic models not among the supported set
• you only tokenize small amounts of text where speed is irrelevant
• you depend on HF tokenizers features like training new tokenizers or advanced preprocessing

Facets

library · maturity active

nlp llm-training parser benchmarking large-language-models machine-learning performance python rust windows cross-platform tokenizer llm-pretraining huggingface-compatibility tiktoken-compatibility high-throughput data-pipeline natural-language-processing linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="marcelroed/gigatoken")