# marcelroed/gigatoken

Language model tokenization at GB/s

Repository: https://github.com/marcelroed/gigatoken
Canonical: https://ross.abutalabs.com/products/gigatoken
Language: Rust
License: MIT
License Family: permissive
Topics: llm, nlp, tokenization, tokenizer
Last push: 2026-08-06T00:11:22+00:00

## Health v2 (maintenance only)
Score: 60/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 96, release rhythm 35, longevity 21
- inputs: {"age_days": 296, "days_push": 28, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 4061, forks 217 (observed 2026-08-28T04:08:34.018313+00:00)

## What it is
Gigatoken is a Rust-based tokenizer library for language modeling that tokenizes text at GB/s throughput, claiming ~1000x speedups over HuggingFace tokenizers. It offers a native fast API plus drop-in compatibility modes with HuggingFace Tokenizers and tiktoken, installable via pip.

## Use cases
- tokenize large text corpora for LLM pretraining
- speed up tokenization bottleneck in training data pipelines
- drop-in replacement for HuggingFace tokenizers
- replace tiktoken with a faster tokenizer
- batch encode large text files at GB/s
- tokenize text for models like GPT-2, Llama 3, or Qwen

## When to choose
- you need to tokenize gigabytes of training data quickly
- tokenization is the bottleneck in your LLM data pipeline
- you want exact-output compatibility with HF tokenizers or tiktoken but faster
- you have many CPU cores available for parallel tokenization

## When to avoid
- you need tokenizers for exotic models not among the supported set
- you only tokenize small amounts of text where speed is irrelevant
- you depend on HF tokenizers features like training new tokenizers or advanced preprocessing

## Facets
- artifact type: library
- maturity: active
- function: nlp, llm-training, parser, benchmarking
- domain: large-language-models, machine-learning, performance
- platform: python, rust, windows, cross-platform
- tags: tokenizer, llm-pretraining, huggingface-compatibility, tiktoken-compatibility, high-throughput, data-pipeline, natural-language-processing, linux, macos

## Member repositories
- marcelroed/gigatoken (main) score 60

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:34.018313+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:23:35.560539+00:00, confidence not recorded.
  - readme: https://github.com/marcelroed/gigatoken (fetched 2026-08-28T04:08:34.018313+00:00, sha a35859b9c398)
- Data as of 2026-08-30T08:39:29.467469+00:00.