microsoft/BlingFire

Health v2 · maintenance only

Adoption not part of the score

1890 stars · 142 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Bling Fire is a fast finite state machine and regular expression manipulation library from Microsoft, primarily used for high-performance natural language tokenization. It supports pattern-based tokenization, WordPiece, SentencePiece Unigram LM and BPE, with precompiled models for BERT, XLNet, and other popular NLP models.

Use cases

• tokenize text for BERT or XLNet models
• fast word and sentence segmentation of large text corpora
• run WordPiece or SentencePiece BPE tokenization in production
• stem or lemmatize words with finite state machines
• match multi-word expressions in search queries
• syllabify or hyphenate words
• tokenize text from Python, Rust, Ruby, C#, or JavaScript

When to choose

• you need extremely fast tokenization with minimal initialization overhead
• you need consistent tokenization across multiple NLP model formats via one API
• you process large volumes of text in a low-latency or resource-constrained environment

When to avoid

• you need to train new tokenizer models from scratch (use SentencePiece or Hugging Face Tokenizers)
• you need rich linguistic analysis beyond tokenization
• you prefer a pure high-level Python ecosystem tool with extensive community support

Facets

library · maturity maintenance

nlp parser machine-learning machine-learning developer-tools cross-platform python cpp rust wasm tokenizer finite-state-machine regular-expressions wordpiece sentencepiece bpe tokenization sentence-breaking stemming natural-language-processing nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/BlingFire")