rsennrich/subword-nmt

Health v2 · maintenance only

Adoption not part of the score

2274 stars · 471 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library and CLI toolset for unsupervised word segmentation into subword units, best known for byte pair encoding (BPE) used in neural machine translation preprocessing. It provides commands to learn BPE codes, apply segmentation, and build vocabularies.

Use cases

• learn bpe codes for machine translation preprocessing
• segment text into subword units for nmt
• tokenize text with byte pair encoding
• build joint bpe vocabulary for multiple languages
• segment rare words into character n-grams
• preprocess training data for neural text generation

When to choose

• you need the classic BPE subword segmentation for NMT pipelines
• you want a simple pip-installable CLI for tokenization preprocessing
• you need reproducible segmentation for research experiments

When to avoid

• you need modern tokenizers like SentencePiece or HuggingFace tokenizers with faster implementations
• you need trained subword models bundled with pretrained vocabularies
• you need runtime tokenization integrated into a model inference pipeline

Facets

library · maturity maintenance

nlp parser cli machine-learning machine-learning developer-tools python cli cross-platform bpe subword-segmentation tokenization neural-machine-translation preprocessing natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="rsennrich/subword-nmt")