# rsennrich/subword-nmt

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

Repository: https://github.com/rsennrich/subword-nmt
Canonical: https://ross.abutalabs.com/products/subword-nmt
Language: Python
License: MIT
License Family: permissive
Topics: neural-machine-translation, segmentation, machine-translation, nmt, subword-units, bpe
Last push: 2024-08-07T14:21:16+00:00

## Health v2 (maintenance only)
Score: 23/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 4019, "days_push": 756, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2274, forks 471 (observed 2026-08-28T04:06:33.353102+00:00)

## What it is
A Python library and CLI toolset for unsupervised word segmentation into subword units, best known for byte pair encoding (BPE) used in neural machine translation preprocessing. It provides commands to learn BPE codes, apply segmentation, and build vocabularies.

## Use cases
- learn bpe codes for machine translation preprocessing
- segment text into subword units for nmt
- tokenize text with byte pair encoding
- build joint bpe vocabulary for multiple languages
- segment rare words into character n-grams
- preprocess training data for neural text generation

## When to choose
- you need the classic BPE subword segmentation for NMT pipelines
- you want a simple pip-installable CLI for tokenization preprocessing
- you need reproducible segmentation for research experiments

## When to avoid
- you need modern tokenizers like SentencePiece or HuggingFace tokenizers with faster implementations
- you need trained subword models bundled with pretrained vocabularies
- you need runtime tokenization integrated into a model inference pipeline

## Facets
- artifact type: library
- maturity: maintenance
- function: nlp, parser, cli, machine-learning
- domain: machine-learning, developer-tools
- platform: python, cli, cross-platform
- tags: bpe, subword-segmentation, tokenization, neural-machine-translation, preprocessing, natural-language-processing

## Member repositories
- rsennrich/subword-nmt (main) score 23

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:33.353102+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:41:32.719239+00:00, confidence not recorded.
  - readme: https://github.com/rsennrich/subword-nmt (fetched 2026-08-28T04:06:33.353102+00:00, sha e301db657a77)
  - registry_pypi: https://pypi.org/pypi/subword-nmt/json (fetched 2026-08-29T10:22:01.063074+00:00, sha 9bad1e31bb77)
- Data as of 2026-08-30T08:39:29.467469+00:00.