bheinzerling/bpemb

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1224 stars · 100 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

BPEmb is a collection of pre-trained subword embeddings in 275 languages based on Byte-Pair Encoding, trained on Wikipedia, distributed as a Python library wrapping gensim KeyedVectors and SentencePiece models. It provides subword segmentation and embedding vectors intended as input layers for neural NLP models.

Use cases

• get pretrained subword embeddings for low-resource languages
• segment text into byte-pair encoded subwords without tokenization
• initialize an embedding layer in PyTorch or TensorFlow for an NLP model
• handle out-of-vocabulary words in a neural NLP pipeline
• build a compact multilingual embedding model smaller than FastText
• encode sentences into subword IDs for downstream models

When to choose

• you need embeddings for many languages, especially low-resource ones
• you want small pretrained embeddings without tokenization or morphological analysis
• you need subword segmentation to handle unknown words in neural models

When to avoid

• you need state-of-the-art contextual embeddings like BERT or transformer models
• you only work with English and can use larger alternatives like FastText
• you need embeddings trained on domain-specific corpora rather than Wikipedia

Facets

library · maturity maintenance

nlp machine-learning parser machine-learning python word-embeddings byte-pair-encoding subword-segmentation sentencepiece gensim pretrained-models multilingual-nlp natural-language-processing multilingual

Member repositories

For agents

markdown · JSON · MCP: product_card(name="bheinzerling/bpemb")