lancopku/pkuseg-python

Health v2 · maintenance only

Adoption not part of the score

6710 stars · 980 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

pkuseg is a Python toolkit for multi-domain Chinese word segmentation based on the paper by Luo et al. (2019). It provides pre-trained models for specific domains (news, web, medicine, tourism, mixed), supports part-of-speech tagging, and allows users to train custom models on their own annotated data.

Use cases

• segment Chinese text into words
• tokenize Chinese text for a specific domain like medicine or news
• perform part-of-speech tagging on Chinese text
• train a custom Chinese word segmentation model on my own labeled data
• get higher Chinese segmentation accuracy than jieba or THULAC
• preprocess Chinese text for downstream NLP tasks

When to choose

• you need accurate Chinese word segmentation, especially in a known domain like news, web, medicine, or tourism
• you want to train a segmentation model on your own annotated corpus
• you need part-of-speech tagging alongside segmentation
• benchmark results show it outperforms jieba or THULAC on your data

When to avoid

• you need a lightweight, fast general-purpose tokenizer without model downloads
• you require active development or recent updates, as the project has not seen releases since 2022
• you need segmentation for languages other than Chinese
• you are on a platform other than 64-bit Linux, macOS, or Windows and cannot compile locally

Facets

library · maturity maintenance

nlp parser machine-learning machine-learning developer-tools python windows chinese-word-segmentation tokenization part-of-speech-tagging multi-domain pretrained-models natural-language-processing linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lancopku/pkuseg-python")