tensorflow/text

Health v2 · maintenance only

Adoption not part of the score

1294 stars · 379 forks observed · 2026-09-30

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

TensorFlow Text is a library of text-processing classes and ops for TensorFlow 2.x, providing tokenization, Unicode normalization, n-grams, and other sequence-modeling preprocessing that runs inside the TensorFlow graph. It ensures preprocessing is identical between training and inference and underpins higher-level libraries like KerasNLP.

Use cases

• tokenize text for tensorflow models
• unicode normalization and script segmentation in a tf graph
• preprocess text for bert fine-tuning
• build n-grams and sliding windows for sequence models
• avoid train/inference tokenization mismatch
• wordshape feature extraction in tensorflow

When to choose

• you need low-level text ops inside the TensorFlow graph
• you must guarantee identical preprocessing at training and serving time
• KerasNLP abstractions are too high-level for your needs

When to avoid

• you use PyTorch or JAX instead of TensorFlow
• you just want pretrained NLP models - use KerasNLP or Hugging Face instead
• you need general-purpose NLP outside a TensorFlow pipeline

Facets

library · maturity maintenance

nlp machine-learning parser serialization machine-learning deep-learning python windows tokenization unicode tensorflow text-preprocessing keras natural-language-processing linux macos gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tensorflow/text")