Tencent/TurboTransformers

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

1548 stars · 208 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A fast, user-friendly C++/Python runtime for transformer model inference (BERT, ALBERT, RoBERTa, GPT2, decoders) on CPU and GPU. It works as a PyTorch plugin, requiring only a few lines of code to accelerate Hugging Face transformer models with variable-length inputs and smart batching.

Use cases

• speed up bert inference on cpu and gpu
• accelerate huggingface transformer models in production
• serve variable-length transformer batches without offline tuning
• run gpt2 decoder inference faster
• boost nlp services like faq matching and sentiment analysis
• replace tensorrt or onnx runtime for transformer serving

When to choose

• you need faster BERT/ALBERT/GPT2 inference than PyTorch with minimal code changes
• you serve variable-length inputs and want to avoid static-shape preprocessing
• you want both CPU and GPU acceleration from one runtime

When to avoid

• you need the latest LLM architectures beyond classic encoder/decoder transformers
• you require a permissively licensed dependency (license is non-standard)
• you want a broadly maintained ecosystem - the project sees infrequent releases

Facets

library · maturity maintenance

llm-inference machine-learning nlp sdk machine-learning deep-learning large-language-models cpp python cross-platform transformer-inference bert gpt2 huggingface pytorch-plugin cpu-gpu smart-batching natural-language-processing linux gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Tencent/TurboTransformers")