lightseekorg/tokenspeed

Health v2 · maintenance only

Flags: young

Adoption not part of the score

1989 stars · 256 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

TokenSpeed is a high-performance LLM inference engine designed for agentic workloads, aiming for TensorRT-LLM-level performance with vLLM-level usability. It features a compiler-backed SPMD modeling layer, a C++/Python scheduler with type-safe KV cache reuse, pluggable kernels, and an AsyncLLM entrypoint.

Use cases

• serve large language models at maximum throughput
• run inference for agentic coding workloads
• deploy frontier models like Qwen, Kimi, DeepSeek, and GLM on day 0
• run FP4 inference on NVIDIA Blackwell and AMD accelerators
• avoid hand-writing tensor parallelism and collective communication code
• serve multi-hundred-billion parameter MoE models efficiently

When to choose

• you need maximum tokens-per-second for production agentic LLM serving
• you run modern datacenter GPUs like NVIDIA B200/GB300 or AMD MI355X/MI455X
• you want vLLM-like usability with compiler-generated parallelism
• you need day-0 support for the latest frontier open models

When to avoid

• you only need lightweight local inference on consumer hardware
• you need broad CPU-only or edge-device support
• you require a long-stable, battle-tested engine with extensive ecosystem tooling
• your stack depends on features specific to vLLM or TensorRT-LLM

Facets

library · maturity active

llm-inference gpu-computing machine-learning compiler concurrency large-language-models machine-learning gpu-computing performance python inference-engine tensor-parallelism spmd kv-cache blackwell agentic-workloads vllm-alternative tensorrt-llm ai-agents gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lightseekorg/tokenspeed")