tile-ai/TileRT

Health v2 · maintenance only

Adoption not part of the score

1831 stars · 128 forks observed · 2026-10-01

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

TileRT is a tile-based runtime for ultra-low-latency LLM inference, achieving hundreds to over 1000 tokens/s decode speeds on frontier models like DeepSeek-V3.2 and GLM-5 using model-system co-design on multi-GPU nodes. It integrates with vLLM via prefill-decode disaggregation behind an OpenAI-compatible endpoint.

Use cases

• run frontier LLMs with millisecond single-stream decode latency
• serve GLM-5 or DeepSeek-V3.2 at 500-1000+ tokens/s
• combine vLLM prefill with TileRT decode via PD disaggregation
• use multi-token prediction to boost decoding throughput
• sustain high tokens/s at very long context lengths up to 200K

When to choose

• you need maximum single-stream decode speed for large MoE models on NVIDIA GPUs
• you want an OpenAI-compatible endpoint with vLLM prefill and TileRT decode
• you are building latency-sensitive agentic or real-time applications

When to avoid

• you need broad model coverage beyond the supported GLM/DeepSeek/MiMo models
• you need a general-purpose serving stack with heavy multi-user batching
• you run on non-NVIDIA hardware or small consumer GPUs

Facets

library · maturity active

llm-inference gpu-computing machine-learning large-language-models deep-learning gpu-computing performance python inference-runtime low-latency decode-optimization multi-token-prediction pd-disaggregation vllm-integration linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tile-ai/TileRT")