GeeeekExplorer/nano-vllm

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

15164 stars · 2516 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A lightweight vLLM-style LLM inference engine implemented from scratch in about 1,200 lines of Python. It offers fast offline inference with optimizations like prefix caching, tensor parallelism, torch compilation, and CUDA graphs.

Use cases

• run LLM inference locally with a minimal codebase
• learn how vLLM-style inference engines work internally
• serve offline batch generation for small transformer models
• benchmark inference throughput against vLLM
• experiment with prefix caching and CUDA graph optimizations
• run Qwen models on consumer GPUs

When to choose

• you want readable, hackable inference code instead of a large framework
• you need vLLM-comparable throughput with a tiny dependency footprint
• you're studying or teaching LLM inference internals
• you run small models on limited GPU memory

When to avoid

• you need production features like a serving API, continuous batching UI, or broad model support
• you require multi-node distributed inference
• you need extensive quantization or hardware backend support
• you want long-term enterprise support and a large community

Facets

library · maturity active

llm-inference machine-learning deep-learning gpu-computing large-language-models deep-learning machine-learning python cross-platform vllm inference-engine lightweight prefix-caching tensor-parallelism cuda-graph transformer qwen natural-language-processing gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="GeeeekExplorer/nano-vllm")