jmaczan/tiny-vllm resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1081 stars · 84 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

tiny-vllm is both a minimal high-performance LLM inference engine written in C++ and CUDA, and a hands-on course that walks through building it from scratch. It implements a Llama 3.2 1B forward pass with CUDA kernels, KV cache, continuous batching, FlashAttention-style online softmax, and PagedAttention.

Use cases

• learn how LLM inference engines work internally
• build a mini vLLM in C++ and CUDA
• understand PagedAttention and KV cache implementation
• study CUDA kernel engineering for transformers
• teach a university course on LLM inference systems
• implement continuous batching from scratch

When to choose

• you want to learn inference engine internals by implementing them yourself
• you need a small readable codebase demonstrating PagedAttention and batching
• you are an educator looking for a teaching resource on GPU inference

When to avoid

• you need a production-ready inference server for real workloads
• you want broad model support beyond Llama-style architectures
• you prefer not to write or maintain CUDA code

Facets

learning-resource · maturity active

llm-inference gpu-computing machine-learning large-language-models deep-learning gpu-computing tutorials cpp cuda-kernels paged-attention kv-cache continuous-batching flashattention inference-engine course gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jmaczan/tiny-vllm")