IST-DASLab/marlin

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1136 stars · 91 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Marlin is a highly optimized FP16xINT4 matrix multiplication CUDA kernel for LLM inference that achieves near-ideal 4x speedups at batch sizes up to 16-32 tokens. It uses advanced GPU optimization techniques to fully saturate memory, cache, and tensor core resources simultaneously.

Use cases

• speed up LLM inference with 4-bit quantized weights
• serve quantized models at medium batch sizes
• accelerate speculative decoding
• run multi-inference schemes like CoT-Majority efficiently
• integrate a fast FP16xINT4 matmul kernel into inference frameworks

When to choose

• you need near-4x speedups from weight-only 4-bit quantization at batch sizes of 16-32 tokens
• you are serving quantized LLMs at larger scale or with speculative decoding
• prior kernels only give speedups at batch size 1-2 and you need more throughput

When to avoid

• you need very large batch sizes where the kernel's advantage diminishes
• you need mixed FP16xINT4 support on non-NVIDIA or older GPUs
• you want a full inference engine rather than a single matmul kernel

Facets

library · maturity active

llm-inference gpu-computing machine-learning large-language-models deep-learning gpu-computing python quantization 4bit-weights cuda-kernel matmul weight-only-quantization gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="IST-DASLab/marlin")