thinking-machines-lab/batch_invariant_ops

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1067 stars · 84 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library that replaces standard PyTorch CUDA kernels with batch-invariant versions, ensuring identical results regardless of batch size. It accompanies Thinking Machines' blog on defeating nondeterminism in LLM inference and includes a proof-of-concept for deterministic vLLM inference.

Use cases

• make llm inference deterministic regardless of batch size
• get reproducible pytorch gpu results
• eliminate nondeterminism in vllm serving
• ensure matrix multiplication gives same output for different batch sizes
• debug floating point nondeterminism in cuda kernels

When to choose

• you need bit-exact reproducible GPU inference results
• you run vLLM and want deterministic completions
• you're debugging batch-size-dependent numerical differences in PyTorch

When to avoid

• you need maximum inference throughput and can tolerate nondeterminism
• you rely on operations not covered (only mm, addmm, log_softmax, mean are supported)
• you're not using CUDA GPUs

Facets

library · maturity experimental

machine-learning llm-inference gpu-computing deep-learning large-language-models performance python determinism pytorch kernels vllm reproducibility gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="thinking-machines-lab/batch_invariant_ops")