srush/llama2.rs

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1063 stars · 56 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A pure Rust implementation of Llama2 inference optimized for fast CPU execution, featuring 4-bit GPT-Q quantization, SIMD support, and memory mapping. It can be compiled with model-specific features and also exposes a Python calling API.

Use cases

• run llama2 inference on cpu in rust
• generate text from a quantized llama2 70b model locally
• load llama2 weights from hugging face and run inference
• call llama2 generation from python
• benchmark llama2 tokens per second on an intel cpu
• run 4-bit gptq quantized llama models without a gpu

When to choose

• you want fast CPU-only Llama2 inference in Rust
• you need to run quantized 4-bit Llama2 models with memory mapping
• you want a Python API backed by a Rust inference engine

When to avoid

• you need GPU-accelerated inference (Triton support is not implemented)
• you want a plug-and-play tool that works with arbitrary models without recompiling
• you need production-grade stability or broad model support

Facets

library · maturity experimental

llm-inference machine-learning cli large-language-models machine-learning developer-tools rust windows cli python llama2 cpu-inference quantization gptq simd memory-mapping text-generation linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="srush/llama2.rs")