# srush/llama2.rs

A fast llama2 decoder in pure Rust.

Repository: https://github.com/srush/llama2.rs
Canonical: https://ross.abutalabs.com/products/llama2rs
Language: Rust
License: MIT
License Family: permissive
Last push: 2023-11-30T00:00:48+00:00

## Health v2 (maintenance only)
Score: 28/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 35, longevity 80
- inputs: {"age_days": 1132, "days_push": 1008, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1063, forks 56 (observed 2026-08-28T04:03:26.368493+00:00)

## What it is
A pure Rust implementation of Llama2 inference optimized for fast CPU execution, featuring 4-bit GPT-Q quantization, SIMD support, and memory mapping. It can be compiled with model-specific features and also exposes a Python calling API.

## Use cases
- run llama2 inference on cpu in rust
- generate text from a quantized llama2 70b model locally
- load llama2 weights from hugging face and run inference
- call llama2 generation from python
- benchmark llama2 tokens per second on an intel cpu
- run 4-bit gptq quantized llama models without a gpu

## When to choose
- you want fast CPU-only Llama2 inference in Rust
- you need to run quantized 4-bit Llama2 models with memory mapping
- you want a Python API backed by a Rust inference engine

## When to avoid
- you need GPU-accelerated inference (Triton support is not implemented)
- you want a plug-and-play tool that works with arbitrary models without recompiling
- you need production-grade stability or broad model support

## Facets
- artifact type: library
- maturity: experimental
- function: llm-inference, machine-learning, cli
- domain: large-language-models, machine-learning, developer-tools
- platform: rust, windows, cli, python
- tags: llama2, cpu-inference, quantization, gptq, simd, memory-mapping, text-generation, linux, macos

## Member repositories
- srush/llama2.rs (main) score 28

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:26.368493+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:56:41.246928+00:00, confidence not recorded.
  - readme: https://github.com/srush/llama2.rs (fetched 2026-08-28T04:03:26.368493+00:00, sha d1de286c704d)
- Data as of 2026-08-30T08:39:29.467469+00:00.