Mega4alik/ollm

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2788 stars · 260 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

oLLM is a lightweight Python library for large-context LLM inference built on Hugging Face Transformers and PyTorch. It offloads weights and KV cache to SSD to run models like Llama 3.1 8B or Qwen3-Next-80B at 100k context on consumer GPUs with as little as 8GB VRAM, without quantization.

Use cases

• run 100k context LLM inference on an 8GB consumer GPU
• serve large-context offline workloads without quantizing models
• run an 80B parameter model from SSD with minimal VRAM
• do multimodal inference with audio or image inputs
• run Llama models with PEFT adapters at long context

When to choose

• you have a small consumer GPU but large SSD and need long-context inference
• you want full-precision (fp16/bf16) inference without quantization
• your workload is offline/batch and throughput matters more than latency

When to avoid

• you need low-latency interactive serving on high-end hardware
• you want quantized models for maximum speed
• you have no fast SSD or NVIDIA GPU available

Facets

library · maturity active

llm-inference machine-learning gpu-computing large-language-models machine-learning deep-learning python large-context vram-offloading consumer-gpu fp16 kv-cache-offload multimodal huggingface-transformers gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Mega4alik/ollm")