Luce-Org/lucebox

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

2788 stars · 265 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Lucebox is an open-source LLM inference engine written in C++ with custom CUDA/HIP kernels, focused on speculative decoding and speculative prefill for heterogeneous hardware and consumer GPUs. It ships as a server with OpenAI- and Anthropic-compatible APIs and is tuned per model and hardware target (e.g., RTX 3090, Strix Halo, AMD R9700).

Use cases

• run large language models locally on consumer GPUs
• speed up long-context prefill on a single GPU
• serve LLMs with an OpenAI-compatible API on self-hosted hardware
• combine unified memory and dedicated GPU for large-model inference
• get faster token generation with speculative decoding
• run inference across mixed CPU/GPU heterogeneous machines
• keep AI inference fully private on owned hardware

When to choose

• you want maximum tokens/sec on consumer GPUs like RTX 3090 or AMD R9700
• you need fast time-to-first-token on very long prompts (64K-256K tokens)
• you want to mix unified-memory APUs with discrete GPUs in one inference setup
• you prefer a C++/CUDA engine with no Python, PyTorch, or Triton in the inference loop
• you want a plug-and-play private local AI server with compatible APIs

When to avoid

• you need broad multi-vendor model support beyond the tuned model list
• you require multi-node distributed inference at datacenter scale
• you need a Python ecosystem for custom model experimentation
• you only need CPU-only inference without GPU acceleration
• you want a mature, widely-deployed engine like vLLM with extensive community tooling

Facets

service · maturity active

llm-inference gpu-computing http-server sdk large-language-models machine-learning gpu-computing self-hosted developer-tools self-hosted cpp speculative-decoding speculative-prefill cuda-kernels rocm heterogeneous-computing consumer-gpus local-ai openai-compatible-api llama-cpp moe-inference paged-attention long-context linux gpu cuda

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Luce-Org/lucebox")