FlashML-org/FreeToken

Health v2 · maintenance only

Flags: young

Adoption not part of the score

8370 stars · 729 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

FreeToken is an edge-native Mixture-of-Experts (MoE) LLM serving engine that runs frontier-scale open-weight models on consumer hardware by treating GPUs, CPUs, and host memory as a unified inference platform. It ships as a desktop app with a GUI and exposes OpenAI- and Anthropic-compatible APIs for integration with coding agents and chat tools.

Use cases

• run large MoE models locally on a gaming PC
• serve open-weight LLMs with an OpenAI-compatible API
• offload LLM inference between CPU and GPU
• run coding agents like Claude Code or Codex against a local model
• fit a 290B parameter model on limited VRAM
• chat with local models through a desktop GUI

When to choose

• you want to run frontier-scale MoE models on consumer NVIDIA RTX GPUs
• you need local LLM serving with OpenAI/Anthropic-compatible endpoints
• you have limited VRAM and need elastic CPU-GPU memory offloading
• you want semantic caching to avoid recomputing context for agentic workflows

When to avoid

• you need multi-node datacenter-scale serving rather than single-machine edge inference
• you rely on non-NVIDIA GPUs, since support focuses on RTX 30/40/50 series
• you need dense (non-MoE) model architectures that may not be supported
• you want a fully managed cloud inference service

Facets

application · maturity active

llm-inference caching http-server gpu-computing large-language-models machine-learning developer-tools self-hosted windows python cross-platform mixture-of-experts local-llm cpu-gpu-offload openai-compatible-api anthropic-compatible-api quantization edge-inference consumer-gpu linux desktop gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="FlashML-org/FreeToken")