algorithmicsuperintelligence/optillm

Health v2 · maintenance only

Adoption not part of the score

4252 stars · 385 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

OptiLLM is an OpenAI API-compatible optimizing inference proxy that applies 20+ inference-time techniques (e.g., Mixture of Agents, MCTS, chain-of-thought) to improve LLM accuracy without training. It acts as a drop-in replacement server in front of OpenAI, Anthropic, Google, Cerebras, and 100+ models via LiteLLM.

Use cases

• improve llm accuracy on math and logic problems without fine-tuning
• proxy openai api calls through reasoning optimization techniques
• get frontier-model performance from smaller cheaper models
• run best-of-n sampling or mixture of agents over any openai-compatible endpoint
• boost coding and reasoning benchmark scores at inference time
• route multiple llm providers behind one openai-compatible api

When to choose

• you want better reasoning accuracy from existing models with zero training
• you need a drop-in OpenAI-compatible proxy for multiple providers
• you can afford extra inference-time compute to improve answer quality

When to avoid

• you need minimal latency and cannot tolerate extra inference compute
• your workload is simple chat where optimization overhead adds no value
• you need a fully managed hosted service rather than self-hosting

Facets

service · maturity active

llm-inference proxy api-gateway prompt-engineering agent-framework middleware large-language-models artificial-intelligence apis developer-tools python self-hosted cli openai-compatible inference-optimization mixture-of-agents mcts chain-of-thought reasoning test-time-compute litellm docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="algorithmicsuperintelligence/optillm")