michaelfeil/infinity

Health v2 · maintenance only

Adoption not part of the score

2917 stars · 197 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Infinity is a high-throughput, low-latency serving engine that exposes text-embedding, reranking, CLIP, CLAP and ColPali models via an OpenAI-compatible REST API. It is built on FastAPI with PyTorch, ONNX/TensorRT and CTranslate2 backends, supporting dynamic batching across CUDA, ROCm, CPU, AWS Inf2 and Apple MPS.

Use cases

• self-host an OpenAI-compatible embeddings API
• serve sentence-transformer models for a RAG pipeline
• deploy reranking models behind a REST endpoint
• run CLIP image and text embeddings on GPU
• batch-serve multiple embedding models from one server
• host ColPali vision retrieval models

When to choose

• you need a self-hosted, OpenAI-spec-compatible embedding server
• you want to mix multiple embedding/reranking/multimodal models in one service
• you need high throughput with dynamic batching and hardware acceleration
• you want an MIT-licensed alternative to proprietary embedding APIs

When to avoid

• you only need embeddings inside a single Python app without a server
• you need full LLM text generation, not embeddings or reranking
• you want a fully managed hosted service rather than self-hosting

Facets

service · maturity active

llm-inference http-server api-framework machine-learning rag machine-learning large-language-models apis self-hosted backend python windows self-hosted cross-platform embeddings reranking clip sentence-transformers openai-compatible-api text-embeddings-inference fastapi docker linux macos gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="michaelfeil/infinity")