huggingface/text-generation-inference

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

10889 stars · 1280 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Text Generation Inference (TGI) is a Rust, Python and gRPC toolkit for deploying and serving large language models with high performance, powering Hugging Face's production services. It supports popular open LLMs (Llama, Mistral, Falcon, StarCoder, BLOOM, etc.) with features like continuous batching, tensor parallelism, token streaming, and an OpenAI-compatible Messages API.

Use cases

• serve llama models on a gpu server
• deploy an open-source llm behind an openai-compatible api
• run high-throughput text generation with continuous batching
• host a self-hosted chat completion endpoint
• serve large models across multiple gpus with tensor parallelism
• stream llm tokens over server-sent events

When to choose

• you need a production-ready, battle-tested LLM serving stack with Prometheus metrics and tracing
• you want an OpenAI Chat Completion-compatible API for open models
• you need multi-GPU tensor parallelism and quantization support
• you are deploying on Nvidia, AMD, Intel Gaudi, or AWS Inferentia hardware

When to avoid

• you want the most actively developed inference engine - TGI is in maintenance mode and recommends vLLM or SGLang
• you need CPU-only or lightweight local inference - llama.cpp or MLX are better fits
• you need to serve model architectures outside TGI's optimized list

Facets

service · maturity maintenance

llm-inference http-server rpc monitoring gpu-computing large-language-models machine-learning deep-learning self-hosted apis python rust self-hosted text-generation serving continuous-batching tensor-parallelism flash-attention paged-attention openai-compatible-api token-streaming huggingface docker linux gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/text-generation-inference")