# huggingface/text-generation-inference

Large Language Model Text Generation Inference

Repository: https://github.com/huggingface/text-generation-inference
Canonical: https://ross.abutalabs.com/products/text-generation-inference
Homepage: http://hf.co/docs/text-generation-inference
Language: Python
License: Apache-2.0
License Family: permissive
Topics: bloom, nlp, pytorch, inference, gpt, deep-learning, transformer, falcon, starcoder
Archived: true
Last push: 2026-03-21T11:34:22+00:00

## Health v2 (maintenance only)
Score: 10/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 73, release rhythm 62, longevity 100
- inputs: {"age_days": 1425, "days_push": 165, "days_rel": 257, "gap_med": 15.5, "n_releases_24m": 21}
- flags: archived
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 10889, forks 1280 (observed 2026-08-28T04:10:44.603621+00:00)

## What it is
Text Generation Inference (TGI) is a Rust, Python and gRPC toolkit for deploying and serving large language models with high performance, powering Hugging Face's production services. It supports popular open LLMs (Llama, Mistral, Falcon, StarCoder, BLOOM, etc.) with features like continuous batching, tensor parallelism, token streaming, and an OpenAI-compatible Messages API.

## Use cases
- serve llama models on a gpu server
- deploy an open-source llm behind an openai-compatible api
- run high-throughput text generation with continuous batching
- host a self-hosted chat completion endpoint
- serve large models across multiple gpus with tensor parallelism
- stream llm tokens over server-sent events

## When to choose
- you need a production-ready, battle-tested LLM serving stack with Prometheus metrics and tracing
- you want an OpenAI Chat Completion-compatible API for open models
- you need multi-GPU tensor parallelism and quantization support
- you are deploying on Nvidia, AMD, Intel Gaudi, or AWS Inferentia hardware

## When to avoid
- you want the most actively developed inference engine - TGI is in maintenance mode and recommends vLLM or SGLang
- you need CPU-only or lightweight local inference - llama.cpp or MLX are better fits
- you need to serve model architectures outside TGI's optimized list

## Facets
- artifact type: service
- maturity: maintenance
- function: llm-inference, http-server, rpc, monitoring, gpu-computing
- domain: large-language-models, machine-learning, deep-learning, self-hosted, apis
- platform: python, rust, self-hosted
- tags: text-generation, serving, continuous-batching, tensor-parallelism, flash-attention, paged-attention, openai-compatible-api, token-streaming, huggingface, docker, linux, gpu

## Member repositories
- huggingface/text-generation-inference (main) score 10

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:44.603621+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:17:26.510223+00:00, confidence not recorded.
  - readme: https://github.com/huggingface/text-generation-inference (fetched 2026-08-28T04:10:44.603621+00:00, sha 729dcd84277b)
  - homepage: http://hf.co/docs/text-generation-inference (fetched 2026-08-29T08:16:26.705798+00:00, sha 1b08ae48e895)
  - site_page: https://huggingface.co/docs (fetched 2026-08-29T08:16:26.709575+00:00, sha bdec26667b98)
  - site_page: https://huggingface.co/docs/text-generation-inference/quicktour (fetched 2026-08-29T08:16:26.714209+00:00, sha cf72731fd6c7)
  - site_page: https://huggingface.co/docs/text-generation-inference/supported_models (fetched 2026-08-29T08:16:26.715927+00:00, sha 279b27da9308)
  - site_page: https://huggingface.co/docs/text-generation-inference/installation_nvidia (fetched 2026-08-29T08:16:26.717666+00:00, sha 35437bb2928f)
  - site_page: https://huggingface.co/docs/text-generation-inference/installation_amd (fetched 2026-08-29T08:16:26.719744+00:00, sha 34b42bf3691b)
  - site_page: https://huggingface.co/docs/text-generation-inference/installation_gaudi (fetched 2026-08-29T08:16:26.721371+00:00, sha 13e741593c24)
  - site_page: https://huggingface.co/docs/text-generation-inference/installation_inferentia (fetched 2026-08-29T08:16:26.723233+00:00, sha 58a62cfd3396)
  - site_page: https://huggingface.co/pricing (fetched 2026-08-29T08:16:26.712093+00:00, sha de6b7a178be5)
- Data as of 2026-08-30T08:39:29.467469+00:00.