# PaddlePaddle/FastDeploy

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

Repository: https://github.com/PaddlePaddle/FastDeploy
Canonical: https://ross.abutalabs.com/products/fastdeploy
Homepage: https://paddlepaddle.github.io/FastDeploy/
Language: Python
License: Apache-2.0
License Family: permissive
Topics: serving, ernie, llm, inference, llm-serving, openai, vllm, ernie-45, ernie-45-vl
Last push: 2026-08-26T11:42:13+00:00

## Health v2 (maintenance only)
Score: 88/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 99, release rhythm 66, longevity 100
- inputs: {"age_days": 1528, "days_push": 7, "days_rel": 146, "gap_med": 32, "n_releases_24m": 8}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3711, forks 759 (observed 2026-08-28T04:08:15.204650+00:00)

## What it is
FastDeploy is a high-performance inference and deployment toolkit for large language models and vision-language models built on PaddlePaddle. It provides production-ready serving with an OpenAI-compatible API, vLLM interface compatibility, PD disaggregation, KV cache transmission, and broad quantization format support across multiple hardware backends.

## Use cases
- serve an OpenAI-compatible API for ERNIE 4.5 models
- deploy LLMs with vLLM-compatible serving on NVIDIA GPUs
- run quantized LLM inference with W4A8 or FP8 weights
- deploy vision-language models like ERNIE-4.5-VL or Qwen-VL
- set up prefill-decoding disaggregation for high-throughput serving
- serve LLMs on Kunlunxin XPU, Ascend NPU, or Hygon DCU hardware
- enable speculative decoding and multi-token prediction for faster generation

## When to choose
- you need to deploy ERNIE 4.5 family models with first-class support
- you want vLLM-style serving on non-NVIDIA accelerators like Kunlunxin or Ascend
- you need industrial-grade PD disaggregation with load balancing and context caching
- you require wide quantization format support for memory-constrained serving

## When to avoid
- you need broad community support and ecosystem plugins of vLLM or SGLang
- your models are not in the supported list and you need flexible custom architectures
- you want a simple local inference library rather than a serving deployment toolkit
- you develop primarily on macOS or Windows without supported accelerator hardware

## Facets
- artifact type: library
- maturity: active
- function: llm-inference, http-server, gpu-computing, machine-learning
- domain: large-language-models, artificial-intelligence, deep-learning
- platform: python
- tags: llm-serving, vllm-compatible, openai-api, pd-disaggregation, quantization, vlm, paddlepaddle, kv-cache, speculative-decoding, linux, nvidia-gpu, docker

## Member repositories
- PaddlePaddle/FastDeploy (main) score 88

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:15.204650+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:30:54.606864+00:00, confidence not recorded.
  - readme: https://github.com/PaddlePaddle/FastDeploy (fetched 2026-08-28T04:08:15.204650+00:00, sha e6db595f1598)
  - homepage: https://paddlepaddle.github.io/FastDeploy/ (fetched 2026-08-29T09:24:21.794204+00:00, sha a15ddce267f2)
- Data as of 2026-08-30T08:39:29.467469+00:00.