PaddlePaddle/FastDeploy

Health v2 · maintenance only

Adoption not part of the score

3711 stars · 759 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

FastDeploy is a high-performance inference and deployment toolkit for large language models and vision-language models built on PaddlePaddle. It provides production-ready serving with an OpenAI-compatible API, vLLM interface compatibility, PD disaggregation, KV cache transmission, and broad quantization format support across multiple hardware backends.

Use cases

• serve an OpenAI-compatible API for ERNIE 4.5 models
• deploy LLMs with vLLM-compatible serving on NVIDIA GPUs
• run quantized LLM inference with W4A8 or FP8 weights
• deploy vision-language models like ERNIE-4.5-VL or Qwen-VL
• set up prefill-decoding disaggregation for high-throughput serving
• serve LLMs on Kunlunxin XPU, Ascend NPU, or Hygon DCU hardware
• enable speculative decoding and multi-token prediction for faster generation

When to choose

• you need to deploy ERNIE 4.5 family models with first-class support
• you want vLLM-style serving on non-NVIDIA accelerators like Kunlunxin or Ascend
• you need industrial-grade PD disaggregation with load balancing and context caching
• you require wide quantization format support for memory-constrained serving

When to avoid

• you need broad community support and ecosystem plugins of vLLM or SGLang
• your models are not in the supported list and you need flexible custom architectures
• you want a simple local inference library rather than a serving deployment toolkit
• you develop primarily on macOS or Windows without supported accelerator hardware

Facets

library · maturity active

llm-inference http-server gpu-computing machine-learning large-language-models artificial-intelligence deep-learning python llm-serving vllm-compatible openai-api pd-disaggregation quantization vlm paddlepaddle kv-cache speculative-decoding linux nvidia-gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="PaddlePaddle/FastDeploy")