FMInference/FlexLLMGen

Health v2 · maintenance only

Flags: no_releases archived

Adoption not part of the score

9352 stars · 591 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

FlexLLMGen is a high-throughput generation engine for running large language models on a single GPU with limited memory. It uses IO-efficient offloading, compression, and large effective batch sizes to maximize tokens-per-second for throughput-oriented batch inference workloads.

Use cases

• run large language model inference on a single consumer GPU
• classify or extract information from millions of documents in batches
• benchmark LLMs like HELM on limited hardware
• process a company's private document corpus with an LLM overnight
• run OPT or GPT-style models with limited GPU memory
• reduce inference costs by trading latency for throughput

When to choose

• you need to process large batches of text and throughput matters more than latency
• you only have a single commodity GPU but want to run large models
• you are running offline batch jobs like data wrangling or benchmarking

When to avoid

• you need low-latency interactive chat responses
• you have enough powerful GPUs to hold the whole model in memory
• you need small-batch or real-time inference

Facets

library · maturity maintenance

llm-inference machine-learning gpu-computing large-language-models deep-learning machine-learning python offloading high-throughput single-gpu model-compression batch-inference gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="FMInference/FlexLLMGen")