Neroued/ninfer

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1252 stars · 241 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

NInfer is a from-scratch C++/CUDA inference engine optimized for maximum single-GPU performance on a narrow, explicitly registered set of Qwen model checkpoints, targeting an NVIDIA RTX 5090. It supports text, image, and video prompts via a local CLI and OpenAI-/Anthropic-compatible HTTP APIs.

Use cases

• run Qwen models at maximum speed on a single RTX 5090
• serve Qwen checkpoints through an OpenAI-compatible API
• run multimodal image and video prompts locally
• decode with high concurrency using MTP and CUDA graphs
• use NVFP4-quantized model weights for faster inference
• self-host a minimal LLM inference server

When to choose

• you own an RTX 5090 and want peak single-GPU throughput for supported Qwen checkpoints
• you want a lightweight, purpose-built engine instead of a general model runtime
• you need OpenAI- or Anthropic-compatible local serving endpoints
• you want multimodal (text, image, video) prompting with quantized weights

When to avoid

• you need to run arbitrary models or checkpoints outside the registered set
• you have a different GPU than the targeted NVIDIA hardware
• you need multi-GPU or distributed inference
• you want broad ecosystem compatibility like vLLM or llama.cpp

Facets

library · maturity active

llm-inference http-server cli gpu-computing large-language-models machine-learning developer-tools cpp cli cuda qwen single-gpu openai-compatible-api anthropic-compatible-api quantization nvfp4 rtx-5090 multimodal mtp linux gpu web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Neroued/ninfer")