ShannonAI/service-streamer

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1241 stars · 187 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Service Streamer is a Python middleware that queues discrete web service requests into mini-batches for deep learning model inference, improving GPU utilization. It works with any web framework and deep learning framework, including multi-GPU setups.

Use cases

• serve BERT inference behind a web API with high throughput
• batch concurrent user requests into mini-batches for GPU inference
• deploy a PyTorch or TensorFlow model as a low-latency online service
• scale model inference across multiple GPUs
• reduce latency of deep learning web services under concurrent load

When to choose

• you serve a deep learning model via a web service and GPUs are underutilized by one-request-at-a-time handling
• you want dynamic batching with minimal code changes
• you need multi-GPU request distribution for inference

When to avoid

• your model inference is CPU-bound or not batch-friendly
• you need a full-featured model server with versioning and monitoring
• you require actively maintained software with recent updates

Facets

library · maturity maintenance

middleware machine-learning llm-inference streaming concurrency deep-learning machine-learning web-development backend python model-serving dynamic-batching gpu-utilization mini-batching inference-serving natural-language-processing linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ShannonAI/service-streamer")