cgarciae/pypeln

Health v2 · maintenance only

Adoption not part of the score

1596 stars · 94 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Pypeln is a Python library for building concurrent, multi-stage data pipelines using processes, threads, or asyncio tasks through a single functional API. It targets medium-sized data workloads where heavyweight frameworks like Spark or Dask are overkill.

Use cases

• parallelize map/filter over large iterables
• build multi-stage concurrent data pipelines in Python
• process medium data with bounded memory using maxsize
• run IO-bound tasks concurrently with asyncio tasks
• speed up CPU-bound transformations with multiprocessing
• control worker counts per pipeline stage

When to choose

• you need concurrent pipelines for medium data without Spark/Dask overhead
• you want one API spanning processes, threads, and asyncio
• you need fine-grained control over workers and queue sizes per stage

When to avoid

• you need distributed computing across a cluster
• your data is small enough that plain Python is fast enough
• you need a maintained project with active development

Facets

library · maturity maintenance

concurrency streaming etl developer-tools developer-tools python cross-platform data-pipelines parallel-processing asyncio multiprocessing functional-api data-engineering automation algorithms

Member repositories

For agents

markdown · JSON · MCP: product_card(name="cgarciae/pypeln")