piskvorky/smart_open

Health v2 · maintenance only

Adoption not part of the score

3457 stars · 390 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A Python library for efficiently streaming very large files to and from remote storages like S3, GCS, Azure Blob Storage, HDFS, WebHDFS, HTTP, and SFTP, with transparent on-the-fly compression/decompression. It is a drop-in replacement for Python's built-in open() with a unified Pythonic API.

Use cases

• stream large files from s3 in python
• read gzip files line by line without extracting
• replace boto3 upload_fileobj boilerplate
• write files to hdfs or webhdfs from python
• stream data over http or sftp like a local file
• convert between compressed file formats on the fly
• process huge remote datasets without downloading them fully

When to choose

• you need a unified open()-like API across S3, GCS, Azure, HDFS, HTTP, and SFTP
• you want transparent gzip/bz2/zstd compression while streaming
• you're tired of boto3 file-like object boilerplate for large files

When to avoid

• you only work with local files and don't need remote storage or compression
• you need fine-grained control over multipart upload tuning that boto3 exposes directly
• you need a non-Python solution

Facets

library · maturity stable

file-system compression http-client streaming etl files cloud-computing developer-tools python cross-platform s3 gcs azure-blob hdfs sftp gzip bzip2 zstd drop-in-open-replacement remote-storage data-engineering

Member repositories

For agents

markdown · JSON · MCP: product_card(name="piskvorky/smart_open")