spotify/scio

Health v2 · maintenance only

Adoption not part of the score

2628 stars · 533 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Scio is a Scala API for Apache Beam and Google Cloud Dataflow, inspired by Apache Spark and Scalding. It provides a unified batch and streaming programming model with type-safe integrations for Google Cloud products and various data formats.

Use cases

• write batch data pipelines in Scala on Apache Beam
• run streaming data processing jobs on Google Cloud Dataflow
• read and write BigQuery tables in a type-safe way
• process Avro, Parquet, and Protobuf data at scale
• integrate pipelines with Pub/Sub, Bigtable, and Cloud Storage
• run machine learning pipelines with TensorFlow IO
• develop and test data pipelines interactively with a Scala REPL

When to choose

• your team works in Scala and needs a Spark-like API on Beam
• you run data pipelines on Google Cloud Dataflow
• you need unified batch and streaming processing with strong typing
• you want deep integration with Google Cloud data products like BigQuery

When to avoid

• you prefer Python or Java directly over Scala
• you need a runner or cloud other than Beam-supported ones without extra work
• your pipelines are small enough that a simpler tool suffices

Facets

framework · maturity active

etl streaming data-science machine-learning serialization big-data data-science cloud-computing microservices jvm cloud apache-beam google-cloud-dataflow bigquery batch-processing scala-api data-pipelines data-engineering

Member repositories

For agents

markdown · JSON · MCP: product_card(name="spotify/scio")