yahoo/TensorFlowOnSpark

Health v2 · maintenance only

Adoption not part of the score

3845 stars · 937 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

TensorFlowOnSpark is a Python library that lets existing TensorFlow programs run distributed training and inference on Apache Spark and Hadoop clusters with minimal code changes. It manages launching TensorFlow workers on Spark executors, data ingestion from HDFS or Spark RDDs, and cluster shutdown.

Use cases

• run distributed TensorFlow training on a Spark cluster
• train deep learning models on data stored in HDFS
• migrate existing TensorFlow code to a shared Hadoop grid
• perform distributed inference across GPU and CPU servers
• integrate TensorFlow pipelines with Spark data processing
• scale deep learning workloads across a cluster of machines

When to choose

• you already run Spark/Hadoop clusters and want TensorFlow training on them
• you want to migrate existing TensorFlow programs to distributed execution with minimal changes
• your training data lives in HDFS or Spark RDDs
• you need GPU or CPU distributed training on cloud or on-premise clusters

When to avoid

• you use modern orchestration like Kubernetes-native TensorFlow or PyTorch distributed
• your workloads are small enough for a single machine
• you need a project with frequent updates and active development
• you use frameworks other than TensorFlow

Facets

library · maturity maintenance

machine-learning deep-learning llm-training machine-learning deep-learning big-data microservices python jvm cloud tensorflow apache-spark hadoop distributed-training gpu hdfs linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="yahoo/TensorFlowOnSpark")