apache/systemds

Health v2 · maintenance only

Adoption not part of the score

1097 stars · 546 forks observed · 2026-09-01

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Apache SystemDS is an open-source machine learning system covering the end-to-end data science lifecycle, from data cleaning and feature engineering to distributed model training and serving. Scripts written in an R-like DML language or via Python/Java APIs are compiled into hybrid runtime plans combining local in-memory CPU/GPU operations with distributed execution on Apache Spark.

Use cases

• train machine learning models on large datasets with Spark
• write ML pipelines in an R-like scripting language
• run distributed matrix computations for data science
• prototype ML algorithms and scale them to a cluster
• clean and prepare data before model training
• serve trained ML models in production

When to choose

• you need scalable ML on large datasets with Spark integration
• you want algorithm customizability via a high-level R-like or Python-like language
• you need hybrid local and distributed execution with automatic optimization

When to avoid

• you need deep learning with modern GPU frameworks like PyTorch or TensorFlow
• you want a large ecosystem of pretrained models and community libraries
• your datasets fit in memory and simpler tools like scikit-learn suffice

Facets

library · maturity active

machine-learning data-science etl compiler machine-learning data-science big-data microservices jvm python cross-platform dml apache distributed-ml spark federated-learning r-like-syntax gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="apache/systemds")