Apache SystemDS

Apache SystemDS
Company type	Open source
Industry	Machine learning software
Predecessor	Apache SystemML
Founded	June 15, 2015; 11 years ago
Founder	IBM Almaden Research Center, Shivakumar Vaithyanathan
Area served	Worldwide
Key people	Shivakumar Vaithyanathan (IBM Fellow)
Products	Machine learning platform supporting full data science workflow
Owner	Apache Software Foundation
Parent	Apache Software Foundation
Website	github.com/apache/systemds

Adapted from the English Wikipedia article Apache SystemDS, available under CC BY-SA 4.0Updated 9 August 2025

Apache SystemDS (previously known as Apache SystemML) is an open-source platform designed for machine learning that supports the complete data science workflow from start to finish.

Notable capabilities include:

Customizable algorithms using languages similar to R and Python.
Diverse execution environments: Standalone mode, Spark Batch, Spark MLContext, Hadoop Batch, and JMLC.
Intelligent optimization tailored to data properties and cluster setup to enhance performance and scalability.

History

SystemML originated in 2010 through efforts by researchers at IBM Almaden Research Center, under the leadership of IBM Fellow Shivakumar Vaithyanathan. The initiative aimed to solve the difficulty data scientists encountered when scaling machine learning models from small-scale environments (R or Python) to large-scale big data platforms, which often required manual rewriting in languages such as Scala—a process prone to errors and time-intensive. SystemML automated this transition, allowing algorithms developed in R-like or Python-like syntax to be executed efficiently on large datasets without manual code conversion.

IBM publicly announced the open-source release of SystemML on June 15, 2015, during the Spark Summit, emphasizing its dedication to Apache Spark and associated technologies. The source code was made accessible on GitHub by August 27, 2015, entered the Apache Incubator on November 2, 2015, and ultimately graduated to an Apache Top Level Project status on May 17, 2017.

Key technologies

SystemDS integrates innovative techniques including:

Compressed Linear Algebra techniques for scalable machine learning processes
A Declarative Machine Learning Language that provides flexible algorithm definition

Principal Component Analysis

The example below demonstrates performing Principal component analysis on an input matrix <math>A</math>, yielding both <math>eigen vectors</math> and <math display="inline">eigen values</math>.<syntaxhighlight lang="r" line="1">

PCA.dml

N = nrow(A); D = ncol(A);

z-score normalization (centering and scaling)

A = scale(A, center==1, scale==1);

covariance matrix calculation

mu = colSums(A)/N; C = (t(A) %*% A)/(N-1) - (N/(N-1))*t(mu) %*% mu;

computation of eigen vectors and eigen values

[evalues, evectors] = eigen(C); </syntaxhighlight>

Invocation script

<syntaxhighlight lang="shell-session"> spark-submit SystemDS.jar -f PCA.dml -nvargs INPUT=INPUT_DIR/pca-1000x1000 \

 OUTPUT=OUTPUT_DIR/pca-1000x1000-model PROJDATA=1 CENTER=1 SCALE=1

</syntaxhighlight>

Database functions

DBSCAN clustering utilizing Euclidean distance.<syntaxhighlight lang="r" line="1"> X = rand(rows=1780, cols=180, min=1, max=20) [indices, model] = dbscan(X = X, eps = 2.5, minPts = 360) </syntaxhighlight>

Improvements

With the release of SystemDS 2.0.0—the first major update under its new name—the system underwent extensive restructuring and introduced enhanced features along with improved support for comprehensive data science workflows. Legacy functionalities were also deprecated.

Introduction of a new framework for DML-script-level builtin functions; expanded built-in operations for data preprocessing such as cleaning, augmentation, feature engineering; inclusion of novel machine learning algorithms and tools for model debugging.
Enhanced data cleaning capabilities including multivariate imputation via chained equations (MICE), SMOTE oversampling addressing class imbalance issues, forward/backward filling of missing values (NA), schema-driven cleaning methods, outlier identification through standard deviation and interquartile range metrics, plus functional dependency detection.
Extensive lineage tracing with reuse strategies encompassing loop deduplication, full and partial reuse options supported by compiler optimizations and novel rewrite techniques aimed at maximizing reuse efficiency.
Federated runtime backend enabling federated matrices/frames alongside federated builtin functions like transform-encode and decode.
Overhauled compression module featuring quantization for lossy compression schemes, binary cell operations support, and experimental left matrix multiplication.
New Python API bindings facilitating multiple builtin functions, matrix operations, federated tensor handling, and lineage trace management.
CUDA-based implementations of cumulative aggregate operators such as cumsum and cumprod.
Improvements in model debugging tools including a slice finder utility.
Experimental support for tensor data models accommodating basic tensors with diverse value types alongside schema-based tensors.
Deployment scripts tailored for AWS cloud environments and federated operation configurations.
Performance enhancements achieved through parallel sorting algorithms, GPU-accelerated cumulative aggregates, and optimized append column-bind operations.
Compiler and runtime system improvements featuring advanced rewrites, decreased overhead in Spark context initialization, a new evaluation framework, list operation support, and updated native kernel libraries.
Added support for JSON frame reading/writing as well as SQL data source integration.
Additional miscellaneous upgrades covering documentation quality, testing procedures, release automation scripts, packaging improvements; availability of Docker containers; introduction of lambda expression support; and various bug fixes.
Removal of the MapReduce compiler/runtime backend along with the pydml parser, Java-UDF framework, and script-level debugger.
Deprecation of the ./scripts/algorithms directory as algorithm implementations migrate into SystemDS builtins.

Contributions

Apache SystemDS welcomes participation in areas such as software development, community involvement, and advocacy. Detailed instructions for contributors can be found at https://github.com/apache/systemds/blob/main/CONTRIBUTING.md

External links