ELKI

ELKI

Company type	Open-source software project
Industry	Data mining, Machine learning, Software development
Founded	July 2008 (2008-07)
Founder	Database systems research group at Ludwig Maximilian University of Munich under Professor Hans-Peter Kriegel
Successor	Continued at Technical University of Dortmund
Headquarters	Munich (original), Dortmund (current) , Germany
Area served	Worldwide (academic and research community)
Key people	Professor Hans-Peter Kriegel
Products	Data mining algorithms framework, visualization tools, index structures
Parent	Ludwig Maximilian University of Munich (original), Technical University of Dortmund (current)
Website	elki-project.github.io

ELKI (Environment for Developing KDD-Applications Supported by Index-Structures) is a data mining (knowledge discovery in databases, KDD) software framework designed primarily for academic research and educational purposes. Originally developed by the database systems research group at Ludwig Maximilian University of Munich in Germany under the leadership of Professor Hans-Peter Kriegel, the project has since been continued at the Technical University of Dortmund. ELKI specializes in creating and assessing sophisticated data mining algorithms, particularly focusing on their integration with database index structures.

Description

Implemented in Java, ELKI features a modular design that supports a wide range of core algorithms including clustering, outlier detection, and management of database indexes. Its object-oriented framework allows seamless combinations of various algorithms, data types, distance functions, indexing methods, and evaluation metrics. Leveraging Java's just-in-time compilation, ELKI ensures efficient execution of these combinations, enabling dependable performance benchmarking. The system’s design promotes component reuse while Java’s strong type checking minimizes coding errors.

As an open-source platform, ELKI serves researchers and students who require effective analysis of large unlabeled datasets. It incorporates specialized index structures to accelerate processing and facilitates straightforward incorporation of new techniques for comparative algorithm studies.

ELKI has found practical use in numerous data science applications, including clustering of sperm whale codas, phoneme grouping, anomaly detection within spaceflight operations, optimization of bike sharing logistics, and forecasting traffic patterns.

Objectives

The university-led initiative aims to support both teaching and research, with source code crafted to maximize extensibility, reusability, and runtime efficiency. Since algorithm performance heavily depends on environmental conditions and implementation specifics, ELKI provides a unified codebase with standardized algorithm implementations for fair comparison.

Currently intended as a research utility, ELKI does not offer integration with business intelligence platforms or connectivity to common database management systems through SQL. Its copyleft license under the AGPL may restrict commercial adoption but permits thorough evaluation of algorithms before committing to proprietary development. Users are expected to have a strong understanding of algorithm parameters and foundational literature; thus the framework primarily addresses students, researchers, data scientists, and software engineers.

Architecture

At its core, ELKI adopts a database-inspired architecture employing vertical data storage that organizes information into column groups akin to NoSQL column families. This foundation supports operations such as nearest neighbor searches, range or radius queries, and distance computations accelerated through various indexing techniques across multiple dissimilarity measures. Algorithms like the k-nearest-neighbor algorithm, local outlier factor, and DBSCAN leverage this indexing for improved performance.

The database core also provides fast and memory-efficient collection types for handling objects and associative data structures like nearest neighbor lists.

Extensive use of Java interfaces allows developers to extend ELKI effortlessly by adding custom data types, distance functions, index implementations, algorithms, input parsers, or output modules without altering existing code. Custom distance measures can take advantage of pre-existing indexes for enhanced speed.

ELKI uses a service loader mechanism to distribute extensions as separate jar file packages.

Performance gains are achieved through optimized collection implementations rather than relying solely on standard Java APIs. For example, ELKI applies iterator patterns inspired by C++ loops that minimize memory consumption by internally utilizing primitive values, thereby reducing overhead from garbage collection. Similar strategies are employed in libraries such as GNU Trove3, Koloboke, and fastutil. Additionally, ELKI incorporates optimized data structures including object collections and heaps specifically designed for efficient nearest neighbor search operations.

Visualization

The visualization component employs SVG to generate graphics that can be scaled without loss of quality. It utilizes Apache Batik for rendering the graphical user interface and enables lossless export to formats such as PostScript and PDF, which supports integration into academic documents like those created with LaTeX. Exported graphics remain editable through SVG-compatible editors like Inkscape, and styling adjustments are simplified through the application of cascading style sheets. Despite these advantages, Batik's relatively slow rendering speed and significant memory demands restrict its ability to handle very large datasets; therefore, only a subset of data is visualized by default when dealing with extensive information.

Awards

At the 2011 "Symposium on Spatial and Temporal Databases," version 0.4 was showcased in a session highlighting spatial outlier detection techniques and earned the conference’s "best demonstration paper award."

Included algorithms

Highlighted algorithms included:

• Cluster analysis:
  • K-means clustering featuring accelerated variants such as Elkan, Hamerly, Annulus, Exponion, and robust k-means--
  • K-medians clustering
  • K-medoids clustering (PAM) with implementations like FastPAM and approximation methods including CLARA and CLARANS
  • Expectation-maximization algorithm applied to Gaussian mixture models
  • Hierarchical clustering methods including fast SLINK, CLINK, NNChain, and Anderberg algorithms
  • Single-linkage clustering
  • Leader clustering technique
  • DBSCAN enhanced with complete index acceleration supporting arbitrary distance metrics
  • OPTICS variants such as OPTICS-OF, DeLi-Clu, HiSC, HiCO, and DiSH
  • HDBSCAN
  • Mean-shift clustering approach
  • BIRCH clustering algorithm
  • SUBCLU (Density-Connected Subspace Clustering)
  • CLIQUE, ORCLUS, PROCLUS subspace clustering algorithms
  • COPAC, ERiC, 4C clustering methods
  • CASH clustering algorithm
  • DOC and FastDOC for subspace clustering
  • P3C clustering technique
  • Canopy clustering algorithm
• Anomaly detection:
  • k-Nearest-Neighbor based outlier identification
  • LOF
  • LoOP (Local Outlier Probabilities)
  • OPTICS-OF variant for outlier detection
  • DB-Outlier (Distance-Based Outliers)
  • LOCI (Local Correlation Integral)
  • LDOF (Local Distance-Based Outlier Factor)
  • EM-Outlier detection method
  • SOD (Subspace Outlier Degree)
  • COP (Correlation Outlier Probabilities)
• Frequent Itemset Mining and association rule learning:
  • Apriori algorithm
  • Eclat algorithm
  • FP-growth method
• Dimensionality reduction:
  • Principal component analysis
  • Multidimensional scaling
  • T-distributed stochastic neighbor embedding (t-SNE)
• Spatial index structures and search indexing techniques:
  • R-tree, R*-tree, M-tree, k-d tree, X-tree
  • Cover tree structure
  • iDistance indexing method
  • NN descent algorithm
  • Locality sensitive hashing (LSH)
• Evaluation metrics:
  • Precision and recall, F1 score, Average Precision measures
  • Receiver operating characteristic (ROC curve)
  • Discounted cumulative gain including NDCG variants
  • Silhouette index
  • Davies–Bouldin index
  • Dunn index
  • Density-based cluster validation (DBCV)
• Visualization tools:
  • Scatter plot creation
  • Histogram generation
  • Parallel coordinates visualization including 3D rendering via OpenGL
• Additional features:
  • Statistical distributions alongside numerous parameter estimators such as robust MAD and those based on L-moment
  • Techniques for Dynamic time warping
  • Methods for detecting change points in time series data
  • Estimators for intrinsic dimensionality

Version history

Version 0.1 (July 2008) marked the introduction of multiple algorithms derived from cluster analysis and anomaly detection, alongside index structures such as the R*-tree, with an emphasis on subspace clustering and correlation clustering techniques.

In Version 0.2 (July 2009), capabilities for time series analysis were incorporated, including specialized distance functions tailored for time series data.

By Version 0.3 (March 2010), the collection of anomaly detection methods was broadened, and enhancements were made to the visualization components.

Version 0.4 (September 2011) brought in algorithms targeting geographic data mining, as well as extended support for multi-relational databases and additional index structures.

The focus of Version 0.5 (April 2012) shifted towards assessing cluster analysis outcomes, featuring novel visualization tools and new algorithm implementations.

With Version 0.6 (June 2013), a three-dimensional version of parallel coordinates was introduced to improve data visualization, complemented by further algorithmic additions and indexing capabilities.

Version 0.7 (August 2015) expanded support to encompass uncertain data types along with corresponding analytic algorithms.

In Version 0.7.5 (February 2019), the library was augmented with a wider array of clustering and anomaly detection algorithms, new evaluation metrics, and advanced indexing structures.

Finally, Version 0.8 (October 2022) incorporated features such as automated index generation, garbage collection, incremental priority search, and numerous fresh algorithms including BIRCH.

Similar applications

• scikit-learn: A machine learning toolkit developed in Python
• Weka: A project from the University of Waikato centered on classification methods
• RapidMiner: A commercial software offering with a partially open source edition
• KNIME: An open source environment combining tools for machine learning and data mining

See also

• Comparison of statistical packages

External links

• – ELKI download and documentation