Apache Spark

Apache Spark
Company type	Open-source software project
Industry	Big data, data analytics, distributed computing
Founded	2009; 17 years ago
Founder	Matei Zaharia
Area served	Worldwide
Products	Apache Spark (analytics engine), Spark SQL, Spark Streaming, MLlib, GraphX
Owner	Apache Software Foundation
Website	spark.apache.org

Apache Spark is a unified analytics engine designed for processing large-scale data, available as open-source software. It provides an API that facilitates cluster programming with built-in fault tolerance and data parallelism. Initially developed at the University of California, Berkeley's AMPLab beginning in 2009, its source code was contributed to the Apache Software Foundation in 2013, which has since continued its evolution.

Overview

At the heart of Apache Spark’s design lies the resilient distributed dataset (RDD), a fault-tolerant and distributed collection of data elements.^[1] The DataFrame API was later introduced as a higher-level abstraction over RDDs, followed by the Dataset API. While RDDs served as the main API during Spark 1.x versions, Spark 2.x promotes the use of Datasets for improved efficiency, although RDD support remains intact.<ref> The Dataset API is fundamentally powered by the underlying RDD technology.<ref>

RDDs were developed to overcome the constraints of the MapReduce model, which enforces a linear dataflow structure and depends heavily on disk input/output. Spark’s RDD abstraction offers a distributed shared memory model with fault tolerance capabilities.<ref>

Spark organizes computational workflows as a directed acyclic graph (DAG), where nodes correspond to RDDs and edges denote operations performed on them. This architecture enables efficient execution of iterative algorithms and interactive data analysis with much lower latency compared to Apache Hadoop MapReduce.<ref> The need to support iterative processes such as machine learning training significantly influenced Spark’s creation.<ref>

For running Spark applications, a cluster manager and distributed storage are necessary. Spark supports various cluster managers including standalone Spark clusters, Hadoop YARN, Apache Mesos, and Kubernetes.<ref> Standalone clusters can be launched either manually or using provided scripts; local mode is also available for development without requiring distributed storage. Regarding storage integration, Spark works with systems like Alluxio, Hadoop Distributed File System (HDFS), MapR File System, Apache Cassandra, OpenStack Swift, Amazon S3, Apache Kudu, and the Lustre file system, among others.<ref><ref><ref> Custom storage backends can be implemented as needed.

Spark Core

The foundation of Apache Spark is Spark Core, which handles distributed task scheduling, dispatching, and essential I/O functionality through APIs available for Java, Python, Scala, .NET, and R. Built around the RDD abstraction, its Java API extends support to other JVM languages and some non-JVM languages such as Julia. This API employs a functional and higher-order programming paradigm: a "driver" program applies parallel transformations like map, filter, or reduce on RDDs by passing functions to Spark, which then orchestrates execution across the cluster. These operations—including joins—take one or more RDDs as input and produce new immutable RDDs that are evaluated lazily. Fault tolerance is maintained by tracking each RDD’s lineage information, allowing lost data partitions to be recomputed when necessary. RDDs can contain objects from Python, .NET, Java, or Scala.

Besides functional transformations on RDDs, Spark also provides two forms of shared variables: broadcast variables for distributing read-only data across nodes efficiently and accumulators which facilitate imperative-style aggregations.

A typical Scala snippet demonstrates how to compute word counts from text files using map, flatMap, and reduceByKey functions that transform RDDs by applying simple operations on dataset elements.

<syntaxhighlight lang="scala" line> val conf = new SparkConf().setAppName("wiki_test") // initialize spark configuration val sc = new SparkContext(conf) // create spark context val data = sc.textFile("/path/to/somedir") // load files from directory into an RDD val tokens = data.flatMap(_.split(" ")) // split lines into words val wordFreq = tokens.map((_, 1)).reduceByKey(_ + _) // aggregate word counts wordFreq.sortBy(s => -s._2).map(x => (x._2, x._1)).top(10) // fetch top 10 most frequent words </syntaxhighlight>

Spark SQL

Spark SQL is a module built atop Spark Core that introduces DataFrames (previously known as SchemaRDDs before version 1.3), enabling the handling of both structured and semi-structured datasets. It provides a domain-specific language (DSL) for DataFrame manipulation available in Scala, Java, Python, and .NET environments. Additionally, Spark SQL supports conventional SQL queries through command-line tools and offers connectivity via ODBC and JDBC drivers. Starting with Spark 2.0, the DataSet API—strongly typed—augments DataFrames to enhance type safety.

<syntaxhighlight lang="scala"> import org.apache.spark.sql.SparkSession

val url = "jdbc:mysql://yourIP:yourPort/test?user=yourUsername;password=yourPassword" val spark = SparkSession.builder().getOrCreate()

val df = spark

 .read
 .format("jdbc")
 .option("url", url)
 .option("dbtable", "people")
 .load()

df.printSchema() val countsByAge = df.groupBy("age").count() </syntaxhighlight>

Alternatively, using SQL syntax: <syntaxhighlight lang="scala"> df.createOrReplaceTempView("people") val countsByAge = spark.sql("SELECT age, count(*) FROM people GROUP BY age") </syntaxhighlight>

Spark Streaming

Spark Streaming utilizes the rapid scheduling capabilities of Spark Core to execute streaming analytics by processing data in small batches through RDD transformations. This approach enables existing batch processing code to be adapted for streaming applications and supports implementations of lambda architecture. However, this mini-batch method introduces latency compared to true event-driven streaming systems like Storm or Apache Flink.

It offers native integration with various data sources, including Kafka, Flume, Twitter, ZeroMQ, AWS Kinesis, and TCP/IP sockets.

With the release of Spark 2.x, Structured Streaming emerged as a higher-level API based on Datasets for stream processing.

Spark can be deployed both on local infrastructure and cloud platforms, providing enterprises with flexible options for their data processing strategies.

MLlib machine learning library

Spark MLlib is a distributed machine learning library built on top of Spark Core. Utilizing Spark’s in-memory distributed computing model, MLlib achieves speeds up to nine times faster than disk-based solutions such as Apache Mahout (prior to Mahout’s integration with Spark) and offers superior scalability compared to Vowpal Wabbit. It delivers an extensive collection of machine learning and statistical algorithms designed to facilitate large-scale machine learning pipelines, including:

summary statistics, correlations, stratified sampling, hypothesis testing, and random data generation
classification and regression algorithms: support vector machines, logistic regression, linear regression, naive Bayes classification, decision trees, random forests, and gradient-boosted trees
Collaborative filtering methods such as alternating least squares (ALS)
clustering techniques including k-means and latent Dirichlet allocation (LDA)
Dimensionality reduction approaches like singular value decomposition (SVD) and principal component analysis (PCA)
Feature extraction along with data transformation utilities
Optimization algorithms including stochastic gradient descent and limited-memory BFGS (L-BFGS)

GraphX

GraphX is a graph-processing system distributed over Apache Spark. Constructed using immutable RDDs, it manages property graphs by associating attributes with both edges and vertices; however, it is not designed for transactional graph modifications commonly found in graph databases. It offers two distinct APIs to facilitate parallel algorithm execution: a Pregel-inspired model and a versatile MapReduce-like API, which together enable efficient scaling of graph computations such as PageRank.

Developed initially at UC Berkeley's AMPLab and by Databricks, GraphX was contributed to the Apache Software Foundation and incorporated into the Spark ecosystem. It succeeded the outdated Bagel framework, delivering improved capabilities for handling property graphs and performing extensive graph analytics.

Language support

Apache Spark provides built-in support for Scala, Java, SQL, R, and Python, while additional language bindings are available through third-party tools for environments like .NET CLR and Julia.

History

The inception of Spark dates back to 2009 when Matei Zaharia developed it at UC Berkeley's AMPLab. It was released as open source in 2010 under a BSD license. In 2013, Spark was donated to the Apache Software Foundation and transitioned to the Apache 2.0 license. By February 2014, it had achieved the status of a Top-Level Project within Apache.

In November 2014, Databricks, founded by Zaharia, set a new world record for large-scale sorting using Spark technology. By 2015, the project had attracted more than 1,000 contributors, establishing itself as one of the most active initiatives within the Apache Foundation and a premier open-source platform for big data processing.

Version	Initial release	Most recent version	Release date
Template:Version	2012-06-12	0.5.2	2012-11-22
Template:Version	2012-10-15	0.6.2	2013-02-07
Template:Version	2013-02-27	0.7.3	2013-07-16
Template:Version	2013-09-25	0.8.1	2013-12-19
Template:Version	2014-02-02	0.9.2	2014-07-23
Template:Version	2014-05-26	1.0.2	2014-08-05
Template:Version	2014-09-11	1.1.1	2014-11-26
Template:Version	2014-12-18	1.2.2	2015-04-17
Template:Version	2015-03-13	1.3.1	2015-04-17
Template:Version	2015-06-11	1.4.1	2015-07-15
Template:Version	2015-09-09	1.5.2	2015-11-09
Template:Version	2016-01-04	1.6.3	2016-11-07
Template:Version	2016-07-26	2.0.2	2016-11-14
Template:Version	2016-12-28	2.1.3	2018-06-26
Template:Version	2017-07-11	2.2.3	2019-01-11

Scala version

Spark 3.5.2 is constructed using Scala 2.13 and offers native support for both Scala 2.12 and 2.13, while also providing compatibility with Scala 3.

Developers

The development of Apache Spark is overseen by a community governed by the Project Management Committee (PMC).

Maintenance releases and EOL

Bug fixes are applied to feature release branches for approximately 18 months. For example, the 2.3.x branch reached its end-of-life in September 2019, after which no further updates were issued. Long-term support (LTS) versions, such as 2.4.8 released in May 2021, receive extended maintenance but ultimately reach end-of-life with no additional bug fixes planned.

External links

↑ Cite error: Invalid <ref> tag; no text was provided for refs named hc10

[hc10-1] Cite error: Invalid <ref> tag; no text was provided for refs named hc10

[1]