Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

A Beginner’s Guide to Spark Streaming For Data Engineers

S
Siddharth Sonkar Last Updated : 22 Dec, 2023
8 min read

Overview

  • Understand Spark Streaming and its functioning.
  • Learn about Windows in Spark Streaming with an example.

Introduction

According to IBM, 60% of all sensory information loses value in a few milliseconds if it is not acted on. Bearing in mind that the Big Data and analytics market has reached $125 billion and a large chunk of this will be attributed to IoT in the future, the inability to tap real-time information will result in a loss of billions of dollars.

Examples of some of these applications include a telco, working out how many of its users have used Whatsapp in the last 30 minutes, a retailer keeping track of the number of people who have said positive things about its products today on social media, or a law enforcement agency looking for a suspect using data from traffic CCTV.

This is the primary reason stream-processing systems like Spark Streaming will define the future of real-time analytics. There is also a growing need to analyze both data at rest and data in motion to drive applications, which makes systems like Spark—which can do both—all the more attractive and powerful. It’s a system for all Big Data seasons.

You learn how Spark Streaming not only keeps the familiar Spark API intact but also, under the hood, uses RDDs for storage as well as fault-tolerance. This enables Spark practitioners to jump into the streaming world from the outset. With that in mind, let’s get right to it.

Spark Streaming introduction
An Introduction to Spark Streaming | by Harshit Agarwal | Medium

Table of contents

Apache Spark

Apache Spark is a unified computing engine and a set of libraries for parallel data processing on computer clusters. As of this writing, Spark is the most actively developed open-source engine for this task, making it a standard tool for any developer or data scientist interested in big data.

Spark supports multiple widely-used programming languages (Python, Java, Scala, and R), includes libraries for diverse tasks ranging from SQL to streaming and machine learning, and runs anywhere from a laptop to a cluster of thousands of servers. This makes it an easy system to start with and scale-up to big data processing or an incredibly large scale. Below are a few of the features of Spark:

  • Fast and general-purpose engine for large-scale data processing

    • Not a modified version of Hadoop
    • The leading candidate for “successor to Map Reduce”

  • Spark can efficiently support more types of computations

    • For example, interactive queries, stream processing
  • Can read/write to any Hadoop-supported system (e.g., HDFS)
  • Speed: in-memory data storage for very fast iterative queries
    • the system is also more efficient than MapReduce for complex applications running on disk
    • up to 40x faster than Hadoop
    • Ingest data from many sources: Kafka, Twitter, HDFS, TCP sockets
    • Results can be pushed out to file-systems, databases, live dashboards, but not only
Spark Streaming

Apache Spark Ecosystem

The following are the components of Apache Spark Ecosystem-

  • Spark Core: basic functionality of Spark (task scheduling, memory management, fault recovery, storage systems interaction).
  • Spark SQL: package for working with structured data queried via SQL as well as HiveQL
  • Spark Streaming: a component that enables processing of live streams of data (e.g., log files, status updates messages)
  • MLLib: MLLib is a machine learning library like Mahout. It is built on top of Spark and has the provision to support many machine learning algorithms.
  • GraphX: For graphs and graphical computations, Spark has its own Graph Computation Engine, called GraphX. It is similar to other widely used graph processing tools or databases, like Neo4j, Giraffe, and many other distributed graph databases.
Spark Streaming - ecosystem

Spark Streaming: Abstractions

Spark Streaming has a micro-batch architecture as follows:

  • treats the stream as a series of batches of data
  • new batches are created at regular time intervals
  • the size of the time intervals is called the batch interval
  • the batch interval is typically between 500 ms and several seconds
Spark Streaming, Abstractions

The reduce value of each window is calculated incrementally.

Discretized Stream (DStream)

Discretized Stream or DStream is the basic abstraction provided by Spark Streaming. It represents a continuous stream of data, either the input data stream received from the source or the processed data stream generated by transforming the input stream. Internally, a DStream is represented by a continuous series of RDDs, which is Spark’s abstraction of an immutable, distributed dataset (see Spark Programming Guide for more details). Each RDD in a DStream contains data from a certain interval.

Spark Streaming - DStream
  • RDD transformations are computed by the Spark engine
  • the DStream operations hide most of these details
  • Any operation applied on a DStream translates to operations on the underlying RDDs
  • The reduce value of each window is calculated incrementally.
DStream

Spark Streaming: Streaming Context

It is the main entry point for Spark Streaming functionality. It provides methods used to create DStreams from various input sources. Streaming Spark can be either created by providing a Spark master URL and an appName, or from an org.apache.spark.SparkConf configuration, or from an existing org.apache.spark.SparkContext. The associated SparkContext can be accessed using context.sparkContext.

After creating and transforming DStreams, streaming computation can be started and stopped using context.start() and, respectively. context.awaitTermination() allows the current thread to wait for the termination of the context by stop() or by an exception.

To execute a SparkStreaming application, we need to define the StreamingContext. It specializes SparkContext for streaming applications.

Streaming context in Java can be defined as follows-

JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, batchInterval);

where:

  • master is a Spark, Mesos, or YARN cluster URL; to run your code in local mode, use “local[K]” where K>=2 represents the parallelism
  • appname is the name of your application
  • batch interval time interval (in seconds) of each batch

Once built, they offer two types of operations:

  • Transformations which yield a new DStream from a previous one. For example, one common transformation is filtering data.

    • stateless transformations: the processing of each batch does not depend on the data of its previous batches.

                      Examples are: map(), filter(), and reduceByKey()

  • stateful transformations: use data from previous batches to compute the results of the current batch. They include sliding windows, tracking state across time, etc
  • Output operations that write data to an external system. Each streaming application has to define an output operation.

Note that a streaming context can be started only once, and must be started after we set up all the DStreams and output operations.

Basic Data Sources

Below listed are the basic data sources of Spark Streaming:

  • File Streams: It is used for reading data from files on any file system compatible with the HDFS API (that is, HDFS, S3, NFS, etc.), a DStream can be created as: 
    ... = streamingContext.fileStream<...>(directory);
  • Streams based on Custom Receivers: DStreams can be created with data streams received through custom receivers, extending the Receiver<T> class 
    ... = streamingContext.queueStream(queueOfRDDs)
  • Queue of RDDs as a Stream: For testing a Spark Streaming application with test data, one can also create a DStream based on a queue of RDDs, using 
    ... = streamingContext.queueStream(queueOfRDDs)

Most of the transformations have the same syntax as the one applied to RDDs

Transformation

Meaning

map(func)

Return a new DStream by passing each element of the source DStream through a function func.

flatMap(func)

Similar to map, but each input item can be mapped to 0 or more output items.

filter(func)

Return a new DStream by selecting only the records of the source DStream on which func returns true.

union(otherStream)

Return a new DStream that contains the union of the elements in the source DStream and otherDStream.

join(other Stream)

When called on two DStreams of (K, V) and (K, W) pairs, return a new DStream of (K, (V, W)) pairs with all pairs of elements for each key.

Example: Word Count-

SparkConf sparkConf = new SparkConf()
.setMaster("local[2]").setAppName("WordCount");
JavaStreamingContext ssc = ...
JavaReceiverInputDStream<String> lines = ssc.socketTextStream( ... );
JavaDStream<String> words = lines.flatMap(...);
JavaPairDStream<String, Integer> wordCounts = words
                                             .mapToPair(s -> new Tuple2<>(s, 1))
                                             .reduceByKey((i1, i2) -> i1 + i2);

wordCounts.print();

Spark Streaming: Window

The simplest windowing function is a window, which lets you create a new DStream, computed by applying the windowing parameters to the old DStream. You can use any of the DStream operations on the new stream, so you get all the flexibility you want.

Windowed computations allow you to apply transformations over a sliding window of data. Any window operation needs to specify two parameters:

  • window length
    • The duration of the window in secs
  • sliding interval
    • The interval at which the window operation is performed in secs
    • These parameters must be multiples of the batch interval
DStream
window(windowLength, slideInterval)

It returns a new DStream which is computed based on windowed batches.

...
JavaStreamingContext ssc = ...
JavaReceiverInputDStream<String> lines = ...
JavaDStream<String> linesInWindow =
lines.window(WINDOW_SIZE, SLIDING_INTERVAL);
JavaPairDStream<String, Integer> wordCounts = linesInWindow.flatMap(SPLIT_LINE)
.mapToPair(s -> new Tuple2<>(s, 1))
.reduceByKey((i1, i2) -> i1 + i2);
  • reduceByWindow(func, InvFunc, windowLength, slideInterval)
    • Return a new single-element stream, created by aggregating elements in the stream over a sliding interval using func (which should be associative).
    • The reduce value of each window is calculated incrementally.
    • func reduces new data that enters the sliding window
    • invFunc “inverse reduces” the old data that leaves the window.
  • reduceByKeyAndWindow(func, InvFunc, windowLength, slideInterval)
    • When called on a DStream of (K, V) pairs, returns a new DStream of (K, V) pairs where the values for each key are aggregated using the given reduce function func over batches in a sliding window.

For performing these transformations, we need to define a checkpoint directory

Window-based – Word Count

...
JavaPairDStream<String, Integer> wordCountPairs = ssc.socketTextStream(...)
.flatMap(x -> Arrays.asList(SPACE.split(x)).iterator())
.mapToPair(s -> new Tuple2<>(s, 1));
JavaPairDStream<String, Integer> wordCounts = wordCountPairs
.reduceByKeyAndWindow((i1, i2) -> i1 + i2, WINDOW_SIZE, SLIDING_INTERVAL);
wordCounts.print();
wordCounts.foreachRDD(new SaveAsLocalFile());

A (more efficient) Window-based – Word Count

In a more efficient version, the reduce value of each window is calculated incrementally-

  • a reduce function handles new data that enters the sliding window;
  • an “inverse reduce” function handles old data that leaves the window.

Note that checkpointing must be enabled for using this operation.

...
ssc.checkpoint(LOCAL_CHECKPOINT_DIR);
...
JavaPairDStream<String, Integer> wordCounts = wordCountPairs.reduceByKeyAndWindow(
(i1, i2) -> i1 + i2,
(i1, i2) -> i1 - i2, WINDOW_SIZE, SLIDING_INTERVAL);

Spark Streaming: Output Operations

Output operations allow DStream’s data to be pushed out to external systems like a database or a file systems

Output Operation

Meaning

print()

Prints the first ten elements of every batch of data in a DStream on the driver node running the application.

saveAsTextFiles(prefix, [suffix])

Save this DStream’s contents as text files. The file name at each batch interval is generated based on prefix.

saveAsHadoopFiles(prefix, [suffix])

Save this DStream’s contents as Hadoop files.

saveAsObjectFiles(prefix, [suffix])

Save this DStream’s contents as SequenceFiles of serialized Java objects.

foreachRDD(func)

Generic output operator that applies a function, func, to each RDD generated from the stream.

Online References-
• Spark Documentation
• Spark Documentation

Conclusion

It should be clear that Spark Streaming presents a powerful way to write streaming applications. Taking a batch job you already run and turning it into a streaming job with almost no code changes is both simple and extremely helpful from an engineering standpoint if you need to have this job interact closely with the rest of your data processing application.

I recommend you go through the following data engineering resources to enhance your knowledge-

Frequently Asked Questions

Q1.How does Spark Streaming differ from batch processing?

Spark Streaming processes data in small, configurable micro-batches, providing low-latency processing compared to traditional batch processing.

Q2.What types of data sources are supported by Spark Streaming?

Spark Streaming supports various data sources, including HDFS, Kafka, Flume, and others, allowing seamless integration with diverse streaming platforms.

Q3.Can Spark Streaming handle both data at rest and data in motion?

Yes, Spark Streaming can analyze both data at rest (static data) and data in motion (live data streams), making it versatile for different use cases.

If you liked the article then please drop a comment in the comment section below.

dstreamSpark Streamingword count
S
Siddharth Sonkar

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
Introduction to Spark Streaming
Introduction to Spark Streaming
Real-time Data Streaming using Apache Spark!
Real-time Data Streaming using Apache Spark!

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details