Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Apache Kafka: A Metaphorical Introduction to Event Streaming for Data Scientists and Data Engineers

Kaushikrch
Kaushikrch Last Updated : 08 Dec, 2020
7 min read

Overview

  • Learn about viewing data as streams of immutable events in contrast to mutable containers
  • Understand how Apache Kafka captures real-time data through event streaming

 

Introduction

Back in 2010, professional networking site LinkedIn faced inconsistency and latency issues on their existing request-response system. To overcome this, a team of software engineers at the company developed an optimized messaging system that could solve their problem of the continuous flow of data. In late 2010, LinkedIn open-sourced the project.

In July 2011, Apache Software Foundation accepted it as an incubator project; thus, giving birth to Apache Kafka that went on to become one of the largest streaming platforms in the world.

Apache Kafka

Capturing real-time data was possible by using Kafka (we will get into the discussion of how later on). Alongside Kafka, LinkedIn also created Samza to process data streams in real-time. Apache added Samza as part of their project repository in 2013.

I always wondered what thoughts the creators of Kafka had in mind when naming the tool. Jay Kreps, one of the cofounders of Kafka (along with Neha Narkhede and Jun Rao), said:

“I thought that since Kafka was a system optimized for writing, using a writer’s name would make sense. I had taken a lot of lit classes in college and liked Franz Kafka. Plus the name sounded cool for an open source project.

So basically there is not much of a relationship.”  (Narkhede, Shapira, & Palino, 2017 p. 16)

The way Kafka treats the concept of data is entirely different from what we have thought of data to be. Though Kreps may be right in saying not to read too much into the name of the tool, I find a lot of similarities between the philosophical underpinnings of 20th-century’s celebrated literary figure Franz Kafka’s works and how Apache Kafka treats data.

In this article, I will take metaphors from Kafka’s stories and relate them to understand the working of Apache Kafka.

 

“The Metamorphosis” (of the data architecture)

In one of the famous stories of Franz Kafka’s The Metamorphosis, the protagonist Gregor Samsa wakes up in the morning to find himself transformed into a giant insect. There is a physical alteration to Gregor Samsa’s body.  The Metamorphosis is a story that deals with the transformation of the protagonist and events that describe the absurdity of life, which remains one of the central ideas in most of Kafka’s works.

You must be wondering how does this story even relates to data architecture in any way? Let me explain.

 

Data as a stream of events

Traditionally, we have thought of data as a collection of a set of values about objects. Data tells us the current state of an entity which may be qualitative or quantitative — for example, some characteristics of a customer, a product, or an economy.

Till recently, data mostly existed in a monolithic framework called the database. If the attributes of the entity change, we would update the database to reflect the changes.

Thus, traditional relational databases are mutable, that is, subject to change its state and once changed remains in that state until any update occurs. You can think of updating a database as changing the data attributes or appending new records.

However, in the world of Apache Kafka, data is not objectified but treated as a stream of events. This stream of events is recorded in the form of log files called topics. For starters, a log is a file that records an event that occurred sequentially.

 

Mutable states and immutable events

For example, a customer identified by customer_id 123 buys one unit of a product having product_id 999. In a traditional relational database, the quantity 1 will be recorded against a row matching the customer id and product id, which reflects the current state of the database. Suppose, the customer changes her mind and buys three units of the same product. We need to mutate the database for this transactional change by updating the quantity from 1 to 3.

But, if we treat data as streams of events, the log file reflects the change of mind as a fact or an immutable event occurring at a specific time. The log appends the event after the previous occurrence of quantity 1 has been recorded in the log stream.

Apache Kafka - Mutable vs immutable events

Source: Kleppmann, M. (2015)

 

Statefulness and Fault Tolerance of Apache Kafka

Returning to the plight of Gregor Samsa, an event at a specific time led to his physical transformation. But his mind remains intact as he worries about getting late to office. The metamorphosis of the physical reality is an event or a fact at a particular point in time. But it is also stateful, in the sense that his mind still thinks like a human.

Just like Gregor accepts the physical transformation without any surprise or shock, his mind changes as per his physical needs. This is even true for event streams in Apache Kafka. Though it keeps a memory of all the events taking place over a while, you have the option of removing the logs dating back to an extended period.

Being stateful ensures that Apache Kafka is fault-tolerant and resilient, i.e., if any problems occur at a future date, you always have the option of returning to the previous working state.

 

Agile response for real-time data

The advent of a multitude of data sources has transformed the process of decision-making by governments, businesses, and individual agents.

Earlier, data was like a source of validation of the decision-making process; that is, strategic decisions were instinctive and experiential and then validated by data. Whereas, nowadays, data-driven decision making is a part of the process itself. Thus, we record data as events taking place within an enterprise. The enterprise analyses these events manipulates them and uses them for more data as output.

“Every byte of data has a story to tell, something of importance that will inform the next thing to be done. In order to know what that is, we need to get the data from where it is created to where it can be analyzed.”  (Narkhede, Shapira, & Palino, 2017 p. 1)

In this fast-changing app oriented world, agility and responsiveness in data transfers are critical. Moving around data quickly within internal systems and responding to requests from external servers become imperative. This is where Apache Kafka’s pub-sub (publish-subscribe) messaging comes in handy.

 

Publish-Subscribe System of Messaging in Apache Kafka

Consider an example where an e-retailer wants to build an application for the checkout process, which is a webpage or app where buyers can pay for their products that they want to order. When a customer shops online and checks out through the webpage or app, the purchase message should reflect on the shipment team’s data to ship the product to the customer who purchased it. This is not a complicated process.

Now, think of a scenario where this event should also trigger a system of other events like sending an automated email receipt to the customer, updating the inventory database, etc. As the front and back-end services get added, and the list of responses to a purchase checkout grows, more integrations need to be built, and this can get too messy. The integrations lead to an inter-team dependency for any changes to the applications, which makes development slow.

Apache kafka - purchasee subscribe process

Source: Narkhede, Shapira, & Palino (2017) p. 2

Apache Kafka helps achieve the decoupling of system dependencies that makes the hard integration go away. This makes the checkout webpage or app broadcast events instead of directly transferring the events to different servers. This is what we mean by publishing. Then, the other services like inventory, shipment, or email subscribe to that stream, which triggers them to act accordingly.

The publish-subscribe model uses the messaging service, that is, the publisher stream event messages and the subscriber chooses to subscribe to the requisite messages.

purchase subscribe model

Retrieved from https://www.confluent.io/what-is-apache-kafka/

 

Producers and Consumers

In Apache Kafka, you can store the streams of events generated by the producers durably and reliably for the specified time.

It also allows you to read and process these streams by the consumers. The producers and consumers are completely decoupled, that is, the producers don’t wait for the consumers to consume the events, and the consumers can consume the streams in any sequence. The consumers also have an option to decide which messages to consume. However, the logs are appended to the topics sequentially.  This property removes the complexity of having complicated routing rules which makes Kafka fast, efficient, scalable, and all these are performed in a distributed manner.

This reminds me of how Gregor Samsa’s mind and body are decoupled. Samsa thinks of going to work or standing upright with an insects’ body. To make space for his physical existence, Samsa’s family members move furniture out of his room. Samsa realizes his possessions are taken away from him.

Through these details, the story suggests that our physical lives shape and direct our mental lives, not the other way around. Alternatively, the need for real-time capture of data makes us think about data as not as physical realities but as streams of events.

 

End Notes

In this article, we looked at an evolved way of understanding data as streams of events. This is very different from how data has been viewed till now. We also understood the need for a decoupled architecture for these data events to be published and subscribed to make the process of data generation, consumption, and update much faster and efficient. This is achieved in the Apache Kafka framework.

I recommend you go through the following data engineering resources to enhance your knowledge:

For more technical details related to Apache Kafka, feel free to refer to some of the below-mentioned resources.

 

References

Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: The definitive guide. Sebastopol, CA: O’Reilly.

IBM Cloud Education (2020). Apache Kafka. https://www.ibm.com/cloud/learn/apache-kafka.

Kleppmann, M. (2015). Turning the database inside-out with Apache Samza. https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/.

Apache Kafka. Introduction. https://kafka.apache.org/intro

Kaushikrch
Kaushikrch

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details