Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

All About Data Pipeline and Kafka Basics

usha bhanu
usha bhanu Last Updated : 11 Jun, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

Introduction on Kafka

In old days, people would go to collect water from different resources available nearby based on their needs. But as the technology emerged, people have automated the process of getting water for their use without having to collect it from different resources by using a single resource for all needs. A Data pipeline in general terms is very similar to this example. It is like a set of steps that involve storing and enriching the data from the source to the destination. Which further helps to gain insights. It also helps to automation of the storing and transformation of the data.

When we hear the term “data pipeline,” the following questions immediately deserve a mention:

  1. What is a data pipeline?
  2. What is its purpose?
  3. What distinguishes it from ETL?

In the case of a data producer and a data consumer, the producer cannot send the data before doing some data processing, data governance, and data cleaning.

The type of data pipeline is determined by the purpose for which it is being utilized. It might be used for data science, machine learning, or business analytics, among other things.

Types of Data Pipelines

1. Real-Time Data Pipeline

2. Batch Data Pipeline

3. Lambda Architecture (Real-Time and Batch)

The producer can send either batch or real-time data.

Batch Data: CSV, database and mainframe are examples of traditional data. For example, weekly or monthly billing systems.

Real-Time Data: This information comes from IoT devices, satellites, and other sources. For instance, traffic management systems.

Before entering the central data pipeline, batch data goes through batch ingestion, while real-time data gets through stream ingestion.

What are Batch and Stream Ingestion?

Batch ingestion is the processing of data acquired over a period of time. In contrast, because stream ingestion involves real-time data, the processing is done piece by piece.

The data pipeline is made up of several separate components. Consider an ODS (Operational Data Store) where the batch data is staged after processing. Stream data can also be staged in a Message hub using Kafka. A NoSQL database, such as MongoDB, can be used as a messaging hub.

Organizations will be able to collect data from a number of sources and store it in a single location thanks to ODS. Kafka is a distributed data storage that may be used to create real-time data pipelines. Even if all of this data has already been analyzed, it is still possible to enrich it. MDM can be used to do this (Master Database Management). It assists with data by reducing errors and redundancies.

When the data is ready, it may be delivered to the intended destination, such as a Data Lake or a Data Warehouse. Which the customer may utilize to create business reports, machine learning models, and dashboards, among other things.

What are Batch and Stream Ingestion?

But How is it Different from ETL?

The data is put into a database/data warehouse at the end of the ETL pipeline. The Data Pipeline, on the other hand, is clearly different since it involves more than just importing data. It serves as a link between the source and the destination. Here, the ETL pipeline may be thought of as a subset of the data pipeline.

Let’s dive into Apache kafka,

Linkedin developed Kafka before handing it to the Apache Foundation. Apache Kafka is an open-source platform that works with events. To fully understand the event-driven approach, we must first understand the differences between the data-driven and event-driven approaches.

Data-driven: Consider an online retailer such as Amazon. When customer A purchases product X on date 1, the database records the transaction. But what would happen if we had to consider more than 200 million customers? All of the data is kept in several databases, which should communicate with one another and the online site.

Event-driven: This approach uses the same sort of interaction with the company website. However, all data is kept in a queue. The database may get the information it needs from the queue.

What is an Event?

It might be a single activity or a group of business actions. For example, if customer A purchases product X on date 1, an event is recorded with the following information:

Customer id-123

Name– A

Order ID– 001

Date– 1.

Kafka encourages the use of queues to store events. Any customer can use this queue to get the information they need.

The next question that should come to mind is: How does Kafka differ from other middleware?

1. We can store large amounts of data for any period of time.

2. It is unique for each event (even if the same customer places a second purchase because he or she may be purchasing product Y on a different date).

Another notable role of Kafka is that it uses the “Log” data structure. The OFFSET field in the log indicates where the data is taken. It will never overwrite existing data; instead, it will append the data at the end.

Log Data Structure: The log is the most basic data structure for describing an append-only sequence of records. Immutable log records are added to the end of the log file in an exact sequential manner.

Message– Row(it is the smallest unit of Kafka architecture)

Topic- Table

Partition– View

Log Data Structure:

The messages (mail symbol), topics (cylindrical pipe), and partitions are all plainly visible in the above image. For fault-tolerance and scalability, these divisions are kept in multiple settings. The events can be sorted or sequenced within one partition only.

Kafka Cluster:

What is an Event? 2

A Kafka broker is another name for a Kafka server. They get data storage and retrieval instructions from the producer and consumer. Data replicates are also kept in separate brokers from the leader. In the event of a failure, the replicated broker will exercise authority. Leaders will always interact with producers and consumers. A Kafka cluster is a collection of many brokers.

To explore further there are Kafka products:

1. Kafka Core (topics, logs, partitions, brokers, and cluster)

2. Kafka Connect (Connecting Mainframe to DB)

3.kSQL (SQL designed for Kafka)

4. Kafka Client ( To connect client)

5. Kafka Streams (manage stream ingestion into Kafka)

Advantages

1. Loose decoupling ( Flexible environment)

2. Fully distributed (Fault tolerance)

3. Event-based approach

4. Zero downtime ( Scalable architecture)

5. No vendor lock-in (Open software)

Conclusion

We can clearly see that this concept of the data pipeline is useful for real-world problems by automation and making data available for the consumers at a single source for them to access and making it easier to gain insights and progressive analysis. It provides a flexible environment.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

blogathonData PipelineKafkaKafka basicspython
usha bhanu
usha bhanu

Hi, I’m Usha!

As a Data Science professional with a Master's from Indiana University and a Bachelor's in Computer Science & Engineering, I excel in transforming complex data into actionable insights. I specialize in advanced data visualization, AI solutions, and regulatory reporting, using tools like Python, R, and SQL.

I have a proven track record of enhancing productivity and reducing errors through innovative AI solutions and data analysis. My strong analytical skills are complemented by my ability to communicate complex concepts clearly and work effectively in collaborative environments.

Passionate about driving innovation and making data-driven decisions, I am eager to leverage my expertise in digital media and other dynamic industries. If you're looking for a skilled data scientist who can turn data into strategic assets, let's connect!

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
Build a Scalable Data Pipeline with Apache Kafka
Build a Scalable Data Pipeline with Apache Kafka
Getting Started with Data Pipeline
Getting Started with Data Pipeline

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details