Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Getting Started with Big Data & Hadoop

H
Harsh Last Updated : 27 Apr, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

Introduction on Big Data & Hadoop

The amount of data in our world is growing exponentially. It is estimated that at least 2.5 quintillions of data are being generated every day. No wonder why Big Data is a fast-growing field with great opportunities all around the globe.

In this article, we’ll be focusing on the basics of Big Data, why Big Data is used & learn about a Big Data framework known as Hadoop. This is a beginner-friendly article and will help you briefly get into the world of Big Data.

Let’s talk about Big Data first…

Scenario

In an alternate universe, you (Zark Muckerburg) had a brilliant idea to create a social media platform called “FaceCook”. Users can share their photos, videos, articles, etc. on this platform with their friends. The first problem you’ll encounter is data storage. The data will include structured as well as unstructured content. It’ll contain texts, images, videos, etc. So we need a robust system to safely store the data.

Big Data

The precise way to define Big Data is using the 5 V’s.

Big Data

1) Volume

The data we need to store is very large in volume. It may be multiple Petabytes & so on.

2)  Velocity

It refers to the high-speed accumulation of data. Imagine millions of users posting on FaceCook every minute.

3)  Variety

Big Data may consist of structured (table), unstructured (pictures, videos, etc) & semistructured (log files) data. There’s a vast variety of data types in Big Data.

4) Veracity

It refers to the uncertainty & inconsistencies in data. The reasons for such inconsistencies may be a multitude of data sources, different data types, etc. This may cause data to be very messy.

5) Value

Data in itself is of no value unless it is extracted, cleaned & analyzed thoroughly to gain insights about it. Hence you can generate value out of Big Data.

Now let’s study the different approaches to storing Big Data.

Traditional Single Database

The first naive approach anyone will have is simply setting up a single centralized database & use it to retrieve, and store the content FaceCook users post. So you set up a database server in your backyard & start using it. At first, this approach may seem viable but as the userbase increases; our database size will increase as well.
But this will lead to a high workload on the server thus putting it at the risk of data loss due to overloading.
A single database isn’t so scalable either so there is no way this approach will work.

Why not Just Divide the Work?

The issue with a single database centre is the processing may get slower. So we can simply divide the data into parts & store a single part per computer instance. This is called a Distributed File System.
Here a cluster of file servers is formed which are interconnected via a computer network.

Let’s say we want to store  4 Gb of data in a distributed file system, we can divide the data into 4 parts and store the data in a cluster of 4 file servers.

But wait. . .
What if due to some technicalities, PC 1 gets destroyed. This will cause some parts of the data to be unavailable. Imagine losing half of your posts due to company failure, you’ll sue the company lol.
So we need a utility to store & manage big data. It should also provide fault-tolerance and good performance as well.
Here Hadoop comes into the picture.

Hadoop

Hadoop is an open-source framework that is used to efficiently store & process large datasets ranging in size from GBs to Petabytes of data. Instead of using a centralized single database server to store data, Hadoop features clustering multiple commodity computers for fault-tolerance & parallel processing.

Hadoop consists of four main modules:

  • Hadoop Distributed File System (HDFS)
  • Yet Another Resource Negotiator (YARN)
  • MapReduce
  • Hadoop Common

We will focus on HDFS in this article/

Hadoop Distributed File System (HDFS)

Apache Hadoop Distributed File System (HDFS) follows a master-slave architecture, where a cluster comprises of single NameNode (Master) & all other nodes are DataNodes (Slaves).

The main activities of NameNode are:
1. Maintains & manages DataNodes.
2. Records metadata of actual data like filename, path, block location, etc
3. Receives heartbeat reports from all DataNodes. If NameNode stops receiving a heartbeat report from any DataNode, that DataNode is considered dead.

The main activities of DataNodes are:
1. Stores actual data
2. Serves read & write requests from the client.

Blocks

Data is firstly divided into blocks of size 128 MB by default in Hadoop 2.x architecture. All the blocks except the last should always be of size 128 MB.

How is Data Stored in HDFS?

The data storage process in HDFS is as follows:

  1. The client asks for available DataNode addresses to store the blocks.
  2. The NameNode which contains metadata like filename, path, block location, etc returns available DataNodes addresses to the client.
  3. The client directly stores the data into the DataNode.

Block Replication & Rack Awareness

We’ve already talked earlier that there might be a partial or total data loss even if a single node in a cluster fails to work. To prevent this, Hadoop has a feature called Block Replication.

In HDFS, Datablocks are replicated & stored in other DataNodes for better fault-tolerance. By default, the replication factor (how many copies of a block should be kept in the cluster) is 3 but it can be tweaked as per user requirements.

A rack is a collection of 30-40 DataNodes in a cluster. Rack Awareness is a concept in HDFS so that NameNode chooses the closest possible DataNode for serving read/write requests.
Rack Awareness is used to reduce network traffic while read/write operations, achieve fault tolerance, reduce latency, etc.

NameNode maintains Block Replication with Rack Awareness policies as below

  • No more than one replica is stored in a single DataNode.
  • No more than two replicas are stored in the same rack.
  • Number of racks for block replication < Number of replicas

So after the client gets the DataNode addresses from the NameNode,

The client directly writes the data on the DataNode.
For the usual case of Block Replication Factor of 3, the client first writes the block into a DataNode. Then this block is replicated into another DataNode within the same Rack. Finally, this block is replicated into a DataNode in a different rack.

This entire architecture is already visualized in the diagram above. .

Conclusion

So, we have studies right from the basics of Big Data to Hadoop framework fundamentals. This is not the end, as Hadoop & Big Data in general consists of more complex topics. Nevertheless, this is a step in the right direction.

Key Takeaways

    • Big Data means Data with high Volume, Velocity, Variety, Veracity & Value.
    • Distributed File System provides better performance than Traditional File System.
    • HDFS follows a master-slave architecture.
    • HDFS provides fault-tolerance by Block Replication.
    • HDFS provides better read/write speeds due to Rack Awareness

 

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Big datablogathonHadoopHadoop Distributed File SystemhdfsTraditional Single Database
H
Harsh

I'm an Engineering student, Mobile Dev Head at GDSC RAIT and an avid Data Science Enthusiast. I love to document everything I learn and thus I'm a technical writer as well. Technical writing helps me question my in-depth knowledge about any topic.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
Workings of Hadoop Distributed File System (HDFS)
Workings of Hadoop Distributed File System (HDFS)
An Overview of HDFS: NameNodes and DataNodes
An Overview of HDFS: NameNodes and DataNodes

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details