Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

The Tale of Apache Hadoop YARN!

S
Shikha Gupta Last Updated : 03 Jun, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

Introduction

YARN stands for Yet Another Resource Negotiator, a large-scale distributed data operating system used for Big Data Analytics. Initially, it was described as “Redesigned Resource Manager” as it separates the processing engine and the management function of MapReduce. Apart from resource management, Yarn also performs various jobs, including job Scheduling, workload management, management of high availability features of Hadoop, implementation of security controls, and maintaining a multi-tenant environment. Furthermore, to make the system more efficient, YARN allows various data processing engines like graph processing, interactive processing, stream processing, and batch processing to run and process data stored in HDFS (Hadoop Distributed File System).

Why YARN?

Hadoop version 1.0, also known as MRV1(MapReduce Version 1), is a proficient data computational tool that performs processing and resource management functions. MRV1 has a single master named job tracker, which performs job scheduling, resource allocation, and job monitoring. It assigns maps and reduces tasks known as task trackers on several subordinate processes. Job trackers get the periodic progress report from the task trackers. Due to a single job, this design resulted in a scalability bottleneck. As a result, Hadoop 1. x had more limitations like delays in batch processing, inefficient utilization of computational resources, scalability issues, etc.

Moreover, it limits only MapReduce for processing big datasets. In 2012, Yahoo and Hortonworks introduced YARN in Hadoop version 2.0 to overcome all these shortcomings. The intention behind YARN is to reduce the overhead of MapReduce by taking over the job of Resource Management and Job Scheduling. With YARN, Hadoop can now run non-MapReduce jobs within the Hadoop cluster. With MapReduce batch tasks, YARN can now run stream data processing and interactive querying.

Features of YARN

YARN is a popular tool due to the following features:

  • Highest Scalability: The architecture of the Resource manager of YARN architecture allows Hadoop to manage thousands of nodes and clusters according to the user requirements.

  • High-degree Compatibility: YARN supports the applications created via the map-reduce framework without disruptions; that’s why it shows compatibility with Hadoop 1.0.

  • Better Cluster Utilization: YARN supports efficient and dynamic utilization of cluster resources in Hadoop, enabling better cluster utilization.

  • Multi-tenancy: YARN is a versatile technology that allows multiple engine access and gives the benefit of multi-tenancy.

Architecture and components of Hadoop YARN

Apache Hadoop 3.3.3 – Apache Hadoop YARN

source:https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html

Resource Manager

Resource Manager, the master daemon of YARN, is responsible for the management of global assignments of resources like CPU and memory with several other applications. With the goal of maximum cluster utilization, it keeps all resources in use against various constraints such as capacity guarantees, fairness, and SLAs. On receiving the processing requests, it forwards parts of requests to the corresponding node manager where the actual processing takes place and allocates resources for the completion of the request accordingly. It is used as the arbitrator of the cluster resources for job scheduling and deciding the allocation of the available resources for competing jobs. It consists of two parts:

  • Scheduler: The scheduler performs scheduling based on the requirement of resources by the allocated applications. It distributes resources to the running applications depending upon the ordinary constraints of capacities, queues, etc. It does not monitor or track applications, hence known as a pure scheduler. Scheduler doesn’t ensure restarting failed tasks either due to hardware failure or application failures. To partition the cluster resources among various queues and applications, it has a pluggable policy plug-in. Examples of the plug-in are Capacity Scheduler and the Fair Scheduler, which are current MapReduce schedulers.

  • Application Manager: It is an interface that manages a list of applications that have been submitted, are currently running, or are finished. The Application Manager manages running Application Masters in the cluster by accepting job submissions and negotiating the Resource Manager’s first container. In addition, it performs multiple tasks like starting Application Master and monitoring and restarting the Application Master container on different nodes in case of failures.

Node Manager

It is the slave daemon of Yarn whose primary goal is to keep up-to-date with the Resource Manager. The responsibility of the node manager is to manage application containers assigned to it by the resource manager. It monitors containers’ resource usage(memory, CPU) and reports it to the Resource Manager. In addition, the yarn Node Manager registers with the Resource Manager to track the health of the node on which it is running and sends the heartbeats with the node’s health status. The node manager is also responsible for performing the log management and killing/destroying the container as directed by the Resource Manager.

Containers

The Containers are a collection of physical resources like RAM, CPU core, memory, and disks on a single node. The containers are monitored by Node Manager and scheduled by Resource Manager. The job of the container is to grant the right to an application to use a definite amount of resources(memory, disk, CPU, etc.) on a particular host. A Container Launch context which is Container Life Cycle (CLC), manages the YARN containers. CLC is a record that carries information like a map of environment variables, security tokens, dependencies stored in remotely accessible storage, the command required to create the process, and the payload for Node Manager services.

Application Master

An application is nothing but a single job submitted to a framework, and each application has a specific Application Master associated with it, which is a framework-specific entity. The chief responsibilities of the application master include negotiating resources with the resource manager, tracking the status, and monitoring the progress of a single application. It manages faults and works with the Node Manager to monitor and execute the component tasks. The application master sends a Container Launch Context(CLC) which includes everything an application needs to run and requests the container from the node manager. Once the application is started, it periodically sends heartbeats to the resource manager to check the health and update records based on its resource demands.

Application Workflow in Hadoop YARN

Perform the following steps to run an application through Hadoop YARN.

Apache Hadoop YARN | Sequence of Execution

source: https://www.softwaretestinghelp.com/what-is-hadoop-yarn/

Step1:- Apply:

The client connects with the Resource Manager to submit the YARN application.

Step 2:- Container allocation:

To launch the Application Manager, the Resource Manager searches for a Node Manager and allocates the container.

Step 3:- Registration:

In this step, the Application Master registers itself with the resource master.

Step 4:- Negotiation:

From the Resource Manager, the application master negotiates the containers.

Step 5:- Notification:

Application Manager gives notification to the Node Manager for launching containers.

Step 6:- Execution:

Application code either gets executed in the container it is currently running, or it can request more containers from the resource manager.

Step 7:- Status Monitoring

To monitor the application’s status, the client contacts the Resource Manager or monitors the status of the Application Manager.

Step 8:- Disconnected

Once the processing is complete, the Application Manager gets disconnected from the Resource Manager.

YARN Command Line Interface

Mostly YARN commands are available for admins, but there are a few commands which developers can also run. These are:

  • Help

To get a list of all commands available in the YARN cluster.

Syntax:

-yarn -help

  • Version

To get the current version of YARN you are working with.

Syntax:

-yarn -version

  • Application id

To print the logs of a particular application id.

Syntax:

- yarn logs -applicationId

Conclusion

YARN is one of the most powerful concepts of Hadoop 2. x. This article has seen all the important concepts of the YARN with good examples of how they work with applications. Key learnings are:

  • We discuss the Yarn and its features.

  • Then, learned about the architecture and components of YARN.

  • Also, learned how an application workflows in Hadoop YARN.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

S
Shikha Gupta

My full name is Shikha Gupta , pursuing B.tech in computer science from Banasthali vidhyapeeth Rajasthan.
I am from East Champaran in Bihar.
My area of interest ,Deep learning,NLP,Java,Data Structure,DBMS and many more.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details