Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Top 10 Hadoop Interview Questions You Must Know

Vibha
Vibha Last Updated : 28 Feb, 2023
6 min read

Introduction

The Hadoop Distributed File System (HDFS) is a Java-based file system that is Distributed, Scalable, and Portable. Due to its lack of POSIX conformance, some believe it to be data storage instead. Still, it does include shell commands and Java Application Programming Interface (API) functions that are similar to other file systems. HDFS and MapReduce are the two elements of a Hadoop. HDFS is used for data storage, while MapReduce is used for data processing.
HDFS

Source: techtarget

HDFS provides the following Five Services:

  • Name Node: HDFS has just one Name Node named Master Node. The master node can monitor files, operate the file system, and contain all stored data metadata. The name node, in particular, carries information like the number of blocks, the location of the data node where the data is kept, where the replications are saved, and other characteristics. The client has direct communication with the name node.
  • Secondary Name Node: This is just for taking care of the file system metadata checkpoints that are in the Name Node. This is called the checkpoint node too. It is the Name Node’s helper Node. The Secondary Name Node orders the name node to generate and deliver the fsimage and editlog files, after which the Secondary Name node creates the compacted fsimage file.
  • Job Tracker: Job Tracker accepts users’ seek for Map Reduce processing. The Job Tracker interacts with the Name Node to determine the location of the data that will be desired in processing. The Name Node responds with the necessary processing data information.
  • Data Node: A Data Node holds data in the form of blocks. This is named the slave node too, and it is in charge of storing the real data in HDFS, which the client may read and write to. These are daemon slaves. Every 3 seconds, each Data node sends a Heartbeat message to the Name node, indicating that it is alive. When a Name Node does not get a heartbeat from a data node for 2 minutes, it considers that data node to be dead and begins the block replication process on another Data node.
  • Task Tracker: It is the Job Tracker’s Slave Node, and it will accept the job from the Job Tracker. It gets information from the Job Tracker too. Task Tracker will apply the code to the file. Mapper is the process of applying that code to the file.

This article was published as a part of the Data Science Blogathon.

Table of Contents

Q1. How is HDFS Fault Tolerant?

Q2. Write the Features of HDFS.

Q3. Differentiate Between Regular FileSystem and HDFS?

Q4. What are The Most Important Parameters for Configuring a MapReduce Program?

Q5. Name the Three Different Modes in Which Hadoop May Operate.

Q6. What are the Functions of RecordReader, Combiner, and Partitioner in a MapReduce operation?

Q7. How Do You Restart NameNode and all of Hadoop’s Daemons?

Q8. In HDFS, How do you Define Block? What is the Hadoop 1 and Hadoop 2 Default Block Size? Is it Possible to Modify It?

Q9. Contrast HDFS with Network Attached Storage (NAS).

Q10. What Will You do if NameNode Fails?

Q1. How is HDFS Fault Tolerant?

Hadoop
Source: data-flair.training

When data is saved on HDFS, NameNode replicates it to several DataNodes. The default replication factor value is 3. You can alter the configuration factor to suit your needs. If a DataNode fails, the NameNode will immediately copy the data from the replicas to another node and make the data available. In HDFS, this provides fault tolerance.

Q2. Write the Features of HDFS.

  • HDFS has a high fault-tolerance level.
  • HDFS may be made up of thousands of server computers. Every computer saves a part of the file system data. HDFS identifies and automatically recovers problems that may occur on computers.
  • HDFS has a high throughput price.
  • HDFS is intended to store and scan millions of data rows and count or add sections of the data. The length of time necessary for this process is determined by the intricacies involved.
  • It was created to handle very large datasets in batch-style processes. However, the emphasis is on high data throughput rather than low latency.
  • HDFS is cost-effective.
  • HDFS is meant to be constructed on commodity machines and heterogeneous platforms that are low-cost and widely available.

Q3. Differentiate Between Regular FileSystem and HDFS?

Regular FileSystem: In a regular FileSystem, all of the data is kept in one place. Data recovery is difficult due to the machine’s low tolerance for failure. Because seek time is longer, it takes longer to process the data.

HDFS: Data is scattered and managed across several platforms using HDFS. Data can still be retrieved from other nodes in the group if a DataNode fails. Reading takes more time due to the need to read data from the disc locally and coordinate data from multiple systems.

Q4. What are the Most Important Parameters for Configuring a MapReduce Program?

The following configuration parameters are required:

  • Enter the job’s location in HDFS.
  • The job’s output location in HDFS
  • Formats for input and output
  • Classes with map and reduce functions
  • The JAR file contains the classes for the mapper, reducer, and driver.

Q5. Name the Three Different Modes in which Hadoop May Operate.

There are three ways in which Hadoop may operate:

  • Standalone (local) mode: This is the default mode if nothing is configured. All Hadoop elements, like NameNode, DataNode, ResourceManager, and NodeManager, operate as a single Java process in this mode. This makes use of the local filesystem.
  • Pseudo-distributed mode: A single-node Hadoop deployment is termed a pseudo-distributed Hadoop system. All Hadoop services, including the master and slave services, were run on a single compute node in this way.
  • Fully distributed mode belongs to Hadoop deployments in which the Hadoop master and slave services operate on distinct nodes.

Q6. What are the Functions of RecordReader, Combiner, and Partitioner in a MapReduce operation?

  • RecordReader- This interacts with the InputSplit and turns the data into key-value pairs that the mapper can read.
  • Combiner- This is an optional step that functions like a micro reducer. The combiner accepts data from the map tasks, processes it, and then forwards the results to the reduction phase.
  • Partitioner- The partitioner determines the number of reduced jobs that will be used to summarize the data. It confirms too, how combiner outputs are routed to the reducer and regulate the important segmentation of intermediate map outputs.

Q7. How Do you Restart NameNode and all of Hadoop’s Daemons?

The following commands will help you in restarting NameNode and all daemons:

  • Stop the NameNode with the ./sbin/Hadoop-daemon.sh stop NameNode command and restart it with the ./sbin/Hadoop-daemon.sh start NameNode command.
  • You may use the ./sbin/stop-all.sh command to halt all daemons and then restart them using the ./sbin/start-all.sh command.

Q8. In HDFS, How do you Define Block? What is the Hadoop 1 and Hadoop 2 Default Block Size? Is it Possible to Modify It?

Blocks are the smallest continuous space on your hard disk where data is stored. Each record is kept in a separate “block” on the HDFS, and then those blocks are dispersed around the Hadoop clusters. Files in HDFS are divided into block-sized pieces and stored in separate units.

  • Hadoop 1 has a 64 MB block size by default.
  • Hadoop 2 has a 128 MB block size by default.

Yes, blocks may be modified. In a Hadoop context, the dfs.block.size option in the hdfs-site.xml file can be used to set the size of a block.

Q9. Contrast HDFS with Network Attached Storage (NAS).

Hadoop
  • A network-attached storage (NAS) server is a file-level computer data storage server that is linked to a computer network and provides data access to a wide group of customers. NAS can be either hardware or software that provides file storage and access services. Hadoop Distributed File System (HDFS) is a Distributed File System that uses a commodity system to store data.
  • Data Blocks in HDFS are spread across all machines in a group. NAS, on the other hand, stores data on a specialized system.
  • HDFS is built to operate with the MapReduce paradigm, which shifts computing to the data. Because data is kept apart from calculations, NAS is unsuitable for MapReduce.
  • HDFS uses a low-cost commodity system, while NAS is a high-end storage system with a high price tag.

Q10. What Will you do if NameNode Fails?

To get the Hadoop group up and running again, the NameNode recovery process includes the following steps:

  • Use the file system metadata copy to start a new NameNode (FsImage).
  • Then, set up the DataNodes and users so that they can identify the newly created NameNode.
  • After loading the latest checkpoint FsImage (for metadata information) and receiving enough block reports from the DataNodes, the new NameNode will begin serving the client.

However, this NameNode recovery process would potentially be time-consuming on big Hadoop clusters, which becomes much more difficult at the time with normal maintenance. As a result, we use HDFS High Availability Architecture.

Conclusion

The Hadoop Distributed File System (HDFS) is a fault-tolerant data storage file system that works on commodity systems. It was made to solve problems that regular databases couldn’t handle. This page focuses on interview questions for all levels, along with the following points:

  • Hadoop Distributed File System is shortened to HDFS.
  • Hadoop Distributed File System (HDFS) and MapReduce are two main elements.
  • Data stored in a regular FileSystem is preserved in a single system.
  • A network-attached storage (NAS) server is a computer data storage server that stores files on a network.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Vibha
Vibha

I am an engineering student. Currently, I am pursing Btech from Vellore Institute of Technology. I am very passionate about programming and constantly eager to expand my knowledge in Data Science and Machine Learning.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details