Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

A Brief Introduction to Apache HBase and it’s Architecture

T
Trupti Dekate Last Updated : 13 Oct, 2022
7 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Since the 1970s, relational database management systems have solved the problems of storing and maintaining large volumes of structured data. With the advent of big data, several organizations realized the benefits of big data processing and started choosing solutions like Hadoop to solve big data problems. Apache Hadoop uses a distributed file system to store big data in a distributed manner and a MapReduce framework to process it. Hadoop can store and process huge volumes of data in structured, semi-structured, or unstructured formats. But Apache Hadoop can only do batch processing, and we can only sequentially access data.
Apache HBase
Source: tibco.com

For simple jobs, we have to search the entire data set. When we process a huge dataset in Hadoop, it will result in another huge dataset, and we have to process this dataset sequentially. Because of this, Hadoop is not so good for record retrieval, updates, and adding small batches incrementally. So there is a need for a new solution that allows us to access any data point per unit of time. Applications such as CouchDB, HBase, Cassandra, MongoDB, and Dynamo came onto the scene. These databases can store huge amounts of data and allow data to be accessed randomly. Now let’s look at an introduction to HBase.

What is Apache HBase?

HBase is an open-source, distributed, scalable, and NoSQL database written in Java. HBase runs on top of the Hadoop Distributed File System and provides random read and write access. It is a data model similar to Google’s big table designed to provide fast random access to huge volumes of structured data. HBase takes advantage of the fault tolerance capability provided by HDFS. It is designed to achieve a fault-tolerant way of storing large numbers of sparse datasets.

HBase is the perfect choice for applications that require fast and random access to huge amounts of data. HBase achieves low latency and high throughput by providing faster read and writes access to large data.
You can store data in HDFS either directly or via HBase. Using HBase, we can read data in HDFS or access it randomly.

Where can we use HBase?

It is not suitable for every problem. Suppose we have hundreds of millions or billions of rows and want to read the data quickly; it is best to use HBase. Mainly used for random read/write access to large data in real-time.
You can use HBase when you want to store huge volumes of data and want high scalability. We can only use it if we can live without all the other features of traditional database systems, such as typed columns, transactions, advanced query languages, secondary indexes, etc.
It’s a good choice if we have a lot of versioned data and want to save it all. If you want column-oriented data, you can opt for HBase. Let’s now look at the storage mechanism in HBase.

HBase Data Model

HBase is a column-oriented database. A column-oriented database stores data in cells grouped into columns, not rows.
Let’s look at the whole conceptual view of data organization in HBase.
Source: tibco.com

1. Table & 2. Row
Several Rows are multiple in Hbase Table. Columns have values ​​assigned to them. HBase sorts rows alphabetically by row key.
The main goal is to store data so that related rows are closer together. The domain of the site is used as a common row-key pattern. For example, if our row keys are domains, we should store them in reverse, i.e. org.apache.www or org.apache.mail or org. Apache.Jira. This way, all Apache domains are close to each other in the HBase table.

3. Column
An HBase column consists of a column family and a column qualifier separated by the : (colon) character.

A. Column family
Column families physically house a set of columns and their values; then, Each column family has a set of storage properties, such as how its data is compressed, whether its values ​​should be cached, how its row keys are encoded, and more. Each row in an HBase table has the same column families.

b. Column qualifications
A column qualifier for qualification is added to the column family to provide an index for that data part.
Example: the column family is content, then the column qualifier can be content: HTML or content: pdf.
The Column families are fixed during table creation, but column qualifiers are mutable and vary widely between rows.

4. The cell
A cell is essentially a combination of a row, a column family, and a column qualifier. Contains a value and a timestamp that represents the version of the value.

5. Timestamp
A timestamp is an identifier for a given value version and is written next to each value. The timestamp default represents the time on the RegionServer when the data was written. However, we can specify a different timestamp value when inserting data into a cell.

Apache HBase Architecture

HBase consists of three main components: HBase Region Server, HMaster Server and Regions, and Zookeeper. Let’s take a closer look at each of these components.
hbase
Source: tibco.com

 

1. HBase area
A range is an ordered range of rows that store data between the start and end keys. A table in HBase is divided into several areas.
The default region size is 256 MB, and we can configure it according to our needs. Region Server has a group of regions for clients. Region Server can serve 1000 regions (approximately) to the client.

2. HBase HMaster
The HMaster in HBase processes the Region Server collection that resides in the DataNode. The HBase HMaster performs DDL operations and assigns regions to region servers. Coordinates and manages the regional server.
HMaster assigns regions to region servers during startup and reassigns regions to region servers during recovery and load balancing. It is responsible for monitoring all Region Server instances in the cluster. It does this with the help of Zookeeper and performs a recovery mechanism if any Region Server fails. HMaster provides an interface for creating, updating and deleting tables.

3. HBase ZooKeeper – Coordinator
The Zookeeper acts as a coordinator in a distributed HBase environment. It helps maintain the server state inside the cluster by communicating through sessions.
Each Region Server and the HMaster server send a continuous heartbeat regularly to Zookeeper. Zookeeper checks which server is active and available. Zookeeper provides notification of server failure so that HMaster can take recovery action. Zookeeper also maintains a path to the.META server. This helps the client search for any region.

4. HBase meta table
META table is a special HBase catalog table that maintains a list of all region servers in the HBase storage system. A . META file manages a table in the form of keys and values. The key will represent the initial key of the HBase region and its id. The value will contain the path to the region server.

Apache HBase Properties

The basic properties of Apache can be defined as follows.

  • Atomic read and write: The atomic read/write at the row level. This means that during one read/write process, it prevents all other processes from performing any read/write operations.
  • Consistent reading and writing: Apache HBase provides consistent reading/writing due to the above feature.
  • Linear and modular scalability: Because HBase runs on top of HDFS, datasets are distributed over HDFS. This makes HBase linearly scalable across multiple nodes. In addition, it is modularly scalable.
  • Easy-to-use Java API for client access: HBase provides an easy-to-use Java API for programmatic access.
  • Thrift gateway and REST-ful web services: HBase also supports Thrift and REST APIs for non-Java front-ends.
  • Block Cache and Bloom Filters: Supports Block Cache and Bloom Filters to optimize large query volumes.
  • Automatic failover support: Apache HBase with Hadoop Distributed File System provides WAL (Write Ahead Log) across clusters, which provides automatic failover support.
  • Sorted row keys: HBase stores row keys in lexicographic order. Using sorted row keys and timestamps, we can create an optimized request.
  • Data replication: HBase provides data replication across clusters.
  • Sharding: HBase offers an automatic and manual division of regions into smaller subregions to reduce I/O time and overhead.

Apache HBase Application

We can use HBase in many sectors. Some of the applications of HBase in various sectors are:
  •  Medicine: Medical sectors use Apache HBase to store genome sequences and run MapReduce. They use them to store the history of people’s diseases and many others.
  •  Sports: We can use HBase to store match history to make better analyses and predictions.
  •  Oil and Petroleum: It is also used in the oil and petroleum industry to store exploration data. This is done to analyse and predict the likely locations where oil may be located.
  •  E-commerce: The e-commerce industry uses HBase to record and store logs of customer search history. They do this to perform data analysis to target interested audiences for profit.
  •  Other Fields: We can use HBase in other areas where we want to store petabytes of data and analyse it, which might take months for an RDBMS.
  •  Companies like Facebook, Yahoo, Twitter, Infolinks and Adobe use Apache HBase internally.

Limitations of Apache HBase

HBase has many benefits, but it has some limitations as well. Let’s see each one by one.

  •  HBase does not provide support for some features of traditional database systems.
  •  HBase cannot perform functions like SQL. It does not support SQL structure, so it has no query optimiser.
  •  HBase is CPU and memory intensive.
  •  HBase integration with Hive and Pig workloads will result in some cluster timing issues.
  •  The HBase setup will require fewer job slots per node to allocate HBase CPU requests in a shared cluster environment.
  •  The cost and maintenance of HBase are too high.
  •  Hbase supports only one default collation per table.
  •  Joining and normalization in the Hbase table are very complicated.

Conclusion

We can say that Apache HBase is a NoSQL database that runs on top of the Hadoop Distributed File System. Provides the BigTable functionality of the Hadoop framework. It consists of HMaster Server, HBase Region Server, and Regions and Zookeeper. The article included the difference between HBase and RDBMS as well as the difference between HBase and HDFS. HBase provides consistent reads and writes. It is open-source and scalable. We can use HBase in many industries, including medicine, sports, e-commerce, etc.
  • A range is an ordered range of rows that store data between the start and end keys. A table in HBase is divided into several areas. The default region size is 256 MB, and we can configure it according to our needs. Region Server has a group of regions for clients. Region Server can serve 1000 regions (approximately) to the client.
  • Applications such as CouchDB, HBase, Cassandra, MongoDB, and Dynamo came onto the scene. These databases can store huge amounts of data and allow data to be accessed randomly. Now let’s look at an introduction to HBase.
  • HMaster assigns regions to region servers during startup and reassigns regions to region servers during recovery and load balancing. It is responsible for monitoring all Region Server instances in the cluster.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
T
Trupti Dekate

I am an Accountant at Global private Analytics Services working with the Data Analysis Team for handling the budget of various Growing Companies. We provide service of analytics and made the work of new tech companies easy by helping them manage their total investment and giving suggestions.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details