Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Building a Lakehouse – Try Delta Lake!

GT
GT Last Updated : 12 Oct, 2022
5 min read

Introduction

Enterprises have been building data platforms for the last few decades, and data architectures have been evolving. In this article, I’ll introduce you to “Lakehouse,” one of the latest approaches for building data platforms, and its underlying technology, “Delta Lake,” that powers such Lakehouses.

Let’s first look at how things have changed and how a path-breaking technology like Delta Lake contributes to this.

Here you go!

How it started…

The late ’90s/early 2000s saw the emergence of enterprise data warehouses. Warehouses that could store data from different source systems by standardizing, integrating, and cleaning data before persisting it.

It was that era when enterprises started realizing the benefits of having a central data store against maintaining department-specific database silos.

Warehouses soon became the default data store for organizations trying to run their heavy BI workloads.

But that was a couple of decades ago.

…How it’s going

The year 2022: We are now discussing building a data platform based on “data mesh” core principles – trying to decentralize how data has been stored, maintained, and managed.

We are now trying to implement domain-driven data platforms with federated governance that every business unit should follow to have data accountability, create data products and get self-servicing capabilities.

Lakehouses can play a major role in these modern architectures – built using the  Data Mesh principles or a centrally managed platform.

Background

In today’s world, technology has been changing rapidly to make cheaper storage & powerful compute engines. It is just apt that the data platforms also change accordingly and leverage these changes to build a more affordable and efficient data ecosystem.

Lakehouse is one such technology that can change how you have been storing and managing your data. It has the potential to provide all the features that the warehouse offers and, at the same time, help you to achieve the cost benefits of a data lake.

What is a Lakehouse?

As the name suggests, “Lakehouse” has the benefits of a lake + warehouse.

It is primarily a data lake built using cloud data storage like AWS S3, Azure ADLS, or Google Cloud storage. It has all the good things that come with cloud storage – less cost, high availability and durability, support for unstructured data, and many others.

Along with all these, Lakehouse also enjoys the features offered only by cloud warehouses, like ACID support, time travel ability, and better performance.

And how does Lakehouse get all these benefits? The magic happens at the storage level – not where the data is stored, but how the data is stored.

Enter Delta Lake – one of the leading technologies behind Lakehouses that makes them ACID compliant and performs better than plain data lakes.

 

What is Delta Lake?

Delta Lake is a storage framework that helps you to build a trustworthy Lakehouse-based data platform.

Delta Lake is not a storage layer nor a compute engine. You should not compare it with S3, ADLS, or Google Cloud Storage. It is a framework that sits on top of these object storages.

If you want to compare it with any existing technology (because that is the easiest way to learn and understand any new tech), you can compare it with Hive table format (not the query engine).

Like Hive provided a table format on top of HDFS storage, Delta gives you the metadata on top of storage objects like the parquet files.

To summarise: Delta Lake = Open Storage Framework for building Lakehouses with cost efficiencies of data lakes & ACID capabilities of a warehouse

 

Benefits of Delta Lake

So why should you care about Delta? Is it the future of all data platforms?

Let me first explain all the key benefits that Delta Lake offers

  • Open source, community-driven framework

Delta Lake is an open framework, unlike other proprietary storage used in some of the leading cloud warehouses. It was initially created by Databricks and is open-sourced under Linux Foundation. Anyone can use Delta Lake and contribute towards expanding its integration ecosystem.

  • Integrates with most of the widely adopted compute engines

It can integrate with compute engines like Apache Spark, Presto, Trino, Kafka and Hive. It also supports leading cloud warehouses like Snowflake, Redshift & Synapse as compute engines to analyze data available in Delta Lake.

  • Enables separation of storage & compute based architecture

A true Lakehouse has separate storage and compute and can enable any external compute engine to analyze data within the Lakehouse. Delta Lake supports this feature by providing an open storage framework that integrates with a variety of compute engines

  • Supports Unified Batch & Streaming Workloads

Delta Lake can also help you to implement “Kappa” architectures by having a single process for your batch & nearly real-time use cases. Delta Lake supports low latency (seconds level) for fetching data.

  • Supports ACID & Time Travel

Delta Lake supports ACID features like warehouses. You don’t have to worry about data getting corrupted or handling concurrent writes and reads. It also supports time travel to restore data from previous timestamps or versions.

  • Supports Schema Evolution & Enforcement

It has built-in features to validate the schema of files that need to be loaded in Delta Lake. It also supports schema evolution to capture the additional attributes in case of schema changes.

 

What’s under the Hood?

So how does Delta achieve all these benefits? How does it bring ACID capabilities to Data Lakes?

The magic happens at the storage layer!

A quick tip: For most modern platforms, the performance gains are because of their storage methodologies, not compute engine. Be it Snowflake, Redshift, Apache Hudi or Delta – try to understand how they store the data, and you will know how they can offer all those cool features.

Let’s try to understand how Delta stores the data.

When you create a Delta Lake and create any delta table, it has two main parts in the storage layer

  • dataset – the actual data files as parquet files
  • delta log – the transactional log that maintains all the changes.
Delta Lake

How Delta Lake stores the data & transaction logs

Delta Log – the Brain of Delta Lake

It is the delta log that makes Delta Lakes so powerful and efficient. It enables the capabilities like ACID, Time Travel, scaling metadata, and many more.

So what Delta log does?

The Delta Log holds the ordered record of every transaction on the delta table from when it is created. It records everything – new files added, files removed, metadata changes, commits, and timestamps.

Since all these are recorded as ordered time logs, Delta Lake can leverage these logs to do time travel, find different versions, and identify metadata changes.

Delta Log acts as the single source of truth and provides a central place to track all changes for that table.

Conclusion

Lakehouses are the latest approach for building your data platforms. They provide cost efficiency and flexibility of data lakes as well as the ACID capabilities & performance of warehouses.

To build Lakehouses, you need storage frameworks that can support these features. Delta Lake is an open framework that can help you implement a true Lakehouse with separate storage and compute.

The backbone of Delta Lake is its transaction log, aka Delta Log. Delta Log holds the ordered record of every transaction, thus enabling its users to track every change related to data and metadata.

That’s it; I hope you have enjoyed reading this article! Have a great day ahead!

GT
GT

I'm a Cloud Data Architect helping teams to get started with their cloud data journey!

I am an AWS/Azure/Snowflake/Databricks certified data professional & work as an independent consultant on various activities like consulting/training/mentoring - all within the cloud data space!

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details