Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Basics of Data Modeling and Warehousing for Data Engineers

Chetan
Chetan Last Updated : 25 Jul, 2022
6 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Companies struggle to manage and report all their data. Even asking basic questions like “how many customers we have in some places,” or “what product do our customers in their 20s buy the most” can be a challenge. The data repository should have solved these challenges. The concept of data storage has changed dramatically since it began in the mid-1980s. It has evolved into a separate discipline to meet the growing challenges and complexities of the business world. This has led to better technology and stronger business practices.
data engineering

Initially, data warehouses were created so that companies could store a source of analytical data that they could use to answer queries. This is still an important factor, but today, companies need easy access to information on a large scale with a diverse set of end-users. The defined user has greatly expanded from specialized engineers to almost anyone who can drag and drop to the Tableau.

Understanding end-users of data storage are essential if you are planning to build one. It can be easy with modern tools to pull data from Snowflake or BiQuery without prioritizing the end-user, but the goal should be to create a basic layer of data that is easy to understand for anyone. At the end of the day, data is a product of a data group and needs to be as understandable, reliable, and easy to use as any other feature or product.

Data as Product

Data is just a useful product. That new oil data still weighs, but the data is really current. It is expected to work. We do not want crude oil. We want high octane fuel. We want to be able to just plug the fuel into our car and operate it without any problems. As people get closer and closer to this product, it needs to be used. This means it should be:

  • It is easy to understand,
  • and easy to operate.
  • It’s solid.
  • He is faithful.
  • It’s on time

Evaluating company data processes can greatly improve the end-user experience with specified data. Overall, the data management part as a product follows advanced processes that help capture data from crude oil to high octane fuel.

Best Data Model Practices

data modeling

 

When building your data repository, it is important to use best practices. However, it is also important that you do not overreact. Many warehouse data solutions do not even easily support (or are not configured) some of the most common methods of data matching. But this does not mean that you can automatically upload data to your database without standards or modeling. You do not need to be assertive, but be consistent. If you are going to make a mistake, make the same mistake. This means you will need to set standards to make it clear to developers what they expect.

Basic advanced practices, such as common names, can make a huge difference in the knowledge of end-users.

  1. Standardize names – To ensure that analysts can quickly identify what the columns mean, having common naming conventions is required. Using consistent data types such as “ts”, “date”, “is_’ and so on, ensures that everyone knows what they are looking at without looking at the data documents. This is similar to the old design principle of the adjective that describes the column recommendations.
  2. Manage data structures – Overall, trying to avoid complex data structures such as arrays and dictionaries in key layers is beneficial because it reduces the confusion that analysts may have.
  3. Organize IDS as much as you can – IDs allow analysts to integrate data across multiple systems. Looking back on my career, this excellent practice has had a profound effect. When IDs were not suspended, I was completely unable to join the data sets, no matter how talented I was. By comparison, when I worked for companies that had systems in place to ensure that system IDs were tracked, I was able to fluently join very different data sets.
  4. Improve processes with software teams – Less than the best performance and the biggest problem you will face is how you ensure that your data does not change too much. Of course, you can store your data in JSON or non-built-in data sets in the raw layer. But as a data engineer, the more you understand what changes in data platforms and organizations are happening upstream, the more you can avoid any failures.

Higher Levels of Data Modeling Concepts

There are many different schools of thought when it comes to building a data warehouse. Galen B recently published “Read Google Data Developers: Dimensional Data Model Dead”. Now, this has received mixed reviews, as there are still many people who strongly support traditional data modeling, but no matter which camp you live in, it is important to note that there is no fast way to a solid basic data model.

Your data engineering team will need to take some time to understand how data is used, what it stands for, and what it looks like. This will ensure that you create data sets that your colleagues will want to use and use effectively. It all starts with the same stages of data processing.

Here are the key sections for most corporate data matching patterns:
  1. Raw – This layer is usually stored in S3 buckets or perhaps a raw table used as the first layer of data. Teams can then conduct a rapid data test to ensure that all data remains healthy. It can also be reconfigured in case of accidental deletion.
  2. Stage – Some form of pre-data processing is usually inevitable. Next, data teams rely on stage layouts to make the first pass of their data. To some extent or another, there is often duplicate data, heavily embedded data, and newly inconsistently named data standardized on a staging layer. Once the data is processed, there will usually be another QA layer before uploading the data to the main data layer.
  3. Core – This layer is where you will find the company’s database. This is where you can track everything that is done or done in the business to the level of granularity. You can think of this layer as a place where all the various organizations and relationships are kept. It is the foundation on which everything else is built.
  4. Statistics – The analysis layer is usually broadly pre-assembled tables to reduce the number of errors and logical applications that can occur as analysts progress on the main data layer.
  5. Integrated – At the top of the analysis layer there is often a combination of metrics, KPIs, and aggregated data sets that need to be created. These metrics and KPIs are used for reporting as dashboard go-to directors, C-suites, and performance managers who need to make decisions based on changing KPIs over time.

Why Invest In Best Practices

Building a solid data storage system, whether a data warehouse or a data lakehouse, gives you a solid foundation on which to build. Whether you are building a data product or doing research, a well-defined basic data layer allows everyone in the company to build their data products with complete confidence in the accuracy of the data.
Additionally, performing small tasks such as setting up IDs makes it easy to join data across all system data sets. This means that end-users as data analysts and scientists can create analyses in very different data sets.
And we do not even consider all the benefits that people get when column names have an expected skip, such as not spending time analyzing columns before using them because they are not sure what kind of data they should expect from what is being said. column. I know, funny things. While all of these advanced processes take time, they ensure that, over time, a company can make decisions with a high level of data reliability.

Conclusion

Companies invest in centralized data storage systems because they provide easy access to large amounts of data without the need to pull data directly from all data sources, place it in excel, and convert all that data there. That is why companies invest so much in data warehouses, data pools, and data warehouses. Creating any type of data storage system always requires a certain level of configuration and data modeling (in one way or another).
  • Data modeling is Important before actually starting utilizing that data. It is easy to understand, easy to operate, and It’s solid and faithful.
  • Data Models Practices like standardizing names, managing data structure, organizing IDE & improve processes with the software team.
  • Higher level of Data modeling concepts &  key sections for most corporate data matching patterns including Raw, core, Stage, and Statistics. I hope you understand the importance of data Modeling and use it.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Chetan
Chetan

Data Analyst who love to drive insights by visualizing the data and extracting the knowledge from it. Automating various tasks using python & builds Real time Dashboard's using tech like React and node.js. Capable of Creaking complex SQL queries to fetch the accurate data.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Joohnse Komala
Joohnse Komala

standards or modelling are totally different aspects. I couldn't find any smell of data warehousing and any modeling techniques discussed here and you never mentioned how data engineers benefit from it. Please bring the context in the article to align with your topic or other way round. just advise. hope you got my point.

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details