Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Importance of Data Governance and its Principles

vijay
vijay Last Updated : 07 Jun, 2023
6 min read

This article was published as a part of the Data Science Blogathon.

Source: https://gocollectiv.com/wp-content/uploads/2020/11/Data-Governance.jpeg

What is DATA by Definition?

What is Data?

Source: https://images.twinkl.co.uk/tr/raw/upload/u/ux/what-is-data_ver_1.jpg

Data are details, facts, statistics, or pieces of information, typically numerical. Data are a set of values of qualitative or quantitative variables about one or more persons or objects.

While running a huge enterprise it is imperative that the data by any means necessary needs to be Protected with utmost care and security, in Today’s evolving world where each and everything you look at or store is in the form of data of some sort where a few examples could be social media, phone applications, cloud storage, customer databases everything in and around us right now is susceptible to any kind of data or security breach leaving the millions and billions worth data in wrong hands.

Thus, we all as individuals are strongly encouraged to bring in a few rightful practices or governance that could help avoid such massacre of data breaches that causes it to fall into the wrong hands. So let’s all practice hard to protect, secure, and abide by all the governance designed to protect against any kind of data infringement, breach, or exploitation of millions or billions of consumers who consume data in any form in life.

Impact of Data Breaches on lives

In 2021, organizations experienced a massive increase in cyber security breaches, even exceeding 2020’s startling numbers by 17 per cent.

The benefits of remote/hybrid work are undeniable, but IT leaders must mitigate the potential new security risks.

Over the past two years, we have seen an increase in data leakage, access control, and device theft. Organizations should follow a clinical cyber security framework for remote & hybrid working environments which should be updated and modernized.

Let’s look at some astonishing number of users getting affected due to these breaches

Data Governance

 

Source : https://www.logically.ai/hs-fs/hubfs/MicrosoftTeams-image%20(4)-Jun-02-2021-12-17-55-36-PM.png?width=2250&name=MicrosoftTeams-image%20(4)-Jun-02-2021-12-17-55-36-PM.png

Data Needs Governance? 

A data governance framework specifies who has the right to take what actions, by using what data and in what situations. Using data governance, an organization ensures that information is used effectively and efficiently to meet its objectives by following processes, roles, policies, standards, and metrics

Big data is the new gold, and companies are not going to take it away from them. Data must always be handled with care, right from storage to usage to security. In any other case, it can be misused by the wrong individuals.

Take on Data Governance

There are laws and rules that govern how data is handled. It should be handled with care and in an unbiased manner. One of the prime examples could be that of The Cambridge Analytica scandal that had already affected the Facebook universe and Zuckerberg. Nevertheless, there are rules and laws that govern it, just as there are for anything else of similar value. With these, you can ensure that your enterprise is protected against most threats while handling large amounts of data.

Each of these principles is vital to the success of any form of data governance and discipline because they guarantee that data at any stage of sharing and consumption is not breached or hacked unethically, preserving its integrity and authenticity of it. This must be observed and taken care of by big enterprises that have multiple levels of security and therefore adhere to the norms of good data governance at the global level

An Overview of Data Governance Principles

So? What are these rules? How are they implemented? There are seven of these principles that govern how the data is used. Some of these principles include the likes of Maintaining the integrity of the data, data Transparency, Accountability and Ownership of the data, Data Audit, Standardization of Data, Change Management, Stewardship.

To understand how each of these functions, let us review them individually.

Data Governance

Source: https://b-eye.com/content/uploads/2019/11/Principles-of-Data-Governence.jpg

Maintaining the Integrity of the Data

A principle of the highest importance is integrity. It depends on the entity using your data whether or not it is being used in the appropriate manner. Data integrity is maintained if their means and goals are ethical. In all decisions about the data, the participants must be honest and forthcoming. This can include decisions about actions, impacts, constraints, etc.

Transparency

In every case where data is used, proper transparency must always be maintained. To use data, as well as whose data is being used, all parties must understand how it is being utilized. Whenever there is a decision about usage or control, it must be communicated effectively to all parties involved. This will prevent any potential conflict in the future.

Accountability and Ownership of the Data

The ownership of the data must be defined. Appropriate procedures should be followed for defining access rights. Data governance applies to any data that is used across functions. As a result, data governance defines all decisions, processes, and controls related to data, i.e., its accountability.

Data Audit

Audits are permitted on every piece of data used. Any decision, control, and process about data that relates to data governance can be audited. Therefore, they must contain documentation proving compliance.

Standardization of Data

A company’s data is used by many teams. In this case, the data in one format might not be compatible with another. It is imperative that specific guidelines and rules be defined in order to standardize data.

In addition to these, there are rules for data definition, accessibility, security, and privacy.

Change Management

There may be some discrepancies in the data that require a change. As a result, there is always a risk of tainting the data. Therefore, data governance ensures proper change management activities, whether proactive or reactive. The data will include reference values, metadata, master data, and its use and structure.

Stewardship

Next, we must adhere to the principle of stewardship. Accountability goes hand in hand with responsibility. It is essential to appoint a data steward in any organization. All rules and regulations must be followed by the data steward. This holds true for groups of stewards as well. It is their responsibility to ensure that the data is stored and used appropriately. It is their responsibility to always follow the best practices when managing data.

Frequently Asked Questions

Q1. What are basic principles of data governance?

A. The basic principles of data governance include:
1. Data Ownership: Clearly define roles and responsibilities for data ownership, ensuring accountability for data quality, security, and compliance.
2. Data Stewardship: Designate data stewards who are responsible for data management, including data quality, metadata management, and ensuring adherence to data governance policies.
3. Data Quality: Establish data quality standards, processes, and controls to ensure data accuracy, completeness, consistency, and validity.
4. Data Security and Privacy: Implement measures to protect data confidentiality, integrity, and availability, while complying with relevant regulations and privacy requirements.
5. Data Lifecycle Management: Define processes for data creation, acquisition, storage, usage, sharing, archival, and disposal, ensuring adherence to legal, regulatory, and business requirements.
6. Data Standards and Metadata: Establish data standards, naming conventions, and metadata management practices to enable effective data discovery, understanding, and integration.
7. Data Governance Framework: Develop a structured and documented framework that outlines the governance policies, procedures, and decision-making processes to guide data management activities.
8. Continuous Monitoring and Improvement: Regularly monitor data governance activities, measure adherence to policies, and continuously improve data governance practices based on feedback and evolving needs.
These principles form the foundation of effective data governance, promoting data quality, security, compliance, and maximizing the value and utility of organizational data.

Q2. Why are data governance principles important?

A. Data governance principles are crucial for several reasons:
1. Data Quality: They ensure data accuracy, consistency, and reliability, enabling informed decision-making.
2. Compliance: They help organizations meet legal, regulatory, and privacy requirements related to data handling.
3. Security: They protect sensitive data, mitigating the risk of data breaches or unauthorized access.
4. Trust: They establish trust in data, fostering confidence among stakeholders and promoting data-driven strategies and initiatives.

Conclusion

Management of data can be a tedious task. Lack of guidelines and standardization can be puzzling when it comes to handling and using data. To learn how to use data effectively, guiding principles in data governance can be useful. As a result, data are effectively used, and promising results are produced.

In case you would like to connect with me and have opinions/feedback to share about content writing please go to my Linkedin profile: https://www.linkedin.com/in/vijaysharmads/

References

  • https://blog.thinscale.com/cybersecurity-for-remote-hybrid-working?utm_campaign=Top+60+ABM+%2526+Steve+Morell+-+Blog+-+IT+leaders+securing+remote+%2526+hybrid+workforces&utm_source=linkedin&utm_medium=paid&hsa_acc=503676840&hsa_cam=614392993&hsa_grp=197566463&hsa_ad=177800983&hsa_net=linkedin&hsa_ver=3&li_fat_id=7967cd26-15f6-4b66-88ea-9217d469b875
  • https://www.hitechnectar.com/blogs/data-governance-guiding-principles/
  • https://www.logically.ai/articles/data-breaches-in-india

The media shown in this article is not owned by Analytics Vidhya and are used at the Author’s discretion. 

vijay
vijay

||📊Lead Technical Analyst at HP||Connecting data to dollar||📋Certified - Data Scientist||SAP ERP||Microsoft Data Analyst,LSS Yellow Belt||đŸ„‡3*MVP,1*APAC Champion@HP||🏆Blogathon Winner’22@Xebia,2*AVCC||AVCC Member’22||

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details