Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

AWS Glue for Handling Metadata

Chetan
Chetan Last Updated : 02 Sep, 2022
6 min read

This article was published as a part of the Data Science Blogathon.

Introduction

AWS Glue helps Data Engineers to prepare data for other data consumers through the Extract, Transform & Load (ETL) Process. The managed service offers a simple and cost-effective method of categorizing and managing big data in an enterprise. It provides organizations with a data integration tool that formats information from different data sources and organizes it in a central repository where it can be used to inform business decisions.

AWS GlueSource: – https://aws.amazon.com

 

How AWS Glue Works

Glue uses ETL jobs to extract data from a combination of other cloud services offered by Amazon Web Services (AWS) and incorporate it into data lakes and data warehouses. It uses an application programming interface (API) to transform the extracted dataset for integration and helps users monitor jobs.
Users can schedule ETL jobs or select events to trigger the job. Once started, Glue extracts the data, transforms it based on code that Glue automatically generates, and loads it into Amazon S3 or Amazon Redshift. The glue then writes the metadata from the job to the embedded AWS glue data catalog.
The service can automatically find enterprise structured or unstructured data when it is stored in data lakes in S3, data warehouses in Amazon Redshift, and other databases that are part of the Amazon Relational Database Service. Glue also supports MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases running on Amazon Elastic Compute Cloud (EC2) instances in the Amazon Virtual Private Cloud.
The service then profiles the data in a data catalog, a metadata repository for all data assets that includes details such as table definition, location, and other attributes. The team can also use Glue Data Catalog as an alternative to Apache Hive Metastore for Amazon Elastic MapReduce applications.
Configure the transform properties in AWS Glue.
AWS Glue

Source: – https://kontext.tech/diagram/

 

To load metadata into the data catalog, the service uses Glue crawlers that scan the raw data store and extract schema and other attributes. An IT professional can customize browsers as needed.

Properties of AWS Glue

The basic properties of the glue are as follows:
  • Automatic schema detection. Glue allows developers to automate crawlers to retrieve schema-related information and store it in a data catalog that can then be used to manage jobs.
  • Task scheduler. Paste jobs can be set up and invoked on a flexible schedule using event-based or on-demand triggers. Multiple jobs can run in parallel, and users can specify dependencies between jobs.
  • Developer endpoints. Developers can use them to debug Glue and create custom readers, writers, and transforms that can then be imported into custom libraries.
  • Automatic code generation. The ETL process automatically generates the code; the only necessary input is the location/path to store the data. The code is either in Scala or Python.
  • Integrated data catalog. It is a unique metadata repository for data from different sources in the AWS pipeline. An AWS account has one catalog.

Advantages and Disadvantages of Using Glue

The advantages of AWS glue are as follows:

  • Fault tolerance. Failed jobs in Glue can be retrieved, and logs in Glue can be debugged.
  • Filtering. Bad data filters.
  • Support, support. It supports several non-native Java Database Connectivity (JDBC) data sources.
  • Maintenance and deployment. Easy maintenance and deployment because AWS completely manages the service.

Source: -https://aws.amazon.com

 

Disadvantages of AWS Glue include:
  • Limited compatibility. While AWS Glue works with several commonly used data sources, it only works with services running on AWS. Organizations may need a third-party ETL service if the resources are not AWS-based.
  • No incremental data sync. All data is first scheduled to S3, so Glue is not the best choice for real-time ETL jobs.
  • Learning curve. Teams using Glue should have a good understanding of Apache Spark.
  • Relational database queries. Glue has limited support for traditional relational database queries, only SQL queries.

AWS Glue Use Cases

1. Improve speed and reduce costs
Financial is a technology company that specializes in product-related financial services. These products are useful and help financial services companies in Southeast Asia. Its headquarters are in Jakarta, Indonesia. She wanted to elevate and transform everyday ETL jobs. The company opted for many other solutions but could not find the ideal solution for the needs. Ultimately, they decided to use AWS Glue, which paid off as the team could easily load the data, run the processes, and transform it for the next process on Redshift. It also allowed the company to speed up the process and save costs. This association could not only prove to be faster and more efficient, but it was also much cheaper than the proven alternatives. For a startup like Financial, cost savings were one of the deciding factors.
2. Build a data intelligence and analytics platform 
Burt Corporation is a start-up data company specializing in data products to transform the new online media landscape. It has New York, Berlin, and Gothenburg (Sweden) offices. Many major online media publishers use Burt’s data intelligence and analytics platform to understand and optimize their online marketing strategies. The platform must have effective data collection, processing, analysis, and decision-making capabilities to meet these requirements. Therefore, the company decided to use AWS Glue, Amazon Redshift, and Amazon Athena to meet these requirements. Burt transformed his data analytics and intelligence platform by integrating Amazon Athena and AWS Glue. As a result, Burt was able to deliver a solution that met the client’s requirements.
3. Optimized cybersecurity environment 
One of the world’s most recognized brands needs no introduction. It specializes in the technology sector and specifically in industrial production. Siemens is a German multinational corporation headquartered in Munich, Germany. She wanted to counter cyber threats with a smart system. This system was expected to prepare data, analyze it and make predictions using machine learning. The company decided to turn to Amazon Web Services for a solution. AWS experts recommended Amazon Sage Maker, AWS Glue, and AWS Lambda. AWS Glue played a key role in this process. It is a tool that extracts data and helps data scientists categorize data easily.
Optimized cyber security environment Source: -https://aws.amazon.com

ETL Engine

After the data is cataloged, it is searchable and ready for ETL jobs. AWS Glue includes an ETL script recommendation system for generating Python and Spark code (PySpark) and an ETL library for running jobs. A developer can write ETL code through the custom Glue library or PySpark code using the AWS Glue console script editor.
A developer can also import custom code or PySpark libraries. In addition, developers could upload code for existing ETL jobs to an S3 bucket and then create a new Glue job to process the code.

Plan and Organize ETL Jobs

AWS Glue jobs can be run on a schedule. A developer can schedule ETL jobs at minimum five-minute intervals. AWS Glue cannot handle streaming data.
The service allows for scheduled, on-demand, and task completion triggers if the development team prefers to organize tasks. A scheduled launcher runs jobs at specified intervals, while an on-demand launcher runs when the user prompts. With the task completion trigger, one or more tasks can be executed after the tasks are completed. These jobs can run concurrently or sequentially and can also be run from an external service such as AWS Lambda.

AWS Glue Price

 AWS charges users a monthly fee to store and access metadata in the Glue Data Catalog. With AWS Glue pricing, there is a per-second fee, either a minimum of 10 minutes or 1 minute (depending on the Glue version users have), for the ETL job and crawler execution. AWS also includes a per-second fee for connecting to a development endpoint for interactive development.

Conclusion

It uses an application programming interface (API) to transform the extracted dataset for integration and helps users monitor jobs. The service then profiles the data in a data catalog, a metadata repository for all data assets that includes details such as table definition, location, and other attributes. The team can also use Glue Data Catalog as an alternative to Apache Hive Metastore for Amazon Elastic MapReduce applications.

  • AWS Glues has Five Basic properties:- Automatic schema detection, Task scheduler, Developer endpoint, Automatic code generation, and Integrated data catalog. 
  • The advantages of AWS glue are as follows:- Fault tolerance, Filtering, Support, support, Maintenance, and deployment. Disadvantages of AWS Glue include:- Limited compatibility, No incremental data sync, Learning curve, and Relational database queries.
  • A scheduled launcher runs jobs at specified intervals, while an on-demand launcher runs when the user prompts. With the task completion trigger, one or more tasks can be executed after the tasks are completed. These jobs can run concurrently or sequentially and can also be run from an external service such as AWS Lambda.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Chetan
Chetan

Data Analyst who love to drive insights by visualizing the data and extracting the knowledge from it. Automating various tasks using python & builds Real time Dashboard's using tech like React and node.js. Capable of Creaking complex SQL queries to fetch the accurate data.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details