Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

AWS Glue: Simplifying ETL Data Processing

Ansaba
Ansaba Last Updated : 29 Dec, 2022
6 min read

This article was published as a part of the Data Science Blogathon.

AWS Glue ETL Service - headingSource: https://cloudnesil.com/2019/08/20/aws-glue/

Introduction

If you are familiar with databases, or data warehouses, you have probably heard the term “ETL.” As the amount of data at organizations grow, making use of that data in analytics to derive business insights grows as well. For the efficient management of these data, ETL (Extract, Transform, and Load) process is necessary.

AWS helps you perform ETL tasks, especially complex ones, with AWS Glue.

AWS Glue is simply a serverless ETL tool that helps you with the analysis and lets you enjoy the results in minutes. Moreover, you do not have to manage any cloud infrastructure, as it is an automatically managed service that’s fully maintained by AWS.

Now, why is it the choice of Data scientists and analysts?

As a cloud-based tool, it reduces costs and decreases the time spent creating ETL jobs. It also removes all the complex tasks associated with ETL data processing as it associates with various platforms for quick data analysis.

This article covers an overview of AWS Glue, why it is an effective tool in ETL data processing, Glue’s architecture and components, it’s working, and some of its use cases.

What Is AWS Glue?

AWS Glue is a serverless data integration service developed to extract, transform, and load data called the ETL process. Because it is a serverless service, users don’t have to manage infrastructure, configure, scale, or provision resources. Glue provides easy-to-use tools to create and follow up job tasks based on schedule, event triggers, or on-demand. AWS Glue provides visual and code-based interfaces to make data integration more manageable. With AWS Glue Studio, users can visually create, run, and monitor ETL workflows with just a few clicks.

Why Consider AWS Glue for ETL Data Processing?

  • Automatic ETL Code Generation: By specifying the source and destination of data, AWS Glue can generate the code in Python or Scala for the entire ETL pipeline. This feature streamlines the data integration operations and allows users to parallelize heavy workloads.
  • Only Pay for the Resources You Use: Glue is cost-effective, and users only have to pay for the resources they use. If the ETL jobs need more computing power from time to time but consume fewer resources, users don’t have to pay for the peak time resources outside of this time.
  • Developer Endpoints: It is helpful for users who prefer to manually create and test their own custom ETL scripts. It can debug, edit and test autogenerated code. Using Developer Endpoints, custom readers, writers, or transformations can be created, which can further be imported as custom libraries into Glue ETL jobs.
  • Automatic Schema Discover: Glue uses crawlers that parse the data and its sources or targets. Crawlers obtain schema-related information and store it in the data catalog, which helps to manage jobs. Hence, users don’t need to design the schema for their data individually to cater to the complex features of ETL processes.

AWS Glue’s Architecture

AWS Glue's architecture showing components such as Data Stores, Crawlers, Data Catalog, Job Scheduling System, and the ETL Engine.

Source: https://docs.aws.amazon.com/glue/latest/dg/components-key-concepts.html

AWS Glue comprises many individual components, such as the Data Catalog, Job scheduling system, Crawlers, ETL Engine, Datastore, etc. Glue depends on the interaction between these components to develop and maintain ETL operations. The different components can be orchestrated through Glue workflows to get a functional data processing pipeline. Workflows define and visualize the order in which crawlers and jobs are supposed to be started to facilitate the data transformation.

The components of the Glue architecture are:

  • Data Catalog: It holds the structure of the data and stores metadata needed for the system to work efficiently. It contains information in metadata tables, where each table points to a single data store. Data catalog stores information related to data, queries, schemas, data sources and destinations, partitions, etc.
  • Job Scheduling System: It is responsible for starting jobs based on various events. The System deals with automating and chaining ETL pipelines by creating an execution schedule or event-based jobs triggering.
  • Crawlers: It retrieves data from the source and build metadata tables in the Data Catalog after determining the schema. Crawlers explore several data repositories in a single encounter and decide the ideal schema for those data.
  • ETL Engine: It handles ETL code generation in Python or Scala and allows code customization. Users can visually compose data transformation workflows and run them on Glue’s Apache Spark-based serverless ETL engine.
  • Data Store: It denotes a data repository that keeps users’ data for a long period. Some of the data stores include AWS S3 buckets and other relational databases.

 

How Does AWS Glue Work?

First, the user has to determine the data source, which is used as input to a process or transformation. Data crawlers analyze and extract necessary information about data sources and targets. It classifies data to determine the format and associated properties of the raw data and groups data into tables or partitions. This data is sent and stored as metadata table definitions to the Glue Data Catalog. After Glue Data Catalog is completely categorized, the data will be ETL-ready, searchable, and queryable. Users can create a script, schedule ETL jobs, or set the trigger events for the jobs to start. While executing the task, the script will extract data from the data source, transform and load it to the data target. Hence the ETL (Extract, Transform, Load) process will be completed.

Data Sources Supported by AWS Glue

Data Sources supported by AWS Glue include amazon S3, Amazon DynamoDB, kafka, amazon KINESIS, Amazon DocumentDB, and AWS RDS.

 

Use Cases

  • Build Event-driven ETL Pipelines: AWS Glue is helpful while building event-driven ETL workflows. It allows developers to start AWS Glue workflows based on events delivered by Amazon EventBridge. This feature enables users to trigger a data integration workflow from any events from AWS services, SaaS providers, or custom applications. Users can start an ETL job by invoking AWS Glue ETL jobs through the AWS Lambda function whenever new data is available in Amazon S3.
  • Non-native JDBC Data Sources: AWS Glue has native connectors to data stores using JDBC drivers, and you can use this in AWS or anywhere else on the cloud as long as there is IP connectivity. Using the JDBC protocol, Glue natively supports data stores like Amazon RDS and its variants, Amazon Redshift, etc. For data sources that Glue doesn’t natively support, like SAP Sybase, IBM DB2, Pivotal Greenplum, or other RDBMS, users can import custom database connectors into AWS Glue jobs from AWS S3.
  • Data Transformation With Snowflake: AWS Glue has a fully managed environment that integrates easily with Snowflake’s data warehouse. With Snowflake and AWS Glue, users can benefit from optimized ETL processing that is easy to use and maintain. Together, these services manage data ingestion and transformation pipelines with more flexibility.
  • Querying With Athena: When using Athena with the AWS Glue Data Catalog, users can use AWS Glue to create databases and tables (schema) to be queried in Athena. Otherwise, Athena can be used for schema creation which can then be used in Glue and related services. AWS Glue jobs help to transform data to a format that optimizes query performance in Athena. It is possible to push data outside of AWS to the Amazon S3 bucket in a suboptimal format for querying in Athena. Users can configure AWS Glue ETL jobs and run them automatically based on triggers.

Conclusion

In this article, we have explored AWS Glue, a cost-effective service that provides easy-to-use tools to sort, validate, categorize, clean, and move data stored in warehouses and data lakes. It works with ETL pipelines and is compatible with other AWS services. Moreover, it provides centralized storage and operates with semi-structured data.

Key Takeaways:

  • AWS Glue is a serverless data integration service developed to extract, transform, and load data called ETL process.
  • By specifying the source and destination of data, Glue can generate the code in Python or Scala for the entire ETL pipeline.
  • AWS Glue architecture comprises Data Catalog, Job scheduling system, Crawlers, ETL Engine, Datastore, and so on.
  • A data catalog holds the structure of the data and stores metadata needed for the system to work efficiently.
  • Crawlers retrieve data from the source and build metadata tables in the Data Catalog after determining the schema.
  • AWS Glue has a fully managed environment that integrates easily with Snowflake’s data warehouse.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

AWSAWS GlueblogathonETLETL PipelineETL Process
Ansaba
Ansaba

I'm a cloud enthusiast and have done graduation in computer science. My skills include C, C++, Php, Python, Cloud services(AWS, GCP), shell scripting, MySQL, etc. I have hands-on experience and have done several projects and certifications in areas of Python, Php, MySQL, Cloud (GCP, AWS), and C++

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
Top 10 Must Use AI Tools for Data Analysis [2024 Edition]
Top 10 Must Use AI Tools for Data Analysis [2024 Edition]
AWS Glue for Handling Metadata
AWS Glue for Handling Metadata

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details