Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Getting Started with Amazon SageMaker Ground Truth

Shikha
Shikha Last Updated : 07 Jul, 2023
7 min read

Introduction

In this era of Generative Al, data generation is at its peak. Building an accurate machine learning and AI model requires a high-quality dataset. The quality assurance of the dataset is the most critical task, as poor data causes inaccurate analytics and unidentified predictions that can affect the entire repo of any business and make a loss of billions or trillions of amount.

https://www.forbes.com/sites/moorinsights/2021/12/03/amazon-sagemaker--the-easiest-way-to-build-artificial-intelligence-models-became-even-easier/
Source: Forbes

Data labeling is the first step towards data quality assurance that makes it understandable for AI models. Nobody can rely on humans to label data as humans can’t label the unlimited/every day generating data, so here we learn about Amazon SageMaker ground truth, a fantastic technique to create an accurately labeled dataset.

This article was published as a part of the Data Science Blogathon.

Table of contents

What is Amazon SageMaker Ground Truth?

Amazon SageMaker Ground Truth is a self-service offering that makes creating an efficient and highly accurate dataset accessible by performing data labeling tasks. Ground Truth also offers you to use human annotators through third-party vendors, Amazon Mechanical Turk, or even our private workforce, and a managed experience to set up end-to-end labeling jobs.

https://www.edlitera.com/blog/posts/amazon-sagemaker-tutorial
Source: Edlitera.com

SageMaker Ground Truth can generate millions of automatically labeled synthetic data without any manual effort of data collection or labeling on our behalf. Ground Truth offers a data labeling facility for various data types, including images, text, and videos. It helps the machine learning models to ease the task of text classifications, segment segmentation, object detection, and image classification.

Building ML Model in AWS SageMaker

Use cases of Amazon SageMaker Ground Truth

Here are some industry use cases of SageMaker Ground Truth:

  1. Autonomous Vehicles: A large amount of labeled data is needed by training models for autonomous vehicles. SageMaker Ground Truth can annotate objects, such as cars, pedestrians, traffic signs, and road markings, to develop accurate perception models and helps with safe autonomous driving.
  2. Healthcare: Label Medical imaging datasets using SageMaker Ground Truth to train models for diagnosing and identifying diseases like cancer, brain tumors, and other abnormalities. It can also transcribe and annotate medical records for natural language processing (NLP) applications.
  3. Manufacturing: Labeling images and sensor data in manufacturing processes can help in quality control, defect detection, predictive maintenance, and optimizing production efficiency.

The flexibility of SageMaker Ground Truth enables its application across multiple industries where labeled datasets are required for training and improving machine learning models.

Automated Data Labeling via Ground Truth

Amazon SageMaker Ground Truth is the application of machine learning algorithms, it uses the concept of Active Learning to label the data automatically and accurately. Active learning is a type of machine learning technique used to identify complex data that the machine cannot understand in the first go, it extracts that data and send it out to the human for labeling. Let’s discuss the working of Ground Truth!

https://www.linkedin.com/pulse/efficient-accurate-data-labeling-amazon-sagemaker-milad-rezaeighale
Source: LinkedIn

Step 1: Data Storage

Collect the raw and unlabelled data from different sources and store it in the S3 bucket.

https://sagemaker-examples.readthedocs.io/en/latest/end_to_end/fraud_detection/1-data-prep-e2e.html
Source: Sagemaker

Step 2: Sending Data to Human

In this step, pick a random piece of a dataset and send it to the human for manual data labeling.

https://www.marktechpost.com/2022/09/28/a-primer-on-data-labeling-approaches-to-building-real-world-machine-learning-applications/
Source: Marktechpost.com

Step 3: Human Labeling

As soon as the workers received the data chunk, they started labeling it.

https://medium.com/anolytics/what-is-data-annotation-and-what-are-its-advantages-95766213351e

Step 4: Label Consolidation Algorithm

Amazon Sagemaker Ground Truth uses this label Consolidation Algorithm to eliminate the risk of human errors and improve the accuracy of labeled datasets. The working of the algorithm includes gathering all labels for each data point in the dataset followed by consolidating them into single labels depending upon the weight of the labels.

https://www.geeksforgeeks.org/sagemaker-exploring-ground-truth-labeling-ml/

Step 5: Resultant Dataset

Now, we stored the resultant dataset, a small labeled dataset.

Step 6: Amazon Sagemaker Model

Now we create a self-learning model based on the machine learning algorithms and install that with the customer account in order to train the model from the small labeled dataset the customer is creating so that it will label the rest of the unlabelled data on its own.

Step 7: Use the ML Model

In this step, we’re using the newly created ML model to label the unlabelled data points of the original dataset.

Step 8: Automated Labeling

Automated Labeling is applied to the remaining Dataset with the help of the Active Learning method.

Step 9: High Confidence

Here we check the confidence score of the model, and we apply the automated annotation only if the score of our model is high.

Step 10: Low Confidence

If the confidence score of the model is low, we can’t apply the automated annotation, and we will then send that portion of the data to humans for the sake of labeling. However, the model will automatically create a new dataset to train and improve its accuracy in this case.

The entire dataset undergoes a cycle of repeating these steps until it is fully labeled.

Impact of Amazon SageMaker Ground Truth to Increase the Accuracy

Sagemaker basically proposes two methods to enhance the training data accuracy:

1. Annotation Consolidation

The purpose of annotation Consolidation is to counteract the error/bias of each worker by sending each data object to two or more workers and then consolidating their responses into a single label for our data objects.

https://aws.amazon.com/blogs/machine-learning/annotate-data-for-less-with-amazon-sagemaker-ground-truth-and-automated-data-labeling/
Source: Amazon

After collecting data from various workers, it applies the consolidation algorithm to compare them.

Algorithm

  • Detect the outlier annotations that are disregarded.
  • Applies a weighted consolidation of the annotations by assigning higher weights to more reliable annotations.
  • The label assigned to each object in the dataset is a probabilistic estimate of a true label. The object may have multiple annotations, but the output is a single label for each object.
  • Although we can choose the number of workers to perform annotation, which will increase the accuracy of our labels, the issue is that it will also increase the labeling cost.

The annotation Consolidation function offered by Ground Truth applies to all predefined labeling tasks, including NER( name entity recognition), bounding box, semantic segmentation, and image and text classification. Let’s understand each function!

  • Named Entity Recognition(NER): The Jaccard similarity is used for cluster text selections in NER. It took the mode of the label to calculate selection boundaries, and if the mode is unclear, it will go with a label median. At last random selection will play the role of this breaker to resolve the most assigned entity label in the cluster.
  • Bounding Box Annotation: In bounding box annotation, the consolidation task is performed by grabbing the bounded boxes from various workers and selecting the most similar ones via the Jaccard index, or intersection over union, of the boxes and averaging them.
  • Multi-class Annotation Consolidation for Image and Text Classification: The consolidation is performed by estimating the true class depending upon the class annotations from separate workers via Bayesian inference.
  • Semantic Segmentation Annotation: The system considers each pixel of an image as a multi-class object and treats the pixel annotations from workers as “votes.” Additionally, it incorporates extra information from surrounding pixels by applying a smoothing function to the image.

2. Best Practices on Annotation Interface

The annotation Interface has various features to improve the accuracy or quality of human labeling tasks. This well-organized and designed interface help worker obtain an adequate dataset with minimal error. The best practices include displaying brief instructions on a fixed-side panel and excellent and bad-label examples. Also, it has a feature to highlight only the image boundary for the bounding box annotations by darkening the background.

Conclusion

We discussed how Amazon Sagemaker Ground Truth will help to generate high-quality datasets for the machine learning model. The key takeaways of this Ground Truth blog include the following:

  • Data labeling is the first step towards data quality assurance that makes it understandable for AI models.
  • It can generate millions of automatically labeled synthetic data without any manual effort of data collection or labeling on our behalf.
  • Annotation Consolidation and Best Practices on Annotation Interface are two ways Sagemaker can enhance training data accuracy.

Frequently Asked Questions

Q1. What do you mean by Amazon SageMaker Ground Truth?

A. A highly managed data labeling service that efficiently creates high-quality labeled datasets for training models. It combines automated labeling through machine learning and human review to deliver highly accurate annotations.

Q2. Explain the working of SageMaker Ground Truth.

A. SageMaker Ground Truth uses a combination of automated and manual annotation techniques. It provides a web-based interface for human reviewers to annotate data based on predefined labeling tasks. The service also incorporates options for active learning, where it trains models on labeled data to propose labels for the remaining unlabeled data, thereby enhancing annotation efficiency.

Q3. Which types of data can SageMaker Ground Truth annotate?

A. SageMaker Ground Truth supports various data types, including images, text, audio, and video. It provides annotation tools for each data type, enabling accurate labeling for different use cases.

Q4. Can SageMaker Ground Truth integrate with other AWS services?

A. Yes, SageMaker Ground Truth seamlessly integrates with other AWS services. Use Amazon S3 for storing data, Amazon Mechanical Turk for sourcing human reviewers, and Amazon Rekognition for automated image and video analysis.

Q5. Explain how does SageMaker Ground Truth ensure the quality of labeled data.

A. SageMaker Ground Truth employs multiple mechanisms to ensure high-quality annotations. It includes features like review workflows, built-in annotation consolidation, and active learning to minimize errors and improve the accuracy of labeled datasets.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion. 

Shikha
Shikha

I am a tech enthusiast, a student, and a learner. I am a critical reader and a lover of words who finds writing blogs interesting. I possess the capability to research and learn new technologies quickly.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details