Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

What is Deep Learning? DL vs ML vs AI Why Deep Learning is So Popular?
Applications of Deep Learning Multi Layer Perceptron Visualizing the Neural Network Understanding Decision Boundary Forward and Backward Propagation Intuition
Applications of Deep Learning Multi Layer Perceptron Visualizing the Neural Network Understanding Decision Boundary Forward and Backward Propagation Intuition
Gradient Descent Algorithm Variants of Gradient Descent Algorithm
Introduction to Loss Function Binary and Categorical Cross Entropy
Why Do We Need Activation Functions? Linear Activation Function Sigmoid and Tanh Function Softmax How to Select Right Activation Function?
Perceptron Weights in Perceptron Introduction to Artificial Neural Network
Understanding Forward Propagation Mathematically Understand Backward Propagation Mathematically Implementing Neural Networks in Python
Problems with Gradient Descent Gradient Descent with Momentum Adagrad and Adadelta
Introduction to Learning Rate Schedulers
Overview of Deep Learning Frameworks Implementing Neural Networks using Keras Functional API in Keras Implementing Neural Networks using Keras Hyperparameter Tuning of MLP in Keras
Understanding Early stopping Understanding Dropout Vanishing and Exploding Gradients Weights Initialization Techniques Implementing Weight Initializing Techniques Batch Normalization Image Augmentation Techniques Image Generator and Fit Generator Model Checkpointing Implementing Model Checkpointing Dealing with Class Imbalance Ensemble Deep Learning
Introduction to GPU and TPU TensorRT
Introduction to Unsupervised Learning How to Solve Unsupervised Learning Problems? Introduction to Autoencoders Implementing Autoencoders
A Beginners Guide to Codeless Deep Learning
TensorFlow Serving Build Deep Learning Models for Android
Introduction to PyTorch and Tensors Mathematical and Matrix Operations in PyTorch Important PyTorch Modules Implement CNN in PyTorch Transfer Learning in PyTorch Working with Text Data in PyTorch Building a RNN model in PyTorch Autoencoders in PyTorch

Understand Machine Learning and Its End-to-End Process

Shanthababu Pandian
Shanthababu Pandian Last Updated : 18 Dec, 2020
5 min read

This article was published as a part of the Data Science Blogathon.

What is Machine Learning?

Machine Learning: Machine Learning (ML) is a highly iterative process and ML models are learned from past experiences and also to analyze the historical data. On top, ML models are able to identify the patterns in order to make predictions about the future of the given dataset.

 

machine learning testimonials

 

Why is Machine Learning Important?

Since 5V’s are dominating the current digital world (Volume, Variety, Variation Visibility, and Value), so most of the industries are developing various models for analyzing their presence and opportunities in the market, based on this outcome they are delivering the best products, services to their customers on vast scales.

machine learning 5Vs

 

What are the major Machine Learning applications?

Machine learning (ML) is widely applicable in many industries and its processes implementation and improvements. Currently, ML has been used in multiple fields and industries with no boundaries. The figure below represents the area where ML is playing a vital role.

machine Learning Applications

 

Where is Machine Learning in the AI space?

Just have a look at the Venn Diagram, we could understand where the ML in the AI space and how it is related to other AI components.

As we know the Jargons flying around us, let’s quickly look at what exactly each component talks about.

 

machine Learning in AI space

 

How Data Science and ML are related?

 

realtion with Data science

Machine Learning Process, is the first step in ML process to take the data from multiple sources and followed by a fine-tuned process of data, this data would be the feed for ML algorithms based on the problem statement, like predictive, classification and other models which are available in the space of ML world. Let us discuss each process one by one here.

machine learning process

Machine Learning – Stages: We can split ML process stages into 5 as below mentioned in the flow diagram.

  1. Collection of Data
  2. Data Wrangling
  3. Model Building
  4. Model Evaluation
  5. Model Deployment

Identifying the Business Problems, before we go to the above stages. So, we must be clear about the objective of the purpose of ML implementation. To find the solution for the given/identified problem. we must collect the data and follow up the below stages appropriately.

Identify the stages

 

Collection of Data

Data collection from different sources could be internal and/or external to satisfy the business requirements/problems. Data could be in any format. CSV, XML.JSON, etc., here Big Data is playing a vital role to make sure the right data is in the expected format and structure.

data collection

Data Wrangling and Data Processing: The main objective of this stage and focus are as below.

 

Data Processing (EDA):

  1. Understanding the given dataset and helping clean up the given dataset.
  2. It gives you a better understanding of the features and the relationships between them
  3. Extracting essential variables and leaving behind/removing non-essential variables.
  4. Handling Missing values or human error.
  5. Identifying outliers.
  6. The EDA process would be maximizing insights of a dataset.

 

Feature engineering:

  1. Handling missing values in the variables
  2. Convert categorical into numerical since most algorithms need numerical features.
  3. Need to correct not Gaussian(normal). linear models assume the variables have Gaussian distribution.
  4. Finding Outliers are present in the data, so we either truncate the data above a threshold or transform the data using log transformation.
  5. Scale the features. This is required to give equal importance to all the features, and not more to the one whose value is larger.
  6. Feature engineering is an expensive and time-consuming process.
  7. Feature engineering can be a manual process, it can be automated
machine learning steps

Training and Testing:

  1. The training data is used to make sure the machine recognizes patterns of the data, cross-validation of data is used to ensure better accuracy and
    the efficiency of the algorithm which is used to train the machine.
  2. Test data is used to see how well the machine can predict new answers based on its training.
  3. The train-test split procedure is used to estimate the ML performance of algorithms when they are used to make predictions on data that is not
    used to train the model.
train and test

 

Training

  1. Training data is the data set on which you train the model.
  2. Train data from which the model has learned the experiences.
  3. Training sets are used to fit and tune your models.

Testing

  1. Test data is the data which is used to check if the model has
    learnt good enough from the experiences it got in the train data set.
  2. Test sets
    are “unseen” data to evaluate your models.

Train data: It trains our machine learning algorithm
Test data: After the training the model, test data is used to test its efficiency and performance of the model

The purpose of the random state in train test split: Random state ensures that the splits that you generate are reproducible. The random state that you provide is used as a seed to the random number generator. This ensures that the random numbers are generated in the same order.

data split

 

Data Split into Training/Testing Set

  1. We used to split a dataset into training data and test data in the machine learning space.
  2. The split range is usually 20%-80% between testing and training stages from the given data set.
  3. A major amount of data would be spent on to train your model
  4. The rest of the amount can be spent to evaluate your test model.
  5. But you cannot mix/reuse the same data for both Train and Test purposes
  6. If you evaluate your model on the same data you used to train it, your model could be very overfitted. Then there is a question of whether models can predict new data.
  7. Therefore, you should have separate training and test subsets of your dataset.

MODEL EVALUATION: Each model has its own model evaluation mythology, some of the best evaluations are here.  

  1. Evaluating the Regression Model.
    1. Sum of Squared Error (SSE)
    2. Mean Squared Error (MSE)
    3. Root Mean Squared Error (RMSE)
    4. Mean Absolute Error (MAE)
    5. Coefficient of Determination (R2)
    6. Adjusted R2
  2. Evaluating Classification Model.
    1. Confusion Matrix.
    2. Accuracy Score.
    3. AUC and ROC.|

Deployment of an ML-model simply means the integration of the finalized model into a production environment and getting results to make business decisions.

Deployment

So, Hope you are able to understand the Machine Learning end-to-end process flow and I believe it would be useful for you, Thanks for your time.

Shanthababu Pandian
Shanthababu Pandian

Shanthababu Pandian has over 23 years of IT experience, specializing in data architecting, engineering, analytics, DQ&G, data science, ML, and Gen AI. He holds a BE in electronics and communication engineering and three Master’s degrees (M.Tech, MBA, M.S.) from a prestigious Indian university. He has completed postgraduate programs in AIML from the University of Texas and data science from IIT Guwahati. He is a director of data and AI in London, UK, leading data-driven transformation programs focusing on team building and nurturing AIML and Gen AI. He helps global clients achieve business value through scalable data engineering and AI technologies. He is also a national and international speaker, author, technical reviewer, and blogger.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

varus
varus

Dear Shanthababu Pandian, I am name is varus Loudou and I have been very blessed with your "Understand Machine Learning and Its End-to-End Process" I fund this morning by serching on internet. I am verry interested in how to learn ML and become a professional in that field. Please, I would like to know if there is any online courses available or any other ways to get full formation in ML. Best, varus Loundou

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details