Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Get to Know About Machine Learning Life Cycle

Ninad
Ninad Last Updated : 08 Jun, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

Introduction

The Machine Learning life cycle or Machine Learning Development Life Cycle to be precise can be said as a set of guidelines which need to be followed when we build machine learning-based projects. The ML life cycle helps to build an efficient machine learning project. The main purpose of the life cycle is to find a solution to the problem or project.

In this article, we will have a detailed study on what is meant by machine learning life cycle and what steps we must need to follow to implement machine learning life cycle. A beginner always thinks that making a machine learning project is just about finding the accuracy of the model or performing EDA(Exploratory Data Analysis). But when it comes to real-life products there is a lot of stuff which needs to be considered to make an end-to-end application. So let’s see what are various steps involved in making real-world machine learning projects.

Machine Learning Life Cycle

Framing the Problem

  • This is the basic as well as an essential step in developing a machine learning project.
  • In this step, we must figure out what is the problem statement of our project so that we can get to know every parameter of the problem.
  • Things to consider are what is the actual problem we are trying to solve, members in teams, how can we approach that solution, the cost involved, how can we get data, an algorithm which can be used, a framework to be used, where to deploy model, etc.

After this all things are figured out properly, then only we can proceed to further steps.

Gathering Data

For college students,  data is easily available from different websites most probably kaggle but in the case of companies, it is a different scenario.

There are different ways for companies to collect data:-

  1. APIs- Hit the API using python code and fetch data in json format.
  2. Web Scrapping- Sometimes data is not publicly available i.e., it is on some website so we need to extract it from there. For eg-Trivago uses this method to collect hotel prices data from every website
  3. Data Warehouse- Data is also stored in databases. But this data cannot be directly used as it is running data. So data from the database is stored in a data warehouse and then used.
  4. Clusters– Also data are sometimes in tools like the spark in the form of clusters basically big data so data is fetched through these clusters.

Data Preprocessing

  • Data which is gathered is most likely to be uncleaned i.e., structural issues,  missing values,  outliers, noisy data, etc.
  • This data can directly not be used in the machine learning model. So we need to preprocess this dirty data so that the data gathered can be useful.
  • Preprocessing involves removing duplicates, missing values, outliers, and scaling values.
  • Basically, by preprocessing we need data in a format which can be useful for machine learning models.

Exploratory Data Analysis

  • By seeing the name it is clear that we analyze the data by seeing the relationship between input and output.
  • This stage gives data insights by visualizing data using charts, graphs, outlier Detection, and univariate analysis i.e., independent analysis of each column(mean, standard deviation), bivariate analysis i.e., analysis of two columns, etc.
  • The whole idea behind this stage is to get a concrete idea about data.
  • It is also important to stage as it gives clear insights into data.
  • The more time we spend on EDA the more we get to know about data which helps in decision making while implementing the model.

Exploratory Data Analysis | Machine Learning Life Cycle

Feature Engineering and Selection

  • Features mean input columns. The whole idea is to create new columns in data by using existing columns or to make intelligent changes in existing columns to make analysis easier.
  • Feature selection is to select only particular columns which are necessary for implementing the model.
  • Sometimes it may happen there are many columns in data but we only require the few essential columns which are necessary for the model.
  • So in such a case, we will select only those few columns.

Model Training

  • Once we are sure about the data now we want to use this data to train our model. Training a model is required so that it can understand the various patterns, rules, and, features.
  • We train the model by using various algorithms and then evaluate algorithms by different metrics like accuracy score, mean squared error, etc.
  • The best model is selected and parameters are tuned so that performance of the model gets improved.
  • It is also known as Hyper Parameter Tuning.

Model Testing

  • Once our machine learning model has been trained on a given dataset, then we test the model.
  • In this step, we check for the accuracy of our model by providing a test dataset to it.
  • Testing the model determines the percentage accuracy of the model as per the requirement of the project or problem.

Model Deployment

  • In this step, we deploy the model in the real-world system. If the above-prepared model is producing an accurate result as per our requirement with acceptable speed, then we deploy the model in the real system.
  • But before deploying the project, we will check whether it is improving its performance using available data or not.
  • For deployment, we can use Heroku, AWS, Google cloud platform, etc. Now our model is online and serves user requests.

For an individual who is working on a personal project or college project, these are the complete steps.

The next 2 steps are used by companies.

A.Testing App/Software:

  • In this step, the companies roll out alpha/beta versions of the deployed model to a particular group of users or clients to check whether the model is working as per requirement.
  • The feedback is collected from these users and then worked upon. If the model is working correctly then it is rolled out to everyone.

B.Optimize:

In this stage companies use servers to take backup of the model, backup of data, load balancing(service the request if many users are requesting), rotting(frequently re-training model as data evolves with time). This step is generally automated.

Conclusion

Machine Learning systems are becoming more important day by day as the amount of data involved in various applications is increasing rapidly. ML technology is the heart of smart devices, household appliances, and online services. The success of machine learning can be further extended to safety-critical systems, data management, and High-performance computing, which holds great potential for application domains.

It is very important for any individual to follow all the steps in the lifecycle. By following these steps all the necessary requirements are collected and executed with proper planning (by performing EDA and feature engineering). These steps ensure that the machine learning model is well optimized and it gives the correct or desired results.

Machine Learning Life Cycle

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

blogathondata preprocessingEDAmachine learning
Ninad
Ninad

Hello, I am Ninad Bhase. Working as Android Developer and enthusiast about A.I. Completed my Computer Engineering from Mumbai University in 2022.
Love to write article as sharing increases knowledge.
Hope you checkout my other article as well

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
Steps to Complete a Machine Learning Project
Steps to Complete a Machine Learning Project
Understand Machine Learning and Its End-to-End Process
Understand Machine Learning and Its End-to-End Process

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details