Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

An End-to-end Guide on Anomaly Detection with PyCaret

sai
sai Last Updated : 26 Jul, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Have you ever wondered how a person or a bank is notified of the wrongful transaction of his credit card, like how did system can notify that particular person or the bank about the transaction, which will help save his money by blocking that particular card immediately? This process is called Anomaly Detection (Outliers). Here, the credit card example comes under the fraud detection problem.

PyCaret
Source: Superior Data Science 

Outliers are the Data Points which lie outside the overall distribution of the dataset; outliers will have a huge impact on the results of any kind of analytics, from the basic analysis to model building. A few models are sensitive to outliers like Linear Regression etc. Even if we are trying to draw some insights from the data, removing the outliers before the analysis will make the result statistically significant.

What is Anomaly Detection?

Anomaly detection is a process which helps to find the Outliers in a dataset here; the process assumes that data will be in a certain range based on the data we have; if any data point is far away from that particular range, that will be considered as the Outlier it is one of the primary steps of Data Cleaning.

In today’s world, lots of data is generated, and monitoring the data is a crucial task where anomaly detection will help us to find any occurrence of the error; monitoring the error explains to us the origin of the error and helps us to inform the error so that the necessary action will be taken.

 

Anomaly Detection is commonly used for:

  • Data Cleaning
  • Fraud Detection
  • Detection based on sensors
  • intrusion detection
  • health monitoring systems

Anomaly Detection Techniques

There are quite a few techniques to detect the outlier. Let’s discuss one of my favourite techniques called THE FIVE NUMBER SUMMARY.

The five-number summary is a descriptive statistic with the help of five values; we can describe the data here. The five-number summary deals with the interquartile range (IQR), which helps to find the outliers in the dataset.

  • Minimum value:  we can find the minimum value using the lower fence formula, which is Q1-1.5(IQR)
  • Q1: Q1  denotes the first quartile, which consists of 25% of the data from the overall data distribution 
  • Median
  • Q3: Q3 denotes the Third quartile, which comprises 75% of data from the overall data distribution.
  • Maximum value: we can find the maximum  value using the higher fence formula, which is Q3+1.5(IQR)

Source: Statistics How to

Another popular technique for anomaly detection is the Isolation forest, an unsupervised anomaly detection technique which is similar to the Random forest, which is constructed based on multiple decision trees; the isolation forest works with the principle that the anomalies are the observations which are different from the data points we have, it follows the approach of processing the samples and randomly selected features to find the outliers in the data.

The approach of the isolation forest algorithm is to  Isolate the outliers from the data with the help of decision trees, Data Attributes are selected randomly to construct the decision trees, and the shortest path of the decision trees is considered as outliers which are easy to isolate from the data this process continues until all the data points are processed. At the end isolation level of each data, the point is noted, and then it generates the anomaly score, which defines whether the particular data point is an outlier or not, if the anomaly score is close to 1, it is likely to be an outlier, and if an anomaly score is less than 0.5 then that data point is not an anomaly.

Source: Machine Learning Geek

PyCaret

PyCaret is the open source learning library in python, which helps us to implement various machine learning tasks more quickly.

PyCaret is very useful. Its low code python library helps us to implement various tasks with few lines of code. Here rather than focusing much on coding our focus will be on experiments.

It is easy to understand and implement almost every task related to the machine learning projects like handing missing values, encoding, feature scaling or hyperparameter tuning.

Implementation of PyCaret

pip install pycaret
from pycaret.datasets import get_data

from pycaret.anomaly import *

anomaly = get_data('anomaly')

exp_name = setup(data = anomaly)

The setup function is one of the basic functions in pycaret, which is responsible for the data preprocessing tasks like handling missing values, encoding, performing the Train -Test – Split, and all other necessary data preprocessing steps; after initializing the setup function, it shows us the data types of all the features then it processes to the environment which will have the information we need to perform data science project tasks.

 

iforest=create_model('iforest')

plot_model(iforest)
PyCaret

By plotting the model, we will get the interactive 3d plot, which helps us understand every data point’s information clearly.

Iforest_predictions = predict_model(iforest, data = anomaly)
Iforest_predictions

Here in the output, we can observe the anomaly detection values and the anomaly scores of the entire data

PyCaret
anomaly=Iforest_predictions[Iforest_predictions['Anomaly']==1]
anomaly
PyCaret

From the output, we have filtered the anomaly from the entire data along with their anomaly score so that we can drop them, or we can also filter them based on the 0 anomaly value, which denotes that the data point is not outliers, which helps us to give a better result in any kind of analytics and also used to perform basic analysis or to build a model.

save_model(iforest, 'IForest_Model')
PyCaret

The final step is to save the save_model function allows us to save the entire transformation pipeline for later use shortly, using the Load_model function, we can use the saved model to predict the unseen data.

Conclusion

Anomaly detection is a wide concept; here in this guide, we have discussed a few concepts of anomaly detection and its implementation using Pycaret:

  • We have understood the concept of outliers and their use cases.
  • We have learnt two techniques to detect outliers. One is the five-number summary basic method using statistics. Another one is the model-based approach, Isolation forest.
  • We have understood the benefits of using Pycaret, which mainly helps us to focus more on experimentation.

 I hope you got a good understanding of anomaly detection with pycaret, share this guide with your network, and kindly let me know if there are any queries or feedback. Connect with me on LinkedIn.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

sai
sai

I am a data driven enthusiast with a curious mind exploring the field of data science, I have the insights into Machine learning and Deep learning , 2 years of hands-on experience in python with libraries such as Sklearn, TensorFlow, Keras, Pandas, Seaborn and NumPy.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details