Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Accuracy and its shortcomings: Precision, Recall to the rescue

Rohit
Rohit Last Updated : 01 Dec, 2020
7 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Have you ever been in a situation, where you were not sure about which metric to apply to evaluate your models? These situations can be tackled by understanding the use-case of each metric.

Everyone knows the basic know-how of all frequently used classification metrics but when it comes to knowing the right one to use to evaluate the performance of their classification model, very few are confident in the next step to take.

Supervised learning normally falls either under regression (having continuous targets) or under classification (having discrete targets). However, in this article, I will try to focus on a very little yet very important part of machine learning, which being a favorite topic of interviewers – “who knows what”, can also help you get your concepts right on classification models and eventually on any business problem. This article will help you learn that when someone tells you that an ml model is giving 94% accuracy, what questions you should ask to know if the model is actually performing as required.

Accuracy and its shortcomings: Precision, Recall to the rescue

So how to decide on the questions that will help?

Now, that’s a thought for the soul.

We will answer that by knowing how to evaluate a classification model, the right way.

We will go through the below topics in this article:

  1. Accuracy

  2. Shortcomings

  3. Confusion Matrix

  4. Metrics based on the Confusion matrix

  5. Recap

After reading this article you will have the knowledge on:

  • What is the confusion matrix and why you need to use it?

  • How to calculate a confusion matrix for a 2-class classification problem

  • Metrics based on confusion matrix and how to use them

 

Accuracy and its shortcomings:

Accuracy (ACC) measures the fraction of correct predictions. It is defined as “the ratio of correct predictions to total predictions made”.

Accuracy text formula

 

Problem with accuracy:

It hides the detail you need to better understand the performance of your classification model. You can go through the below examples that will help you understand the problem:

  1. Multi-class target variable: When your data has more than 2 classes. With 3 or more classes you may get a classification accuracy of 80%, but you don’t know if that is because all classes are being predicted equally well or whether one or two classes are being neglected by the model.

  2. Imbalanced dataset: When you have imbalanced data (does not have an even number of classes). You may achieve an accuracy of 95% or more but is not a good score if 95 records for every 100 belong to one class as you can achieve this score by always predicting the most common class value. Example dataset – Breast cancer dataset, spam vs not spam dataset.

A typical example of imbalanced data is encountered in an e-mail classification problem where emails are classified into spam or not spam. Here, the count of spam emails is considerably very low (less than 10%) than the number of relevant (not spam) emails (more than 90%). So, the original distribution of two classes leads to an imbalanced dataset.

If we take two classes, then balanced data would mean that we have 50% points for each of the classes. Also, if there are 60-65% points for one class and 40% for the other class, it should not cause any significant performance degradation, as the majority of machine learning techniques can handle little data imbalance. Only when the class imbalance is high, e.g. 90% points for one class and 10% for the other, Accuracy and few other optimization measures may not be as effective and would need modification. 

Classification accuracy does not bring out the detail you need to diagnose the performance of your model. This can be brought out by using a confusion matrix.

 

Accuracy and its shortcomings: Precision, Recall to the rescue

Confusion Matrix:

Wikipedia defines the term as “a confusion matrix, also known as an error matrix is a specific table layout that allows visualization of the performance of an algorithm”.

confusion matrix

A confusion matrix for two classes (+, -) is shown below.

There are four quadrants in the confusion matrix, which are symbolized as below.

Precision recall

confusion matrix

  • True Positive (TP): The number of instances that were positive (+) and correctly classified as positive (+v). 
    • You predicted a mail is a spam and it actually is.
  • False Negative (FN): The number of instances that were positive (+) and incorrectly classified as negative (-). It is also known as Type 2 Error.
    • You predicted a mail is not spam and it actually is.
  • True Negative (TN): The number of instances that were negative (-) and correctly classified as (-).
    • You predicted a mail is not spam and it actually is not.
  • False Positive (FP): The number of instances that were negative (-) and incorrectly classified as (+). This also known as Type 1 Error.
    • You predicted a mail is not spam and it actually is.

To add a little clarity:

Upper Left: True Positives for correctly predicted event values.
Upper RIght: False Positives for incorrectly predicted event values.
Lower Left: False Negatives for correctly predicted no-event values. 
Lower Right: True Negatives for incorrectly predicted no-event values.

Metrics based on Confusion matrix:

  • Precision

  • Recall

  • F1-Score

Precision

Precision calculates the ability of a classifier to not label a true negative observation as positive.

                  Precision=TP/(TP+FP)

Using Precision

We use precision when you are working on a model similar to the spam detection dataset as Recall actually calculates how many of the Actual Positives our model capture by labeling it as Positive.

Recall (Sensitivity)

Recall calculates the ability of a classifier to find positive observations in the dataset. If you wanted to be certain to find all positive observations, you could maximize recall.

                    Recall=TP/(TP+FN)

Using Recall

We always tend to use the recall, when we need to correctly identify the positive scenarios, like in a cancer detection dataset or a fraud detection case. Accuracy or precision won’t be that helpful here.

F-measure

In order to compare any two models, we use F1-Score. It is difficult to compare two models with low precision and high recall or vice versa. F1-score helps to measure Recall and Precision at the same time. It uses Harmonic Mean in place of Arithmetic Mean by punishing the extreme values more.

Understanding the Confusion Matrix

Let’s say we have a binary classification problem where we want to predict if a patient has cancer or not, based upon the symptoms (the features) fed into the machine learning model (classifier).

As studied above, the left-hand side of the confusion matrix shows the class predicted by the classifier. Meanwhile, the top row of the matrix shows the actual class labels of the examples.

If the problem set has more than two classes, the confusion matrix just grows by the respective number of classes. For instance, if there are four classes it would be a 4 x 4 matrix.

  In simple words, no matter the number of classes, the principal will remain the same: The left-hand side of the matrix is the predicted values, and the top the actual values. What we need to check is where they intersect to see the number of predicted examples for any given class against the actual number of examples for that class.

While you could manually calculate metrics like confusion matrix, precision, and recall, most machine learning libraries, such as Scikit-learn for Python, have built-in methods to get these metrics.

Generating A Confusion Matrix In Scikit Learn

We have already covered the theory on working of the confusion matrix, here we will share the python commands to get the output of any classifier as a matrix. 

To get the confusion matrix for our classifier, we need to create an instance of the confusion matrix we imported from Sklearn and pass it the relevant arguments: the true values and our predictions.

from sklearn.metrics import confusion_matrix

c_matrix = confusion_matrx(y_test, predictions)

print(c_matrix)

 

Recap

In a brief recap, we looked into:

  • accuracy

  • problems it can bring to the table

  • confusion matrix to better understand the classification model

  • precision and recall and scenario on where to use them

We lean towards using accuracy because everyone has an idea of what it means. There is a need to increase the use of better-suited metrics such as recall and precision which may seem foreign. You now have an intuitive sense of why these work better for some problems such as imbalanced classification tasks.

right tool

Statistics provide us with formal definitions to evaluate these measures. Our job as a data scientist is about knowing the right tools for the right job, and this brings in the need to go beyond accuracy when working with classification models.

Using recall, precision, and F1-score (harmonic mean of precision and recall) allows us to assess classification models and also makes us think about using only the accuracy of a model, especially for imbalanced problems. As we have learned, accuracy is not a useful assessment tool on various problems, so, let’s deploy other measures added to our arsenal to assess the model!

Rohit Pant

Github

accuracyblogathonf1-scoreprecisionrecall
Rohit
Rohit

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Aashish garg
Aashish garg

Very well written. Its really helpful and got the concept quiet easily. Kudos to the author

123

N
N

Hi, Very nice explanation. Can you please let me know....what range is acceptable for specificity and f-measure for a good classifier

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details