Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Playing Data with Statistics

S
Suchithra Last Updated : 27 Mar, 2021
5 min read
This article was published as a part of the Data Science Blogathon.

Introduction

Instead of starting with, Statistics is everything and is everywhere, let me put it in some other useful statement. “Where there is Data, Statistic Flags”

We can do a lot with the data like analyzing, predicting, and so on. The most important thing is choosing the model that should be suitable for both the business problem and the data. For some budding statisticians, they will have confusion on which model suits on which data for the general analysis. Here, I am sharing some key points, where to use the adequate model. Before getting into the modeling part, first, we get some information about the basics of statistical terms. What I mean as the basic statistical terms are Descriptive, Predictive, and Prescriptive.

 

Descriptive, Predictive, and Prescriptive Statistics:

Descriptive, Predictive, and Prescriptive Statistics

Source

Without changing any values, looking into the data either by graphical or table is Descriptive, in statistics. In the Descriptive model, the values or data with converted to a diagrammatic representation. This makes the statistician or the audience, understand the overall view of the dataset. After getting the idea about the data, we can proceed to the main appropriate model. Not going to make any changes in the data but converting to a diagrammatic representation and the data is as it is. Too simple you are, Descriptive.

Predictive, as the name defines, we are going to predictive something with the data we hold. The keyword that needs to pop, when you see something related to predictive is “How much/ How many”. The expectation of something is on the term predictive. In some business problem, we can frame the keyword as, how much the profit will be in next week? Or how many products can be expected to sell next month? If any business problem suits the keyword, then go for the predictive modeling. Unlike Descriptive, Predictive needs some effort.

Prescriptive is the next phase of Predictive. Rather than stopping at “How much? Or How many?”, we can further develop with the question, “How to make something happen?”. For instance, to increase the profit to 30%, What we can do? Or What we can do to sell 20% more products? Or What are the possible ways to achieve the customer’s need? This prescriptive type of model comes under the suggestion category. Giving suggestions is not that much easier as we think. Of course, a lot of work and analysis should be taken part for giving the worthful suggestion. Prescriptive is way more meaningful!!

 

Model or Analysis:

Different types of data can be adopted with different types of statistical analysis. Some of the major models in statistics will be discussed here. And they all have many sub-divisions. But in this article, let’s have some notes, only on the major model and when to use them.

Statistical analysis

Source

Correlation

There will a finding between everything in the data world. Using the correlation technique is one of the ways for comparing and get how strongly one variable is related to the other variable. Correlation analysis helps in finding the relationship or association between two or more variables. Variables are nothing but factors. Correlation analysis works both in quantitative and qualitative data. Simply, the Correlation value lies between -1 and 1. How close the correlation value to 1, is that much strong positive correlation is present between the variables, and how close the correlation value to -1, is that much strong negative relationship between the variables. 0 defines no correlation between the variables/factors.

When one variable increases with the other variable, shows a positive correlation. (E.g., As sales increases profit will also increase). When one variable increases, the other tends to decrease, which is the negative correlation. (E.g., As absenteeism increases production decreases). This model sometimes also helps us in reducing the variable count. Like, we can choose the variable that has a strong relationship for doing more research. So, we can work with the really needed variables.

 

Regression:

Regression analysis is useful when we need to find the dependencies of one variable on the other. This will also work for qualitative and quantitative data. The regression value lies between 0 and 1. 1 defines the perfect fit and 0 defines no fitting at all. This gives us how well one variable is dependent on the other variable. Using the regression analysis, we can perform predictive modeling.

 

Statistics Regression

Source

 

For example, a best educational institution is dependent on the qualified staff, environment, technology usage, etc., for such data we can prefer using the Regression analysis and this is a more appropriate model. This gives us how much a dependent variable is dependent on other independent variables. As per our example, the dependent variable is the institute and the independent variables are staffs, environment, technology usage, etc. So, do you want to know how one variable is dependent on another, then one of the best options to go for is Regression?

 

Survival Analysis:

If the data contains time duration and some incident occurrence, we can go for survival analysis. Survival analysis is most useful in the clinical trial data and in product life expectancy (E.g., how much time will an expected incident takes to happen). The most common term used in the survival analysis is censoring and is one of the major problems in survival analysis. Censoring means missing complete information on the survival time (i.e., we have the information on duration but not complete information).

Survival analysis should have a bivariate variable. Considering industrial data, we will have the time period of how long a machine can work. The dataset will contain the time and whether the machine failed or not as an event. Here, the event will be denoted as 0 and 1. In this case, prefer Survival Analysis. When we talk about survival analysis, the value lies between 0 and 1, which is plotted with the time period. Using the plotted graph, we can interpret the data.

Depending on the data and business problem, choosing the analysis may vary. There are a lot of packages to do the analysis so it will not be that much complicated for you to do a statistical model. But, based on the output, drawing the inference is most important and is where more attention to be provided. Hope you found some idea about, when and where to use the respective model.

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion.

S
Suchithra

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Robert Sasaka
Robert Sasaka

This was a very insightful article. Kindly send me more of them. Am trying to research on the use of logistic regression to predict the spread of Corona virus.

123

Suchithra Ponmudi
Suchithra Ponmudi

Thank you for your response. Will do for sure 👍

123
Vignesh DN
Vignesh DN

Useful information and the analysis was pretty good (survival analysis )

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details