Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Statistical Inference Using Python

E
ELLURU PAVAN KUMAR Last Updated : 03 Feb, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Hey folks!

Data science is an emerging technology in the corporate society and it mainly deals with the data. Applying statistical analysis to data and getting insights from it is our main objective. A company wil store millions of records for analysis. A data scientist will collect all the required data and conduct statistical operations to arrive at conclusions. This type of statistical analysis is called Descriptive Statistics. Suppose they collect a subset of data, known as Sample in statistical terminology, from the entire data known as Population. The sample is analyzed and conclusions are drawn about the population. This type of analysis falls under Statistical Inference (also known as Inferential Statistics).

In this article, I will explain some Statistical Inference concepts using Python Programming.

Context

1. Sampling Methods

2. Hypothesis Testing

1. Sampling Methods

Why this Sampling is necessary for Statistical Analysis? Because to reduce the

maximum permissible error, confidence level, and population variance/ standard deviation. maximum permissible error is defined as the difference between actual output and predicted output.

Confidence Level is defined as the probability that the value of a parameter falls within a specified range of values.

Population Variance is defined as the value of variance that is calculated from population data.

The determination of sample size depends on three major factors, such as:

1. confidence level

2. maximum permissible error

3.population variance/standard deviation

The formula given below can be used to calculate sample size:

Sampling method | Stastical Inference Using Python

Where,

n = sample size

critical = critical Z statistic for the specified confidence level

σ = population standard deviation

E = maximum permissible error

The critical Z value for a specified confidence level is found in the Z table. Z table for various confidence levels is given below.

Stastical Inference Using Python

There are two types of sampling methods

1.1 Random Sampling

1.2. Stratified Sampling

Let’s take an example, Suppose 1000 students are present in a class and select 10 students with different characteristics(like marks, etc..) from them.

population=1000

sample =10

1.1. Random Sampling

If we use random sampling, It randomly selects the 10 students(sample) from 1000 students(population).

Drawback

By using Random Sampling, there is a chance to select the same type of students as a sample. It may give a biased result and such a sample is called a Biased Sample.

a=[]
for i in range(1,1001):
    a.append(i)
#Importing the NumPy library
import numpy as np
#Choosing 8 Random states from ‘states’ without repetition
np.random.choice(a, size=10, replace=False)


output:












array([715, 864,  18, 911, 309, 115, 598, 294, 651, 578])












To overcome this we can use Stratified Sampling.


1.2. Stratified Sampling


If we use Stratified Sampling, the population is divided into groups based on characteristics. these groups are called Strata.


The sample is chosen randomly from each of these groups.


Suppose you have a list of 12 employees along with their department and job level information


You can sample the data by grouping it based on department and job level. There are two departments (D1 and D2) and two job levels (2 and 3).




Stratified Sampling | Stastical Inference Using Python 




#Taking a random sample from the population using the groupby function based on Department and Job Level
data.groupby(['Dept','Job_Level'], group_keys=False).apply(lambda x: x.sample(1))


output:




Stastical Inference Using Python 




You can observe that the output sample has all combinations of ‘Dept’ and ‘Job_Level’.


The above output is the stratified sample.


2. Hypothesis Testing


Why hypothesis testing?


Let’s consider an example, The data scientist has successfully estimated the population means, population variance, and standard deviation using various point estimation techniques.


According to the data scientist, the confidence interval of the population mean is (408 to 417). The client wants to verify this claim.


Therefore, the client randomly chooses 100 students from the country and conducts the surprise test (consisting of the same questions) for them. The mean marks scored by those 100 students are found to be 403, which does not lie in the range specified by the data scientist.


Now the question arises, whether this observation is sufficient for the client to conclude that the estimation by the data scientist is not valid.


In inferential statistics, a Hypothesis Test is conducted to find answers to such questions.


What is hypothesis testing?


In statistics, the hypothesis is a statement about the population and it deals with collecting enough evidence about the hypothesis. Then, based on the evidence collected, the test either accepts or rejects the hypothesis about the population.


Hypothesis testing needs to be performed to find evidence in support of this hypothesis. Based on the evidence found, this hypothesis can be accepted or rejected.


In each hypothesis testing, there are two parameters Null hypothesis and Alternate hypothesis.


Null Hypothesis (H0): It is a statement that rejects the observation based on which the hypothesis is made. You can start the hypothesis testing considering the null hypothesis to be true. It cannot be rejected until there is evidence that suggests otherwise.


Alternate Hypothesis (Ha): It is a statement that is contradictory to the null hypothesis. If you find enough evidence to reject the null hypothesis, then the alternative hypothesis is accepted.


if the probability of occurrence of the given data is less than the level of significance (0.05) you can reject the null hypothesis.


if the probability of occurrence of the given data is greater than or equal to the level of significance (0.05) you cannot reject the null hypothesis.


steps to calculate the Hypothesis:-


Step 1: Let assume the null hypothesis, alternate hypothesis, and the level of significance.


Step 2: Calculate the P-value.


Step 3: Conclude whether to reject the null hypothesis or not based on the P-value i.e.


    

  • If P-value < significance level, then reject the null hypothesis
    • 

  • If P-value >= significance level, the null hypothesis cannot be rejected
    • 



Step 4: State the conclusion.


For the above example,


Step-1


Null hypothesis(H0): The estimate given by the data scientist is correct.


Alternate Hypothesis(Ha): The estimate given by the data scientist is incorrect.


Step-2


calculate P-Value:


#Importing the norm function from the scipy.stats module
from scipy.stats import norm
#Finding the probability of getting a value that is 0.56 standard deviation from mean using norm.cdf() function
print(norm.cdf(0.56))
output:- 0.712


Step-3


P-Value > level of significance.


So, it is failed to reject the null hypothesis.


Step-4


The estimate given by the data scientist is correct.


Conclusion


A statistical approach is the best to deal with data. I hope you understand the above concepts. if any queries feel free to contact me.


 


Reference


LinkedIn:-  https://www.linkedin.com/in/pavan970/


GitHub:-  https://github.com/pawankumarreddy1999


 


Read more articles on Statistics on our website.


 


The media shown in this article is not owned by Analytics Vidhya and are used at the Author’s discretion. 

                                                

                            

                                

                                    
                                      
                                  E
                                  
                                
                                

                                

                                    
                                    ELLURU PAVAN KUMAR                                  
                                                            

                            

                                                    

                        
                         
                    

                                       
                                        

                        
Free Courses

                        

                            
                                

                                    

                                        Generative AI
                                        
                                            
                                                
                                                
                                            4.7
                                        
                                    

                                    

                                        
Generative AI - A Way of Life

                                        
Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.

                                    

                                

                            
                        

                        

                            
                                

                                    

                                        Large Language Models
                                        
                                            
                                                
                                                
                                            4.5
                                        
                                    

                                    

                                        
Getting Started with Large Language Models

                                        
Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.

                                    

                                

                            
                        

                        

                            
                                

                                    

                                        Prompt Engineering
                                        
                                            
                                                
                                                
                                            4.6
                                        
                                    

                                    

                                        
Building LLM Applications using Prompt Engineering 

                                        
This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.

                                    

                                

                            
                        

                        

                            
                                

                                    

                                        LlamaIndex
                                        
                                            
                                                
                                                
                                            4.8
                                        
                                    

                                    

                                        
Improving Real World RAG Systems: Key Challenges & Practical Solutions

                                        
Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.

                                    

                                

                            
                        

                        

                            
                                

                                    

                                        RAG systems using LlamaIndex
                                        
                                            
                                                
                                                
                                            4.7
                                        
                                    

                                    

                                        
Microsoft Excel: Formulas & Functions

                                        
Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

                                    

                                

                            
                        

                    

                

                  
                  

                    
                    
                    

                        

                              
Recommended Articles

    

                        

                    

                

            

        

    

    
    
        
    
   
    
    


        

            

                 

                    
Responses From Readers

                    

    	

		 



Clear 


	

	                        

        








       
    
        
    
    

        

        

          

            

              
                

                  Write for us
                  
                    
                    
                  
                

                
Write, captivate, and earn accolades and rewards for your work

              
              
                
    
                  
  • Reach a Global Audience
    • 
                  
  • Get Expert Feedback
    • 
                  
  • Build Your Brand & Audience
    • 
                

                
    
                  
  • Cash In on Your Knowledge
    • 
                  
  • Join a Thriving Community
    • 
                  
  • Level Up Your Data Science Game
    • 
                

              
            

            

              
                imag
              
            

          

          
          

            

                

                    
                            

            

          

        

      

    


  
    
        

            

            

            

                

                    We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our  Privacy Policy & Cookies Policy.
                

                
Show details