Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

How to Use Chi Square to Fuel A/B Test?

aakash93
aakash93 Last Updated : 15 Jun, 2023
9 min read

Introduction

You may have heard that “Customer is King“. This is because the customer decides which product will stay and which will not. Whether marketers treat a customer as a ‘King‘ or not, he is always a ‘King’. He has the money the marketers want. He is not going to give that away for free. So, how do we know which product is accepted more by customers? The answer is to do an experiment known as A/B tests.

In this article, I will talk about A/B Test and we will also see a method that works well for comparing click rates across different advertisements within a campaign. I will also explain how to calculate the same in Excel and in Python and talk about how to interpret the results.

This article was published as a part of the Data Science Blogathon.

Table of contents

What is A/B Testing?

A/B testing, also known as split testing, compares two versions of a variable (for example, a advertisement A vs advertisement B), showing each version to equal numbers of users at random to determine which performs better against a business goal.

A/B testing can be useful when you want to test metrics of two variables against each other. However, traditional A/B testing will only take you so far. Keep reading to learn more about the limitations of A/B testing and discover an alternative approach.

Let us understand the same from a Data Scientist’s perspective. For a Data Scientist, experiments are important part of day to day work. But how do we validate those experiments?

Let’s say you observe the below metrics for an email advertisement campaign.

chi-square | A/B Test
image.png

Now, you want to understand whether the click rates of the advertisement are different.

So, you can calculate the click rate for each advertisement (like below)

chi-square | A/B Test
image.png

Inference: You see that Advertisement B has better click rates. But is it significant enough??

Once, the Data Scientists create a solution to the problem, they usually deploy the solution and perform an A/B test to evaluate whether the solution is performing well practically.

Limitations of Traditional A/B Test

Traditionally, most Data Scientists use t-test (or z-test) to determine the significance of A/B test. It is generally used to compare the means of 2 groups to understand of there is any statistical difference. t-test assumes that the distribution is Gaussian (or normal). However, it may produce non-reliable metrics if the distribution is not normal.

Since, we cannot always have a normal distribution with the data, we need to use a better and more generic approach.

In this article, I will talk about a particular A/B Testing method that works well for comparing click rates across different advertisements within a campaign. I will also explain how to calculate the same in Excel and in Python and talk about how to interpret the results.

What is Chi-square (or Chi-Sq)?

Chi-sq test is used to determine associations between 2 or more categorical variables. The Chi-square formula is used in data that consist of variables distributed across various categories and helps us to know whether that distribution is different from what one would expect by chance.

Understanding with example:

We will use the same example we saw in the introduction. Since, we are doing a statistical test, lets understand a few terminologies.

Null Hypothesis : H0 : The two categorical variables have no relationship (independent)

Alternate Hypothesis : H1 : There is a relationship (dependent) between two categorical variables

We will define a significant factor to determine whether the relation between the variables is of considerable significance. Generally, a significant factor or alpha value of 0.05 is chosen. This alpha value denotes the probability of erroneously rejecting H0 when it is true.

A lower alpha value is chosen in cases when we expect more precision. If the p-value for the test comes out to be strictly greater than the alpha value, then we will accept our H0.

Actual Calculations Using Excel

Step 1: Create a contingency table for observed values

Create a Total column and row. Also, create a Click rate row. It will represent the Click/Total for each advertisement.

chi-square | A/B Test
image.png

Step 2: Calculate Expected values

Create a contingency table for expected values.

Expected Frequency = (Row Total x Column Total)/Grand Total

Value for the 1st cell = (51*100)/200 = 25.5

You will get the below table for the expected values.

image.png
image.png

Notice how the total values are same, but other cell values have changed.

Step 3: Calculate Chi-sq statistic

Use the tables created in above 2 steps and calculate the chi-sq statistic using the below formula.

image.png
image.png

Here,
O => Observed values or the actual values that we saw in Step 1.

E=> Expected values that we computed in Step 2.

χ2  => Chi-Sq statistic

For each cell of the table,

  • Numerator = squared value of (Observed value – Expected value)
  • Denominator = Expected value
  • Chi-Sq statistic = Numerator/Denominator

For e.g. Value for the 1st cell = (25 – 25.5)^2/25.5 = 0.01

Similarly, calculate values across all cells and add them up. See the highlighted cell in Yellow.

image.png
image.png

We get an overall Chi-sq value of 0.03

Step 4: Calculate DOF (Degrees of freedom)

Definition: In statistics, Degrees of freedom are the number of independent variables that can be estimated in a statistical analysis and tell you how many items can be randomly selected before constraints must be put in place.

E.g. there are 100 people who were recommended Advertisement A through email. A person can either Click or not Click the advertisement (N=2). So, if there are 25 people who clicked on that Advertisement, we can calculate that (100-25=) 75 people did not click the advertisement. So, we need only 1 variable to ascertain that information. Hence, the DOF will be 1. It can be calculated using (N-1).

Importance of DOF:

Degrees of freedom are important for finding critical cutoff values for a statistical test.

Degree of Freedom for a Chi-sq Test = (rows − 1) * (columns − 1)

DOF = (2 − 1)*(2 − 1) = 1×1 = 1

Step 5a: Use Critical value table to accept/reject Null hypothesis.

Now, we need to find the critical value of the chi-square distribution.

We can obtain this from the chi-square distribution table.

Now, let us look at the table and find the value corresponding to 1 degrees of freedom and a 0.05 significance factor.

chi-square | A/B Test
chi-sq-dist2.png

The tabular or critical value of chi-square here is 3.841

Since, our calculated chi-sq value <= Critical chi-sq value, we fail to reject null hypothesis.

In simpler terms, there is no significance in clicks between the two advertisements.

Step 5b: Calculate and interpret p-value to accept/reject Null hypothesis.

Assuming that we already have the observed values from step 1, use the below steps for Excel:

  1. Calculate Expected value using Step 2 in Excel.
  2. Now, we can directly use the CHISQ.TEST function in Excel
  3. Formula = CHISQ.TEST(Observed Range, Expected Range). Here, we will use CHISQ.TEST(C4:D5,H4:I5) since the Observed data is in cells C4:D5 and the Expected data is in cells H4:I5. (See image Below)
chi-square | A/B Test
image.png

Upon running this test, you will observe a p-value of 0.871

We can use the below grid to interpret the p-value.

chi-square | A/B Test
image.png

Since the p-value>0.05, we will fail to Reject the Null Hypothesis.

It means that the clicks are independent of the advertisement.

Step 5c: Calculate and interpret statistical significance to accept/reject Null hypothesis.

Statistical Significance = 1-p-value = 1 – 0.871 = 0.129.

It means that we are only 12.9% confident that these click rates will be observed for the two advertisements. Since, this significance is less than 0.95, we fail to reject null hypothesis.

Code to Implement Chi-Sq in Python

# python
from scipy.stats import chi2_contingency

# defining the table
# [a_click, a_noclick], [b_click, b_noclick]
data = [[25,75], [26,74]]

stat, p, dof, expected = chi2_contingency(data,correction=False)

# interpret p-value
alpha = 0.05
print("p value is ",(p))

if p <= alpha:
    print('Dependent (reject H0)')
else:
    print('Independent (H0 holds true)')

Make sure that you pass the correction=False argument in the function.

Output:

p value is 0.8711230866931309

Independent (H0 holds true)

How to Count of Instances or Volume Plays an Important Role in Determining Significance?

Let us increase the volume of the data as below:

chi-square | A/B Test
image.png

Now let us rerun the same using Python snippet to see how the p-value changed.

# python
from scipy.stats import chi2_contingency

# defining the table
# [a_click, a_noclick], [b_click, b_noclick]
data = [[2500,7500], [2600,7400]]

stat, p, dof, expected = chi2_contingency(data,correction=False)

# interpret p-value
alpha = 0.05
print("p value is ",(p))

if p <= alpha:
    print('Dependent (reject H0)')
else:
    print('Independent (H0 holds true)')#import csv

Output:

p value is 0.10473464597187702

Independent (H0 holds true)

You see – how the p-value reduced from ~0.87 to ~0.1. If the traffic increases further and the click rate remains same, the p-value will further decrease.

Applications of Chi Square

Given below are a few most common applications of the chi-square formula

  1. Biologists use it to determine if there is a significant association between the two variables, such as the association between two species in a community.
  2. Genetic analysts use it to interpret the numbers in various phenotypic classes.
  3. It is used in various statistical procedures to help to decide if to hold onto or reject the hypothesis.
  4. It is used in the medical literature to compare the incidence of the same characteristics in two or more groups.
  5. It is used in A/B testing to compare multiple groups.

Advantages of Using Chi-square Test

  1. Easy to understand
  2. Does not require the distribution to be normal
  3. Can be used to test multiple metrics across multiple groups. For instance, if we were tracking 3 metrics : Click_on_button, Click_on_anywhere_else, No_Click for 3 advertisements. We could have a contingency table of 3 X 3 and DOF (Degree Of Freedom) = (3-1) * (3-1) = 4

Conclusion

  1. A/B testing, also known as split testing, is a method of testing that compares two versions of a variable (for example a advertisement A vs advertisement B), showing each version to equal numbers of users at random to determine which performs better against a business goal.
  2. Traditionally, t-test (or z-test) is used to determine the significance of A/B test. t-test assumes that the distribution is Gaussian (or normal). Since, we cannot always have a normal distribution with the data, we need to use a better and more generic approach.
  3. Chi-sq test is more reliable for such cases.
  4. It is easy to calculate and interpret Chi-sq in Excel and in Python.
  5. The significance value of a Chi-sq statistic increases with increase in traffic.

Feel free to connect with me on LinkedIn if you want to discuss this with me.

Frequently Asked Questions

Q1. Why is it called chi-square?

A. The term “chi-square” is used because the test statistic follows a chi-square distribution. The distribution was first introduced by the statistician Karl Pearson, who named it after the Greek letter “χ” (chi), which resembles the shape of the distribution curve.

Q2. What is a chi-square test used for?

A. A chi-square test is used to analyze categorical data and determine if there is a significant association between two variables. It helps to assess whether the observed frequencies in different categories deviate significantly from the expected frequencies, indicating a relationship or independence between the variables.

Q3. What is the difference between t-test and chi-square?

A. The main difference between a t-test and a chi-square test lies in the types of data they analyze. A t-test is used to compare means between two groups, typically for continuous numerical data. In contrast, a chi-square test is used for categorical data to examine the association or independence between variables.

Q4. What is chi-square test and its types?

A. A chi-square test is a statistical method used to analyze categorical data. It has different types based on the research question and nature of data, including the chi-square goodness-of-fit test (assessing whether observed data fits an expected distribution) and the chi-square test of independence (evaluating the association between two categorical variables).

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

A/B testingblogathonchi square analysisdistributionExcelGuidep valuepythontesttesting
aakash93
aakash93

Data Scientist with extensive experience in solving many real world business problems across different domains. Possess fine blend of business knowledge, maths/stats and technology/programming.

Experienced in handling client facing roles, stakeholder management, effective communication with presentation & negotiation skills.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
What is a Chi-Square Test? Formula, Examples & Application
What is a Chi-Square Test? Formula, Examples & Application
Discovering Insights with Chi Square Tests: A Hands-on Approach in Python
Discovering Insights with Chi Square Tests: A Hands-on Approach in Python

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details