Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Limitations of Pre vs. Post analysis and Importance of testing

Kunal Jain
Kunal Jain Last Updated : 27 Aug, 2021
5 min read

During a recent interview with an analyst working for a big multi-national retail store chain, I asked:

How do you check if promotion of a particular product had cannibalization effect on future Sales of the product or on other similar products?

The response I got was

By simply comparing Pre promotion Sales to Post promotion Sales.

This answer might fetch a good score quite a few Organizations and you might be surprised to see the percentage of analysts who give this answer. However, in any analytically evolved Organization, it will be considered incomplete, if not wrong!

In case, you think that this answer is right, think again!

This simplistic analysis works in steady state when there is no impact due to changes is external factors or factors you can not control for. However, this is hardly the case in any practical application. In today’s world, external factors can change overnight and the cost of making wrong conclusions can be hefty.

In this particular example, imagine what would happen if one of your competitor launches a better promotion for same product which causes your Sales to shrink by 50% in case of no promotion and 25% in case you are also running promotion. If you just do a simple pre vs. post comparison, you may end up concluding that the promotion caused your Sales to drop by 25%, while the reality is that it actually increased it by 50% (25% absolute increase on a base of 50%).

testing

[stextbox id=”section”]Challenges with simple Pre vs. Post analysis?[/stextbox]

There are bunch of practical challenges you face in doing a simple Pre vs. Post comparison. How do you account for regulatory changes happening year on year (making it difficult to compare against base of previous years)? Or for Seasonal products (e.g. Ice-cream), how do adjust for sudden heat waves or temperature drops? What do you do when the entire Industry is expanding or shrinking?

In each of these scenarios, a Pre vs. Post analysis might lead you to a wrong conclusion. So, how do you arrive at effectiveness of Sales promotion?

A simple answer to this is performing tests or experiments in order to size the impact more accurately.

While testing is a subject which deserves multiple posts, I plan to cover simplest form of testing in this post. The simplest form of testing is popularly known as A / B testing or Champion / Challenger testing. This is your solution to the kinds of problems mentioned above in this post.

[stextbox id=”section”]So, what is A / B testing?[/stextbox]

Conceptually, it is a simple solution, where you change only the parameter / factor you want to test and run two alternates in parallel to find out which one works better. For example, to answer my initial question, we will find 2 sets of stores similar to each other, run promotion in one set and don’t run promotion in other set of stores. Once we run the test for required duration (to achieve statistical significance), we compare Pre vs. Post for both the sets of stores and find out if promotion has actually benefited the store or not?

The reading you get now will adjust itself for all external changes as long as the 2 sets of stores are similar. Isn’t it elegant?

AB-Testing

[stextbox id=”section”]I understand A / B testing, where else can I use it?[/stextbox]

Applications of testing are limited by your thinking. It can help you in situations where coming out with a neat analytical solution is difficult or impossible. Following are just a few such examples:

  • When you are in a situation where there is no right or wrong answer. How do you decide whether application form should be one page or multiple page? Should you suggest a particular credit card to your customers or let them select it themselves? Should you call a customer with 30 day default once a day or twice a day? All these can be run as tests to find out what works and what does not.
  • When you want to measure impact of making a change. How does your application conversion change with change in layout of website? How does a new credit card collection strategy change the collection rate or default rate?

[stextbox id=”section”]It sounds simpler than it actually is[/stextbox]

In case you have started thinking that this sounds easy, let me warn you. It sounds easier than it actually is. Following are some precautions you need to keep in mind while doing these tests / experiments:

  • The two scenarios you test should be different only on the parameter you want to test. For example, if you want to test Subject line of a communication to your customers, you need to make sure that only the subject line changes. Creative, content, delivery address, delivery timing across the two sets of population should remain same. If any of these change, then it might become difficult to understand which factor is responsible for what change. If you want to test multiple factors, i’ll discuss this in one of the future posts.
  • The two sets of population should be chosen randomly. There should be no bias in the samples chosen. Normally it is a good idea to check distributions on 2 – 3 key parameters before rolling out the tests. If the test is to check whether making a particular offer to customers increases their chances of charge-offs, it makes sense to check whether the distribution of credit scores and vintage of the customers in two sets is similar. Again, if you don’t check for these distributions, you might end up drawing wrong conclusions.
  • Make sure you have entire control on how are the two sets being treated. This might become a huge challenge especially if you are dealing with a cross-functional team. In one of the tests designed by a friend, he wanted to size the impact of a tele-calling intervention. So we designed a no-call test where a small group of customers were not called when their collection was due. The business unit responsible for collections took a decision to send additional communication to these customers with an intention to maximize collection without informing the analysts. Result – the test ended up under-estimating the impact of tele-calling and might have led us to believe that tele-calling has limited impact which was not the case!
  • Monitor / Follow the test intensely for first few days. A lot of Organization may not have capability / systems to run multiple options in parallel. These arrangements are typically made outside the system or require a lot of change. Once the test goes live, you need to make sure it is actually what you intended and not otherwise. A healthy practice here is to include people with in project group in various sets and see if things are as per expectation.

[stextbox id=”section”]What next?[/stextbox]

 Hopefully this post gives you a good overview of limitations of simple pre vs. post analysis and advantages of performing tests. Some time in future, I’ll cover multi-variate tests and Design of experiments. I’ll also share the best practices in designing a test.

For now, do let me know if you think you can apply this elegant concept in your respective areas and how can you benefit from it.

Image 2 credit: Maxymiser.com

If you like what you just read & want to continue your analytics learning, subscribe to our emails or like our facebook page

A/B testingBusiness AnalyticsBusiness Discoveryexperimentstesting
Kunal Jain
Kunal Jain

Kunal Jain is the Founder and CEO of Analytics Vidhya, one of the world's leading communities of Al professionals. With over 17 years of experience in the field, Kunal has been instrumental in shaping the global Al landscape. His expertise spans diverse markets, from developed economies like the UK to emerging ones like India, where he has successfully led and delivered complex data-driven solutions. As a recognized thought leader, Kunal has empowered countless individuals to realize their Al ambitions through his visionary approach to Al education and community building. Before founding Analytics Vidhya, Kunal earned both his undergraduate and postgraduate degrees from IIT Bombay and held key roles at Capital One and Aviva Life Insurance across multiple geographies. His passion lies at the intersection of analytics, Al, and fostering a thriving community of data science professionals.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

AP
AP

Hi Kunal, Thanks for writing such a gr8 article.My doubt is that when I launch a mega sale which I want to expose it to every customer (new+repeat) ,then in this case do I have the choice of using any analysis technique for checking the efffectiveness of sale event other than Pre Post Analysis. Regards, AP

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
Questions to ask while designing A/B (or multi-variate) tests
Questions to ask while designing A/B (or multi-variate) tests
Beginner’s guide to Design of Experiments (with case study on banner advertisement)
Beginner’s guide to Design of Experiments (with case study on banner advertisement)

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details