Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Nobody Tells You – 5 things Big Data ‘CAN’ and ‘Cannot’ Do

Tavish Srivastava
Tavish Srivastava Last Updated : 19 Apr, 2023
5 min read

Introduction

 

“Big Data makes us smarter, not wiser.” – Tim Leberecht.

The term ‘Big Data’ got introduced in 1940s. Companies around the world have put in ceaseless efforts to explore its potential. The global tech giants have massively increased their spending on leveraging big data technologies. This trend got quickly replicated among major industry players.

As a result, according to a forecast issues by research firm(IDC), big data technology and services will grow at a CAGR of 23 percent through 2019. The big data annual spending will reach $48.6 billion in 2019.

That’s how big data services are being accepted worldwide!

Big Data has given a ‘ray of hope’ to companies and enabled them to make use of data of any size and volume. The bits of data collected via our mobile phones, GPS, sensors devices is no longer useless. Every bit of data collected gets collected and processed to derive useful insights about us (customers).

Amidst, the increasing benefits of Big Data, people fail to see the things it “can’t” do. This was surprising for me too. But soon I realized, Big data always complements business intuition but can’t ever replace it.

In this article, I present you my research of last 7 days. My crazy curiosity led me here. I just couldn’t digest the fact that big data has all it takes for a company to succeed.  Big Data is ‘capable’ of many things. But ‘incapable’ too.

Note: My thoughts are not exhaustive but an attempt to put a framework. Feel free to add your perspectives in the comments section below.

 

Black Friday Data Hack – 9 Days to Go – Win Amazing Prizes

 

big data can and cannot do

‘Tiny’ Exercise on ‘Big’ Data

This exercise will prepare us for the future. We must know about things which are yet to come. Hence, If you are reading this, I invite you to take a stab on this question. You just need to write(I’ve already shared the answer though):

” 5 things which Big Data can do” & “5 things big data can never do”.

For instance, if I conclude using a logic that X is not possible using any technology platform with Big Data. I will simply eliminate all the business problems which are related to X. Getting it?

Below is my list. If you disagree with any of the element in my list, please justify! I will love to modify this list with time. Let’s start with a short note on my ideology on using business intuition and business analytics

 

The 80:20 rule

The rule says,

“80% of time is spent in creating stories from past data, and 20% of time is spent in connecting those stories with current business”

 

Explanation: I believe no analytical insights are useful until they are in sync with business intuition. Agree ? Moreover, with time, the data driven component has grown exponentially. Companies are now flooding with data. But would that really make a difference? No!

Companies must realize, a right combination of  successful business analytics over required business intuition is in ratio of 80:20.

If we can build a story using analytics which describes the past to predict future expectations 80% of time, we need to invest 20% of time thinking, how this information is useful to business. We must think of ways which can change our future and meet our broader business objectives. This requires a strong business understanding and sound knowledge of business rules.

The 20% component in this rule is non-replaceable. Hence, humans intervention is required for solving this 20% and possibly no machine can make up for it. Not even artificial intelligence. Because, humans think in a non-defined fashion which leads to creativity. Creativity is what I believe no machine can bring to the table. My list is inspired though this rule.

 

5 things Big Data ‘CAN’ do

  1. Diagnostic analysis : We do it every day. Machines are superb at this. Once an event occurs, we find interested in seeking its causes. For example, suppose, there is a sand storm in Desert A. We have all the parameters in different regions of Desert A: Temperature, Pressure, Camels , Roads, # Cars etc. If we can relate the parameters to the sand storm in that area, if we know a few causal relationship, we can possibly avoid sand storms. Imagine how powerful big data is!
  2. Predictive analysis : We do this often. Predictive analysis is in our DNA! For instance, we have a hotel chain all over the globe. Now we need to find which of these hotels will not meet their target sales. And if we know it, we can focus our efforts on these hotels. This becomes a classic problem of predictive analysis.
  3. Find relation between unknown elements/events : I love this part of the analysis. Let’s say, number of sales employees has literally no relation with sales. Then possibly you can reduce the number of sales employees if that does not do any other loss.
  4. Prescriptive analysis : This is the future of analytics. Let’s say we are trying to predict a terrorist attack in a popular destination and possible strategy to safely move people. To make this prediction, you need to make a series of prediction, which might involve predicting number of tourist then number of tourist in that location, then predicting area which can be affected by a blast etc. etc.
  5. Monitoring an event as it happens : Majority of people in the industry work on monitoring of events. For instance, you need to monitor the response of a campaign and find segments which responded most and least. These analysis become crucial for running a business.

 

5 things Big Data ‘CANNOT’ do

  1. Predict a definitive future : We can reach higher 90s in terms of accuracy using sophisticated machine learning tools. However, you never reach 100% in accuracy. If we could do that, I could have told you exactly whom to target and achieved a 100% response rate every time. But sadly, never going to happen!
  2. Imputation of new data source : Imputation takes most of the time in any analysis. And I believe this is where you bring in your creativity and business understanding. Possibly, one of the most boring piece in your analysis which you will never get rid of.
  3. Find a creative solution to a business problem : Creativity is one thing which will always be a patent of human race. No machine can ever find a creative solution to a problem. This is because even AI is coded by human and creativity is never learned through algorithms.
  4. Find solution to a not so well-defined problem : The biggest challenge in analytics is to shape an analytics problem from a business problem. If you can do this well, you are on the right track to become a analytics superstar. This piece of role is something machines can never take away from you. For instance, your business problem is to manage attrition. Now until unless you define responders, the time wondows etc. No predictive algorithm can help you.
  5. Data management/Simplify data for a new data source : With growing data, management of data is becoming difficult. We are progressing with different types of data structures for different types of data. For instance, graph data might be well suited for network analysis but is useless for campaign data. This piece of information is again which machine cannot analyze.

 

End Notes

I believe this article will reach its true potential if people start trying out the exercise in this article. Try thinking in a more holistic view where you can see what machine cannot do ever. For instance, my starting point was the 80:20 rule that machine cannot bring creativity. This starting point helped me think of what are the pieces which needs creativity in the process of analytics.

What is your list of do/don’t? Did you like this post? Write your comments in the box below.

If you like what you just read & want to continue your analytics learning, subscribe to our emails, follow us on twitter or like our facebook page.

80:20 ruleBig datacan docant dodescriptive analysispredictive analysisprescriptive analysis
Tavish Srivastava
Tavish Srivastava

Tavish Srivastava, co-founder and Chief Strategy Officer of Analytics Vidhya, is an IIT Madras graduate and a passionate data-science professional with 8+ years of diverse experience in markets including the US, India and Singapore, domains including Digital Acquisitions, Customer Servicing and Customer Management, and industry including Retail Banking, Credit Cards and Insurance. He is fascinated by the idea of artificial intelligence inspired by human intelligence and enjoys every discussion, theory or even movie related to this idea.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Tesfaye Demissie
Tesfaye Demissie

Thanks Tavish, Great article. I loved it and actually I am thinking of sharing it with my circle of network. Of course source will be acknowledged in some form. Any objection? Regards,

123

Tavish Srivastava
Tavish Srivastava

Sure. Go ahead. Request you to add your list as well.

123
Arun
Arun

Good article Tavish.

123

rajesh,pleti@gmail.com
rajesh,[email protected]

Thanks Tavish for posting good article. most of the companies are thinking in such away that they would like to change their database schema models in terms of Big data without knowing what it is. so your article may helpful to those companies. as big data analysis is always a data driven approach to improve the growth of organizations to make future decisions. Big data is not a computational approach. big data analysis is like a "trying to search for black cat in dark room which there in it" .like insights of data to find

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
13 Tips to make you awesome in Data Science / Analytics Jobs
13 Tips to make you awesome in Data Science / Analytics Jobs
Getting Started with Analytics: Data Challenges
Getting Started with Analytics: Data Challenges

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details