Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

The art of structured thinking and analyzing

Kunal Jain
Kunal Jain Last Updated : 27 Aug, 2021
5 min read

It took me 3 months to complete my first analytics project. If I would have worked on a similar project 6 months into the job, I would have completed it with in a month. Today, I can complete the same project in less than a week.

So what has changed during this time?

  • I have started thinking in structured manner
  • I have gained experience in best data management practices
  • My business thinking has evolved over time

While the last two points improve only with time, structured thinking can improve quickly through simple training and disciplined approach towards analysis.

But before we go forward, let me bring out the benefits from structured thinking through following graph:

time required project completion

Here is how to read this graph:

  • Red line in the graph shows how time to complete a project (in weeks) has come down with experience
  • With in each of three blocks (< 1 year; 1 – 3 year; 3+ years), the area of color shows the factor responsible for drop in time.
  • For example, during the first block, time required to complete the project comes down from 12+ weeks to 3 weeks and 75% of this drop is because of structured thinking.

As you can see, structured thinking is a a big differentiator between a good analyst and bad analyst. Not only this, you can not become a good analytics manager until you can put structure to complex and ambiguous problems. Hence, this post is aimed to help you progress on path of structured thinking.

[stextbox id=”section”]What is structured thinking?[/stextbox]

Structured thinking is a process of putting a framework to an unstructured problem. Having a structure not only helps an analyst understand the problem at a macro level, it also helps by identifying areas which require deeper understanding.

[stextbox id=”section”]Why is it important and how can it help?[/stextbox]

Without structure, an analyst is like a tourist with out a map. He might understand where he wants to go (or what he wants to solve), but he doesn’t know how to get there. He would not be able to judge which tools and vehicles he would need to reach the desired place.

How many times have you come across a situation when the entire work had to be re-done because a particular segment was not excluded from data? Or a segment was not included? Or just when you were about to finish the analysis, you come across a factor you did not think off before? All these are results of poor structured thinking.

Following are some results of poor structured thinking:

  • Multiple iterations of work
  • Change of scope and work half way through
  • Longer turn around time
  • Difference in expectation between analyst and stakeholders

[stextbox id=”section”]So how to enhance structured thinking?[/stextbox]

Following are some of the best practices which I have learned over time. These practices have helped me and a lot of other people immensely to structure our thoughts and making sure we stay on track during our projects:

  • Create a Scope of Work document: This document helps by bringing all stakeholders on same page at the start of any project. A typical document can be any where between half a page to maximum of 2 pages. Following are typical sections which I keep in this document:
    • Background
    • Problem statement
    • Customers, Sponsor and stakeholders
    • In scope and out of scope areas (please note that both are equally important).
    • Reference to any work which has happened in past

Once you create this document, make sure you circulate it to all stakeholders and take their thought at this stage only. If their is any confusion or contradiction to the scope, sort it out now! There is hardly any point in going further, until you define the scope clearly.

  • Create presentation and lay out the analysis without touching the data: You will be surprised to see the power of this technique. It might sound counter-intuitive to people outside analytics industry, but this technique helps you remove any biases and makes sure you think of all the aspects before you zero in on any specific area. For example, if I want to understand why charge offs have increased suddenly in credit card portfolio over last month, I would lay it down in a structure similar to this:

structureOnce this structure is exhaustive and complete, you can simply plug in the data for each set of variables and see what is the reason.

The beauty of this practice is that you can layout any problem by spending an hour on it. If you do this right, solving the problem would then be a simple walk through. If you don’t, there is a big risk that you might focus on only internal factors (that too only from a specific area).

  • Perform back of the envelope calculations: Once you have the entire structure laid out, perform quick and dirty back of the envelope calculations to see if some things should be analyzed first or can be removed. As long as they are 80% right, go ahead. For example, if charge offs have doubled with in a month, it is unlikely to be driven by unemployment (unless you are aware of huge scale layoffs in the economy). Similarly, if Credit limit increase only impacts 2% of your portfolio, the charge offs from this programme need to increase by 50x, if portfolio risk has doubled, which is unlikely unless you have given free Credit without looking at population.
  • Layout the data requirements and hypothesis before looking at what data is available:  If you layout these after looking at what is available, there is a high chance that you restrict your thinking to those variables and data points which may not bring out the best outcome.
  • Finally… practice, practice and practice: There is no point reading all this stuff, if you don’t apply it (too bad that I am saying it so late)! The more you apply it, the better you will become at it. Take any problem / business scenario, break it down on a piece of paper, see how things add up. How much profit or loss can you expect from a strategy change? What is the most critical factor? What assumptions need to hold out in order to make profits? Just apply structure to any problem you come across and see the fun. In matter of hours, you might know more than the  owner of the problem.

So, next time when you come across a problem to solve, don’t dive into the data. Put a structure around it, make sure you cover all the aspects, understand what is the most important factor, layout possible hypothesis and data requirements and then solve the problem. You will be surprised to see the reduction in time, if you follow these steps religiously.

As usual, if you know of any other practice, which has helped you think more structurally, do let me know.

If you like what you just read & want to continue your analytics learning, you can subscribe to our emails or like our facebook page

Kunal Jain
Kunal Jain

Kunal Jain is the Founder and CEO of Analytics Vidhya, one of the world's leading communities of Al professionals. With over 17 years of experience in the field, Kunal has been instrumental in shaping the global Al landscape. His expertise spans diverse markets, from developed economies like the UK to emerging ones like India, where he has successfully led and delivered complex data-driven solutions. As a recognized thought leader, Kunal has empowered countless individuals to realize their Al ambitions through his visionary approach to Al education and community building. Before founding Analytics Vidhya, Kunal earned both his undergraduate and postgraduate degrees from IIT Bombay and held key roles at Capital One and Aviva Life Insurance across multiple geographies. His passion lies at the intersection of analytics, Al, and fostering a thriving community of data science professionals.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Ayush Biyani
Ayush Biyani

Hi Kunal, One thing I have realized is that even within the realm of structured thinking, there are different structures possible.( say S1 and S2). There has to be obviously some analysis which should result in a person deciding that S1 is better/worse than S2. For example lets see somebody says what are the no of pizzas shop..should first question be "is it a city " or " does it have more youngsters "..obviously both of them are important but comes as the first filter is more often than not a victim of personal discretion. what do you think ?

123

Kunal Jain
Kunal Jain

Hi Ayush, Very valid question. The way I would look at it is that both of these structures ultimately fall under a single structure. The overall question being what factors influence location of pizza shop. There would be one set of hypothesis / variable related to geography (city, state, tier of cities), another set with economic variables of the city (average income, spend, credit, number of jobs, number of students or target market etc.), another set of variables with type of markets available (e.g. malls, residential markets) so on and so forth. Ultimately all these variables would go into a model, which would highlight which ones have higher influence and which would not. I don't see any need to judge whether S1 is better or S2. In fact a good analyst should never start create a solution starting from a biased position. Hope this answers what you had asked. Kunal

123
Aatash
Aatash

Great points, Kunal. Very well explained and written.

123

Ram Kapoor
Ram Kapoor

Hi Kunal, I am an HR Analyst, working for a financial services organization. I want to understand, how to create a data repository for a startup organization for your business function, and perform process monitoring and performance analytics. Regards, Ram

123

Kunal Jain
Kunal Jain

Hi Ram, Unfortunately, the process is relatively long, complex and one where decisions should be taken with all stakeholders. There are various parameters which impact the outcome (for example size of data, granularity of data, how long do you want to retain data). Depending on these answers, the outcome would vary. If your data is relatively small and you want to perform simple analysis and reporting, tools like Qlikview and Tableau can be good value for money. Thanks, Kunal

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details