The art of structured thinking and analyzing

It took me 3 months to complete my first analytics project. If I would have worked on a similar project 6 months into the job, I would have completed it with in a month. Today, I can complete the same project in less than a week.

So what has changed during this time?

• I have started thinking in structured manner
• I have gained experience in best data management practices
• My business thinking has evolved over time

While the last two points improve only with time, structured thinking can improve quickly through simple training and disciplined approach towards analysis.

But before we go forward, let me bring out the benefits from structured thinking through following graph:

Here is how to read this graph:

• Red line in the graph shows how time to complete a project (in weeks) has come down with experience
• With in each of three blocks (< 1 year; 1 – 3 year; 3+ years), the area of color shows the factor responsible for drop in time.
• For example, during the first block, time required to complete the project comes down from 12+ weeks to 3 weeks and 75% of this drop is because of structured thinking.

As you can see, structured thinking is a a big differentiator between a good analyst and bad analyst. Not only this, you can not become a good analytics manager until you can put structure to complex and ambiguous problems. Hence, this post is aimed to help you progress on path of structured thinking.

[stextbox id=”section”]What is structured thinking?[/stextbox]

Structured thinking is a process of putting a framework to an unstructured problem. Having a structure not only helps an analyst understand the problem at a macro level, it also helps by identifying areas which require deeper understanding.

[stextbox id=”section”]Why is it important and how can it help?[/stextbox]

Without structure, an analyst is like a tourist with out a map. He might understand where he wants to go (or what he wants to solve), but he doesn’t know how to get there. He would not be able to judge which tools and vehicles he would need to reach the desired place.

How many times have you come across a situation when the entire work had to be re-done because a particular segment was not excluded from data? Or a segment was not included? Or just when you were about to finish the analysis, you come across a factor you did not think off before? All these are results of poor structured thinking.

Following are some results of poor structured thinking:

• Multiple iterations of work
• Change of scope and work half way through
• Longer turn around time
• Difference in expectation between analyst and stakeholders

[stextbox id=”section”]So how to enhance structured thinking?[/stextbox]

Following are some of the best practices which I have learned over time. These practices have helped me and a lot of other people immensely to structure our thoughts and making sure we stay on track during our projects:

• Create a Scope of Work document: This document helps by bringing all stakeholders on same page at the start of any project. A typical document can be any where between half a page to maximum of 2 pages. Following are typical sections which I keep in this document:
  • Background
  • Problem statement
  • Customers, Sponsor and stakeholders
  • In scope and out of scope areas (please note that both are equally important).
  • Reference to any work which has happened in past

Once you create this document, make sure you circulate it to all stakeholders and take their thought at this stage only. If their is any confusion or contradiction to the scope, sort it out now! There is hardly any point in going further, until you define the scope clearly.

• Create presentation and lay out the analysis without touching the data: You will be surprised to see the power of this technique. It might sound counter-intuitive to people outside analytics industry, but this technique helps you remove any biases and makes sure you think of all the aspects before you zero in on any specific area. For example, if I want to understand why charge offs have increased suddenly in credit card portfolio over last month, I would lay it down in a structure similar to this:

Once this structure is exhaustive and complete, you can simply plug in the data for each set of variables and see what is the reason.

The beauty of this practice is that you can layout any problem by spending an hour on it. If you do this right, solving the problem would then be a simple walk through. If you don’t, there is a big risk that you might focus on only internal factors (that too only from a specific area).

• Perform back of the envelope calculations: Once you have the entire structure laid out, perform quick and dirty back of the envelope calculations to see if some things should be analyzed first or can be removed. As long as they are 80% right, go ahead. For example, if charge offs have doubled with in a month, it is unlikely to be driven by unemployment (unless you are aware of huge scale layoffs in the economy). Similarly, if Credit limit increase only impacts 2% of your portfolio, the charge offs from this programme need to increase by 50x, if portfolio risk has doubled, which is unlikely unless you have given free Credit without looking at population.
• Layout the data requirements and hypothesis before looking at what data is available:  If you layout these after looking at what is available, there is a high chance that you restrict your thinking to those variables and data points which may not bring out the best outcome.
• Finally… practice, practice and practice: There is no point reading all this stuff, if you don’t apply it (too bad that I am saying it so late)! The more you apply it, the better you will become at it. Take any problem / business scenario, break it down on a piece of paper, see how things add up. How much profit or loss can you expect from a strategy change? What is the most critical factor? What assumptions need to hold out in order to make profits? Just apply structure to any problem you come across and see the fun. In matter of hours, you might know more than the  owner of the problem.

So, next time when you come across a problem to solve, don’t dive into the data. Put a structure around it, make sure you cover all the aspects, understand what is the most important factor, layout possible hypothesis and data requirements and then solve the problem. You will be surprised to see the reduction in time, if you follow these steps religiously.

As usual, if you know of any other practice, which has helped you think more structurally, do let me know.

If you like what you just read & want to continue your analytics learning, you can subscribe to our emails or like our facebook page

Kunal Jain

Kunal Jain is the Founder and CEO of Analytics Vidhya, one of the world's leading communities of Al professionals. With over 17 years of experience in the field, Kunal has been instrumental in shaping the global Al landscape. His expertise spans diverse markets, from developed economies like the UK to emerging ones like India, where he has successfully led and delivered complex data-driven solutions. As a recognized thought leader, Kunal has empowered countless individuals to realize their Al ambitions through his visionary approach to Al education and community building. Before founding Analytics Vidhya, Kunal earned both his undergraduate and postgraduate degrees from IIT Bombay and held key roles at Capital One and Aviva Life Insurance across multiple geographies. His passion lies at the intersection of analytics, Al, and fostering a thriving community of data science professionals.