Is survival analysis the right model for you?

I was a post-graduate in Mechanical Engineering when I joined the analytics industry as a fresher.

The only background I had in analytics industry was based on a few courses in Operations Research. I was scared to join the industry, where I knew lesser than a student of statistics in 12th standard. In less than a month, I realized that Analytics industry does not require you to have a masters in statistics or economics, but requires structured thinking with sharp mathematical reflexes. It took me no more than 3 months to build and implement my first logistic regression model. Most of the people in India still use basic analytics tools such as CART, regressions and time series. We still are scared of using complex statistical techniques, such as neural network and survival analysis. Last year, I used survival analysis in one of the analytics projects and realized the power of the tool without getting into the Limbo of statistics behind the tool. This article will help you find if the survival analysis is the right tool for your next project. The article will end with a case study, which we will solve using survival analysis in the next article.

[stextbox id=”section”]How is survival analysis different from regression models?[/stextbox]

Regression models have a single output function. In case of a logistic regression, the output is the response function which can only take two values. In any other model, we can define the output function in a single objective function. For instance, if we are building a customer attrition model, which predicts whether a customer will attrite in next 3 months, following is the objective of the logistic model :

f(x) =  0    if no attrition in next 3 months

=  1    if customer attrited in next 3 months

Say, we want to profile the customers, who are likely to attrite early and restrict the acquisition of such profile customers. Let’s assume for simplicity that there are only 2 variables : Gender and Tenure. We build a logistic model in Jan ’13 and found that out of 100 Males 30 attrite till Jan’13, whereas out of 100 females only 10 attrite till Jan’13. My model profiles females as better profile, but because of some reason we did not implement this factor into our acquisition strategy. Now, we stand in Jul’13 and if we look at the same population as considered in the logistic model,  out of 100 Males 35 attrite and out of 100 Females 55 attrite.

The results seem to have swapped. In last 6 months, females saw high attrition, whereas male population seems to be very stable over this period. Now you observe that females had a lower tenure in Jan’13 as compared to the male population. The possible solution in this case is to take same month tranches/acquisitions to build a model. You now take only customer who was acquired before Jan’12. You get a population of 50 Males and 10 Females. However, you have reduced the noise coming from new tranches, you have also reduced the population you are building a model on. How do you address this issue?

Such data for whom the results (attrition in this case) are unknown, are called censored data. We can include this data without compromising on the model accuracy in a survival analysis model. This is because the output or target variable of a survival analysis is a combination of death (attrition in this case) and time on books (tenure of customer).

[stextbox id=”section”]Applications of survival analysis[/stextbox]

There are four major applications of survival analysis into analytics:

1. Business Planning : Profiling customers who has a higher survival rate and make strategy accordingly.

2. Lifetime Value Prediction : Engage with customers according to their lifetime value

3. Active customers : Predict when the customer will be active for the next time and take interventions accordingly.

4. Campaign evaluation : Monitor effect of campaign on the survival rate of customers.

  Following are some industrial specific applications of survival analysis :

• Banking – customer lifetime and LTV

• Insurance – time to lapsing on policy

• Mortgages – time to mortgage redemption

• Mail Order Catalogue – time to next purchase

• Retail – time till food customer starts purchasing non-food

• Manufacturing – lifetime of a machine component

• Public Sector – time intervals to critical events

[stextbox id=”section”]Case study [/stextbox]

You are the head of the analytics team with an online Retail chain Mazon. You have received a limited number of offers which costs you $200/customer targeted . The offer breaks even if a customer makes a purchase of minimum $20,000 in his entire lifetime. You want to target customers who are likely to make a purchase of $20,000 as early as possible. You have their relationship card segment (Platinum cards are expensive and Gold card is cheaper) and their response rate on last campaign they were targeted with. You have been targeting customers with this offer for past 1 year. Now you want to learn from the past response data and target accordingly. You need to find which customer base should you target for this offer. You have data for a similar test campaign in the past, based on which you will have to build the strategy.

You will have to use survival analysis in this case because the dependent variable is the time to respond the campaign. This again contains censored data which are people who did not respond till date. We will solve this case study in the next article where we will lay out a step by step process doing a survival analysis to find profile of customers who respond early.

[stextbox id=”section”]End notes [/stextbox]

In the last three years, I have realized that analytics project are not always created by the business. The business itself is constrained by what value they have seen in the past coming from the analytics team. Learning new modelling techniques and learning from nextgen analytics project make us capable of seeing the value beyond what business can see. We will like to know the new techniques you have learnt in recent past and their applications.  The objective to start a discussion on survival analysis here is not restricted to only this technique.

Did you find any opportunity in your line of business where you can implement survival analysis? Did you find this article helpful? Have you worked on other cutting edge modelling techniques in the recent past? Were the results encouraging? Do let us know your thoughts in the comments below.

If you like what you just read & want to continue your analytics learning, subscribe to our emails, follow us on twitter or like our facebook page.

Tavish Srivastava

Tavish Srivastava, co-founder and Chief Strategy Officer of Analytics Vidhya, is an IIT Madras graduate and a passionate data-science professional with 8+ years of diverse experience in markets including the US, India and Singapore, domains including Digital Acquisitions, Customer Servicing and Customer Management, and industry including Retail Banking, Credit Cards and Insurance. He is fascinated by the idea of artificial intelligence inspired by human intelligence and enjoys every discussion, theory or even movie related to this idea.