Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Winning Solutions & Codes from AV’s Signature Hackathon – Lord of the Machines

Pranav Dar
Pranav Dar Last Updated : 13 May, 2019
7 min read

Introduction

Lord of the Machines, Analytics Vidhya’s recently concluded signature hackathon, was one of the most intriguing and challenging competitions  we have hosted. It featured a real-world dataset and some really awesome innovative solutions from data scientists around the world.

Initially planned for a duration of 2 days, the hackathon was extended to span 9 days in total to give participants more time to fine tune and improve their models. An incredible 3500+ participants registered for the signature hackathon!

In this article, the top three winners have shared their approach through which they climbed to the top of the leaderboard. We have also provided the GitHub code link for each approach. All the three winning solutions were executed in python.

For all those who were not able to participate, you missed out on a cracking competition! Nevertheless, there’s always a next time. Stay tuned for upcoming hackathons.

 

Table of Contents

  1. About the Competition
  2. The Problem Statement
  3. Winners!
  4. Winner’s Solutions
  5. Learnings from the Hackathon

 

About the Competition

The journey to become a data scientist is often a long, difficult and obstacle-filled path. There are problems to solve, models to be built and conclusions to be drawn.

Analytics Vidhya hosted “Lord of the Machines”, a data science / machine learning hackathon designed to discover the best data scientists in the community. In this hackathon, you – the participants – were given the opportunity to come up with innovative and exciting data science solutions to claim your supremacy.

 

The Problem Statement

Email Marketing is still the most successful marketing channel and the essential element of any digital marketing strategy. Marketers spend a lot of time in writing that perfect email, laboring over each word, catchy layouts on multiple devices to get them best in-industry open rates & click rates.

How can I build my campaign to increase the click-through rates of email? – a question that is often heard when marketers are creating their email marketing plans. Can we optimize our email marketing campaigns with Data Science? It’s time to unlock marketing potential and build some exceptional data-science products for email marketing.

Analytics Vidhya sends out marketing emailers for various events such as conferences, hackathons, etc. For this hackathon, we provided a sample of user-email interaction data from July 2017 to December 2017. Participants were required to predict the click probability of links inside a mailer for email campaigns from January 2018 to March 2018.

The evaluation metric used was AUC ROC.

 

Winners!

And the wait is over! Below are the final top 3 winners of the Lord of the Machines:

Rank 1: Kunal Chakraborty

Rank 2: SRK and Mark

Rank 3: Aditya and Akash

Here are the final rankings of all the participants in the Lord of the Machines hackathon.

The top 3 winners have shared their approach with us. We have listed them below in their own words, along with the code, for your perusal.

 

Winner’s Solutions

Rank 3: Aditya and Akash

Aditya and Akash worked on different models and then teamed up. Below are both their approaches.

Aditya’s Approach

Multiple classification model has been create to predict the click probability of links inside mailer for email campaign. Following derived features has been created, for training different model

  1. Different users have been clustered (using k-means clustering algorithm) based on email of different campaign received by them and each user is assigned a cluster_id.
  2. Count of email received by users
  3. Count of emails sent on a given day
  4. Count of emails pertaining to communication_type
  5. Day of week
  6. Target encoding of user_id based on ‘is_click’.
  7. Target encoding of user_id based on ‘is_open’.
  8. Target encoding of Communication Type on ‘is_click’.
  9. Target encoding of Communication Type on ‘is_open’.
  10. Target encoding of cluster id on ‘is_click’.

Cross Validation

Campaign_id has been used to split data into train and validation.

  1. For xgboost, 2 fold had been used and test predictions of 2 models has been averaged. With cross validation and public leaderboard score, number of trees has been fixed to 70
  2. For lgbm, 5 fold has been used and test predictions of 5/(Top 4) models has been rank averaged. Mainly 3 different type of model has been created, 1 with 45 trees, 1 with 51 trees and 1 with 55 trees

Hyper Parameter Tuning

Manual tuning has been performed based on public leaderboard and cross validation score.

Akash’s Approach

Modelling

I posed it as a problem of sequence prediction where we want to find whether a user will click on an email, given his past interactions on platform. The first thing that comes to mind when we think of sequence prediction problems is RNN or more specifically LSTM.

Features

I formed sequences of users’ actions (in form of clicked and opened). 4 sequences were formed:

  1. Clicked (0s and 1s)
  2. Opened (0s and 1s)
  3. No of sections in each email
  4. No of images in each email

These sequences acted as 4 features for sequential input.

Network Architecture(s)

  1. I started with LSTM model followed by a couple of dense layers and tuned its optimizers and no of cells in the LSTM layer.
  2. I added a CNN model followed by a GlobalMaxPool followed by a couple of dense layers and tuned its architecture.
  3. Finally I added a CNN layer followed by a LSTM layer followed by couple of dense layers as the third model.
  4. Prediction and Cold Start

The output of these models gave me a probability whether these users will click the next email or not. I used this probability across all emails sent to that user (I did not want to add prediction to sequence and predict again because that can cause errors to propagate further).

This allowed me to make predictions for the users for whom we have some data (previous behavior), but in the test set, 20% of the entries were for users for whom we do not have any data (aka cold start).

To deal with Cold Start, I grouped by campaign_id and sent_weekday and sent_quarter_of_day and filled the missing values by 90% quantile across each group.

Ensemble Model

All the model prediction has been rank averaged to reach the final submission.

Link to Code.

 

Rank 2: SRK and Mark

Most of our time was spent on creating new features. We did validation split based on campaign ids. Our best single model is a light GBM that scored 0.7051 in the leaderboard. The list of important features we used are:

  1. Target encoding on the user ID, user ID – communication type
  2. Min, max, mean and standard deviation of the mail sent time.
  3. One hot encoding of the campaigns.
  4. Time between current mail and previous mail
  5. Number of campaigns inbetween current mail and previous mail
  6. Total number of mail campaigns per user ID
  7. Cumulative count of the mail at user level
  8. Hour of the mail

Link to Code.

 

Rank 1: Kunal Chakraborty

I created several features based on textual information and user behavior to arrive at my final solution. The features created were:

  1. Target encoding of user_id with respect to is_open and is_click
  2. Target encoding of campaign_id with respect to is_open and is_click
  3. Target encoding of communication_type with respect to is_open and is_click
  4. Length of email body (word wise)
  5. Length of subject
  6. Key feature : I pre-processed the text in the subject by removing stop words, lemmatizing them, removing punctuations etc. After that I used a bag of words (unigram) representation of different campaign_ids based on their subject. This was followed by merging this dataset with campaign ids present in the train and test data. After this merge operation, I used groupby sum based on user_id to obtain a unique representation for every user. This was followed by PCA to reduce the dimensions to 50. This operation added the biggest jump to my score.
  7. Number of mails received by different users
  8. Cross tab of user_id vs communication type
  9. Numerical features present in the campaign_data

This became my general frame work for data preparation before feeding it into any model. An xgboost model with these set of features gave me a score of 0.695+ on the public leaderboard. What followed after this was sheer pragmatism. I created several models based on approximately the same framework and differentiated them by adding variability. Some of the important variations were:

  1. Using bi-grams for BOW representation
  2. Using tri-grams for BOW representation
  3. Using all three of them
  4. Using tf-idf with same (unigram,bi-gram,tri-gram)
  5. Using lightboost, xgboost and catboost on each of the three representations above
  6. Using truncated SVD instead of PCA for dimension reduction
  7. I even dropped the best performing feature and tuned the hyper-parameters in such a way to arrive at similar scores using remaining features
  8. Target encoding of weekday of sent mail
  9. Cosine distance among the Glove vector representations of different campaign ids

These are just some of the features. I created many notebooks and added/dropped/modified many features and performed many experiments which most of the time gave me a public leaderboard score in the vicinity of (0.685 – 0.69). Even though the performance of all the models was similar, their predictions were not highly correlated. This gave me the opportunity to take advantage of weighted ensembles to arrive at a higher score.

I took the most similar scoring prediction files with the least correlation and took their weighted average. I continued this process in an uphill fashion. This led to my four best performing predictions with scores (0.699 – 0.7011). I again followed the same heuristic to arrive at my final score which gave me a public leader board score of 0.704. This entire process is very similar to model stacking where diverse base classifiers prediction is fed to a meta classifier to arrive at better predictions. Only in my case, it was me manually adjusting the weights assigned to different models by validating them against the public leader board.

Link to Code.

 

Key learnings from the Competition

Below are the key learnings from this competition:

  • Before the start of the contest, it was believed that text features extracted from the subject would play a significant role in predicting the open rate and thereby the click rate. Only 52 campaigns however did not have enough signal for the model to learn anything significant from these text features
  • Akash’s approach was completely out of the box. He treated predicting click for existing users as a sequence prediction problem and used an LSTM+CNN neural network architecture to predict the next click for the user. This shows that sometimes looking at the problem with a different perspective can land you among the winners
  • A unanimous feedback from the winners was that with more features related to the user, the model would certainly have performed better in predicting the click, as well as open probability, for each email sent
  • Carefully created encoded features, particularly the ‘key feature’ used by Kunal Chakraborty, were very important in improving the score beyond a certain threshold and distinguished the winner from the other submissions

 

End Notes

This was one of the most interesting and challenging competitions we have hosted so far on Analytics Vidhya. It saw great participation, some really good solutions. I highly recommend going through the approaches and code links mentioned above to gain a deeper understanding of how these competition winners structure their thinking.

For those who missed out this time, don’t worry! We will regularly host hackathons so be sure to head over to our DataHack platform and get cracking on the practice problems we have for you on various domains.

 

Participate in our Hackathons and compete with the best Data Scientists from all over the world.

 

Pranav Dar
Pranav Dar

Senior Editor at Analytics Vidhya.Data visualization practitioner who loves reading and delving deeper into the data science and machine learning arts. Always looking for new ways to improve processes using ML and AI.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Rajesh Kumar Pandey
Rajesh Kumar Pandey

Thanks for sharing. This hackathon was a great learning.

123

Bhavik Patel
Bhavik Patel

Thank you so much for sharing..All of the approches are really great and very intuitive for learning..Thanks again!!

123

Braj
Braj

Thanks for sharing. This is very helpful. Where can we find the data-set (train and test csv files)?

123

Pranav Dar
Pranav Dar

Hi Braj, Glad you found this helpful! The contest is closed for now so the dataset is currently not available. We might float this as a practice problem in the future on our DataHack platform. Meanwhile you can take a look at the other practice problems on that platform. :)

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details