Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Learning Path for Developers & IT Professionals to become a Data Scientist

Kunal Jain
Kunal Jain Last Updated : 26 May, 2016
9 min read

Introduction

This guide to meant to help web developers, software engineers and other IT industry people to transition into analytics / data science industry.

Last week, I was taking a guest lecture with one of the well known institutes in India. Rather (un)surprisingly, more than 60% of the students comprised of experienced IT Professionals. Most of them are facing a common problem, “I have been in IT / software / web development for more than a few years and want to up-skill myself in analytics. I have taken a few MOOCs and have tried using a few books / platforms. Still, I don’t get it what should I do next?”

This scenario is not very different from several learnups / meetups we have conducted over the last year. In order to help these people as much as I can, I’ve created this comprehensive career guide to get you quickly started with data science. Once you finish reading this post, you would know the next steps in making a transition.

webdevelopertodatascientist

 

Self-assessment: A summary of where you stand

People working in IT industry are generally comfortable with coding, working with databases and using frameworks. After spending a few years as a developer, you would know at least some languages like Java, ASP.net, Javascript, C++, C, HTML, Python, PHP, and you would have worked with several databases including SQL, MongoDb, Oracle etc. With these skills, you are envied by people trying to transition from non-programming background. Ask them, how they feel about it!

far away from goalLet me explain the set of skills (eventually advantages) which I expect every good IT professional to have:

  1. Coding Experience: If you are comfortable with programming, you can focus on understanding the concepts and solving the problems. This can be a huge benefit in the initial days. Even though you might not have experience in languages used in data science, you still have an advantage as you understand the concepts like indexes, functions, objects, referencing etc.
  2. Logical Thinking: Data Scientists are logical. They make data based decisions. So, would you! Coding requires logic. Be it the programming logic. Your brain is accustomed to think that way. Definitely a plus point for you!
  3. Numerical Ability: Mathematics forms the core of data science. To be good with numbers is the biggest achievement one could get. I’ve seen that people who are good at coding are also good with numbers.
  4. Database Knowledge: IT professionals are known to have good knowledge of database. This knowledge is immensely helpful for data scientist to quickly obtain data from database teams.

On the other hand, you can be fooled to believe that data science is about learning a few more tools. Just like in programming, knowing a few languages or frameworks would not make you a good software engineer. What differentiates a good analyst from a bad one is the problem solving and structured thinking skills. Tools are just a way to implement this thinking. Hence, I recommend people to just pick one tool depending on their convenience and then focus on getting hands on experience.

Here are the areas you need to focus on going forward:

  1. Structured Thinking: Structured thinking refers to the art of taking an ambiguous problem and then putting a structure around it. Remember those days, when you got a very loosely / generically written requirements document from a client? This is bound to happen often in analytics. Your clients and stakeholders will tell you the business problem and you will be expected to put a structure around it and solve it using data science.
  2. Math & Statistics: You should know some basics of statistics, algebra and numerical inferences to start with. You can enrich this further over time.
  3. Domain knowledge: As a data scientist, you would need to know the domain very closely. Why is a particular process being done that way? If you are in banking, you need to understand why risk management is important and what are some of the common terms used in the industry. You need to spend some time understanding the domain and current processes.
  4. Presentation skills: As a developer, you are mostly good as long as you can explain your code to fellow developers. Not all developers need to co-ordinate with the client and present their solutions to the client directly. On the other hand, a data scientist / analyst would need to present his / her findings in a manner which is easy and exciting for the business.
  5. Technical Data Science skills: Starting from ways and biases which can arise in data collection, cleaning to the predictive modeling and implementation of data science solutions – you will need to pick all of it in coming days.

 

Where should you start ?

The problem in today’s world is the problem of plenty. I am sure you couldn’t agree more. Try searching for resources on Internet for R / Python / Data Science and you end up with a long list of resources. Talk to a few people who have made the transition and they would add a few more resources, which worked for them. If you are an avid reader, you can add a few books and blogs over and above this. Check out the platforms offering MOOCs and you can see a few good courses.

The sad part is that while you have access to plenty of resources today, you find it difficult to find your way through these resources. Hence, I have created this learning path:

 

Step 1: Getting your machine ready

The Hardware:

Data Science is computationally intense (this should not be a news to you!). So, the first thing you should do is to set up a machine which helps you in your learning. Ideally, I would say that any machine used for serious data science work should have at least 8 GB RAM and an equivalent of i5 / i7 intel chips. Of course, the higher capacity you buy, the better it is! If you have some more money to spend at this stage, you can even get a SSD upgrade.

The Operating System:

Ironically, there is no single OS which works perfectly for Data Science. You would likely need a mix of Unix and Windows machine. Unix is better in resource management and for performing the data science work. On the other hand, you would need Windows for Powerpoint and Excel, both of which are used very heavily in data science work flows.

Also, there will be a few visualization tools, which work better in Windows environment. Hence, I would recommend to use Linux as the core OS with a virtual machine running Windows or vice versa. If you are used to Mac and can work comfortably on Excel on Mac, you might be good too.

The Softwares:

You would need to choose the language / tool of your preference here. If you have experience in coding with Object oriented languages, I would recommend Python. It is easy of learn and has a vibrant community on internet. If you aren’t used to object oriented programming, you can give R a try as well. If you need more details before making a decision, read this comparison – SAS vs. R vs. Python

Here is a list of softwares, I would recommends at the minimum:

  1. MS Office for Excel, building presentations and writing documents.
  2. FileZilla for transferring files using FTP
  3. Git & GitHub for version management.
  4. VMWare / Oracle Virtual Box / Vagrant for running virtual machines
  5. Cygwin / Putty (for windows)
  6. I use Evernote for taking notes. In case of Linux, you need to run it in the browser.
  7. Terminator (for Linux) to run multiple terminals in a single view (it is awesome!)
  8. Sublime Text (or any other editor of your choice) for editing codes. You should install additional plugins for the languages you use.

If you have chosen Python as your preferred tool:

  • Install Python (2.7 or 3.4) with numpy, scipy, matplotlib, scikit-learn, pandas
  • Install Jupyter notebooks
  • In addition, Dato (Graphlab), vowpal-wabbit, import.io are some interesting additional libraries

If you have chosen R as your preferred tool:

  • RStudio is a good choice of environment.
  • Install important packages like dplyr, lubridate, ggplot, caret, randomForest etc.

Your machine is ready to crunch some numbers now!

 

Step 2: Getting used to solving ambiguous problems (i.e. Developing Structured Thinking)

The art of structure thinking is a tacit requirement but profoundly being sought in every data scientist. Otherwise, below are some good resources to enhance your skill.

Assignment: Solve this case study on operational analytics: Call Center Optimization. It’s a beginner level assignment. Once you complete it successfully, you can move to medium and advanced level.

Think you are ready for the next level, try out our practice problem on Strategic thinking

 

Step 3: Basics of Math & Statistics

Mathematics plays an important role in defining data science. Thankfully, you don’t need to learn all of math, just a few topics would do. You can start from the basic topics (marked mandatory) and pick up the rest of the topics as you progress:

Assignment: Do a statistical analysis on Big Mart Practice Problem. After you have finished with this assignment, you can showcase it on your LinkedIn profile as project work.

 

Step 4: Basics of new tools (R & Python)

After step 3, you should do programming. Coding in data science is laconic in nature. Best practitioners avoid redundant lines of code and adopt ways to make it faster. Your prior knowledge of programming basics, should give you a nice head start in solving practical problems using R or Python.

Get used to the basics of R / Python from any of the following introductory courses:

Apart from these 2 tools, you can also use Julia, Go, Java to build predictive models. However, a possible drawback with other programming languages is the lack of community support. Till now, Python and R have the best community support on web. Thus, would help you to debug issues and learn faster.

Assignment: Already given the practice links above.

 

Step 5: Get your hands dirty – Your First Project

Time to get your hands on your first project. Like programming, the best way to learn data science is to do data science. Hence, let us start by taking up a problem to work on. You can choose any of our Practice problems or any of the projects mentioned here to start with. Perform an exploratory analysis on the data to get you started.

Here are a few good places to look at:

Assignment: Perform a similar analysis on the project of your choice.

 

Step 6: Follow the steps in learning path

Now that you have tasted blood, go for the kill! Check out our learning paths on R & Python – follow them step by step. Skip the steps you are comfortable with. Do as many exercises as you can!

Here are some additional machine learning resources you can look at. Remember that the best way to do data science is to learn data science thoroughly:

Assignments: Build a machine learning model on Loan Prediction Problem using the following algorithms:

  • Logistic Regression
  • Decision Tree
  • Random Forest

For Decision Tree and Random Forest, you can seek help from: Complete Guide on Tree Based Modeling.

Make sure you understand how each of these algorithms works. Just implementing them and obtaining predictions wouldn’t be a success for you. The real success lies in gaining knowledge of how they work!

 

Step 7: Start participating in the competitions

Time to step in battle ground. A benefit of being a part of analytics community is that you get to access so many thrilling ways of learning concepts. You no longer need to stick to traditional ways of learning.

Competitions: Several data science competitions get organized across the globe where you can participate and win prizes too. After you’ve completed above steps you must participate in these competitions to assess your learning level.

Assignment: In 6 months – 1 years, try to rank in top 100 in the competition rankings on both websites. This would give a massive boost to your profile.

 

Mistakes You Should Avoid While Learning

We are prone to make mistakes (unknowingly) in pursuit of learning concepts quickly. Nothing to worry, we all are susceptible to such things. But, we need to be prudent enough to analyze our learning pace and proceed accordingly. Below are the list of mistakes ( bad practices) you should avoid while completing this learning path:

  1. Choosing difficult problems: Working on difficult problem during initial days wouldn’t make you a kick ass data scientist. You’d only end up torturing yourself. Follow a natural pace. Start with simple problems as given in assignments above. It would establish your much needed confidence.
  2. Learning too many concepts: You might get overwhelmed with the availability of online resources. Don’t get swayed away by their abundance. You don’t have to be a “jack of all trades and master of none”. Become a master. Take one tool or one concept at a time. Once you become confident about it, move to next.
  3. Losing way in coding:  You might have done die hard coding in last few years. But, data science is a bit different. You are no longer writing softwares. You are using them here. Don’t waste time in writing algorithms. Use Libraries (or packages). Tools like R and Python enjoy an incredible support of powerful libraries and packages. Use them. In short, figure out ways to do more in less lines of code.
  4. Getting Impatient: This is the most important of all. I’ve met people who’ve claimed to spend hours on learning data science. Yet, they haven’t got a break in this industry. You need to stay patient. After all, good things take time. You’ve got access to amazing community support @ Discuss. Engage more. Share your queries. Let people know what you are up to. Remember, you are not alone with this, you are with us.

 

Finally, a peek into the Life of Data Scientists at Companies

Just for some motivation…

https://www.youtube.com/watch?v=7mXFkvu98vI

 

End Notes

I hope this guide will help people from IT / software development background to take up data science / machine learning as a career option. In summary, rely on your strengths, focus on developing structured thinking & problem solving, practice a lot and get your hands dirty on as many real life problems as possible. In the process, if you get stuck, leverage the communities and people in your network to help you out.

As usual, if you have any questions or suggestions I might have missed out on, feel free to reach out to us through the comments below.

You can test your skills and knowledge. Check out Live Competitions and compete with best Data Scientists from all over the world.

Kunal Jain
Kunal Jain

Kunal Jain is the Founder and CEO of Analytics Vidhya, one of the world's leading communities of Al professionals. With over 17 years of experience in the field, Kunal has been instrumental in shaping the global Al landscape. His expertise spans diverse markets, from developed economies like the UK to emerging ones like India, where he has successfully led and delivered complex data-driven solutions. As a recognized thought leader, Kunal has empowered countless individuals to realize their Al ambitions through his visionary approach to Al education and community building. Before founding Analytics Vidhya, Kunal earned both his undergraduate and postgraduate degrees from IIT Bombay and held key roles at Capital One and Aviva Life Insurance across multiple geographies. His passion lies at the intersection of analytics, Al, and fostering a thriving community of data science professionals.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Vishwachandra
Vishwachandra

Kunal, The effort that you are making to help the data ppl is very commendable.Please keep it up.

123

Kunal Jain
Kunal Jain

Thanks Vishwachandra!

123
Venkat
Venkat

Hi Kunal, Great stuff!! I am sure many aspiring data scientists will find this useful. Thank you for your help. Venkat

123

Kunal Jain
Kunal Jain

Glad that you liked this piece Venkat!

123
Akshatha
Akshatha

Thanks a lot, Kunal. This is very useful to me. I'm planning to switch my career to Analytics and I hope I will be successful soon by adopting the above mentioned steps.

123

Kunal Jain
Kunal Jain

I hope that this should help tremendously in your journey. Let us know, if you need any further help.

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details