Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Top 5 Must-Read Answers – What does a Data Scientist do on a Daily Basis?

Shubham singh
Shubham singh Last Updated : 14 Jun, 2020
7 min read

Overview

  • What does a data scientist do on a day-to-day basis? A popular and must-know question
  • We analyze this question from a data scientist’s perspective through the lens of 5 detailed and insightful answers from experienced data scientists

 

Introduction

I’m a curious person by nature. Whenever I come across a concept I haven’t heard of before, I can’t wait to dig in and find out how it works. This has come in quite handy in my own data science journey.

But before I landed my first break in data science, I was always curious about what data scientists actually did every day. Was I supposed to simply build models all the time? Or was the oft-quoted saying about spending 70-80% of our time cleaning data actually true?

I’m sure you have asked (or at least wondered) about this too. The role of a data scientist might be the “sexiest job of the 21st century”, but what does that entail on a day-to-day basis?

what does a data scientist do

I decided to research this. I wanted to expand my horizons and understand how data scientists look at their role in different domains (such as NLP). This helped me gain a broader understanding of our role and why we should always read different perspectives when it comes to data science.

So, here is a list of the top 5 answers to help you get a sense of what the typical routine of a data scientist is. Prepare to be surprised – building models isn’t the primary (and only) function in a data scientist’s day-to-day tasks!

I also encourage you to take part in a discussion on this question here. This will enrich your current understanding of what a data scientist does and your thoughts will foster a discussion among our community!

Note: I have taken the answers verbatim from Quora and added my thoughts right at the beginning of each answer. This will help you get a good perspective of what the answer covers without diluting the author’s thoughts. Enjoy!

 

Machine Learning is Very Process Oriented – Mike West

I like this answer because it’s crisp, to-the-point and simple. The author has even designed a flow diagram and explained his thought process in a wonderfully illustrated way. Here is his answer in full:

Machine learning is very process oriented. Therefore, I’m always somewhere in one of the pictures below:data_scientist_role

Machine learning engineers spend a ton of time in the first two pictures (or stages). The fun part is really in the third stage but it’s only a small part of what happens in the real world.

Some key things to keep in mind about data science in the real world:

  1. Almost all applied machine learning is supervised. That means we build models against structured datasets
  2. Data wrangling is a large part of what happens in the real world
  3. When you hear the word supervised, think classification and regression. Most of my models are classification problems
  4. Model building is about 20% of my work. Yep, that’s it!
  5. Many small and medium-sized companies don’t use deep learning at all. Why? Because structured data algorithms like XGBoost win every time
  6. Everything I do is programmatic
  7. Most real-world data resides in relational databases. It will be your job to craft queries to pull out the data you need
  8. Big Data is unstructured data. If you have to build your models against big data, then you’ll need to learn another set of skills
  9. The cloud is here to stay. I use BigQuery for my really large structured data. Most large models can’t be built on your laptop
  10. Computers are monolingual. They only speak numbers. When you pass data to your model, you are passing a highly structured, well cleansed numerical dataset

 

A Percentage-wise Breakdown of a Data Scientists’ Day-to-Day Role – Vinita Silaparasetty

I really like the use of visualization by Vinita. The percentage-wise description of each data science task is helpful and insightful. Vinita has also leaned on her experience to explain the step-by-step work a data scientist does. It’s a must-read answer!

Contrary to popular belief, Data Science is not all glamour. The following survey results by CrowdFlower accurately sum up a typical day for a Data Scientist:

data_scientist_role

There is a lot of backtracking involved. Sometimes you even need to be able to predict what consequences removing/adding a variable might have.

  • Collecting Datasets: Data is the lifeline of Data Science, so we spend plenty of time curating it. On rare occasions, some projects might already have plenty of data
  • Cleaning & Organizing Data: This is the most time consuming and crucial step in the entire process. It has a great impact on the final results. Usually, after this step, the once large amount of data reduces and so we may need to collect more data for effective training
  • Data Mining: It is the practice of examining large pre-existing databases in order to generate new information. Once data is organized and stored in databases, we can finally begin to derive value from it by finding patterns within the data
  • Building Training Sets & Test Sets: Once we have a decent amount of data, we need to split it into the training set and the test set. A training set is a set of data used to discover potentially predictive relationships. It contains all the information about the expected output. A test set is a set of data used to assess the strength and utility of a predictive relationship. It contains mixed variables
  • Refining Algorithms: We start with a skeletal algorithm. It is very basic and defines roughly what output is expected. After a few sessions, the accuracy, precision, etc. are recorded and the algorithm is refined to maximize its efficiency

 

Data Scientist Perspective from a Small-Sized Company – Justin Fister

This is a superb answer and one I can relate to. Note that machine learning, the most anticipated aspect of a data scientist’s job, only occupies 5% of the total time! Just like Vinita, he has also explained his tasks in terms of percentage. Here is Justin’s view:

  • NLP-related tasks (15%). It’s no surprise that PaperRater’s automated proofreading technology requires heavy use of parsers, taggers, regular expressions, and other NLP goodies as part of the core algorithms and feedback modules
  • Machine learning (5%). This tends to be the most enjoyable part. Data cleaning, feature extraction/engineering/selection, and model building
  • Reporting and analytics (10%). Running queries, reviewing analytics, and assisting with strategic decision making
  • Data management (5%). Setting up and managing database servers including MySQL, Redis, and MongoDB. Larger projects may require Hadoop or Spark
  • General software development (40%). Many data scientists’ have a background in computer science, so expect to pitch in if you have an applicable background. API integration, web-development, and wherever else I can add value. Even at an AI startup, most of the development is not going to involve AI
  • Other (25%). This includes a wide variety of tasks, including blog posts, marketing, management, technical documentation, technical support, website copy, emails, meetings, etc.

 

The “Data Scientist” is a bit of a Myth – Tim Kiely

The author, Tim Kiely, uses a Venn diagram to explain what data science is. Just take a look at this Venn diagram below – it will blow your mind. Tim additionally talks about what data scientists are supposed to be by taking a somewhat contradictory view of the general definition. Here is Tim’s answer:

The “Data Scientist” is a bit of a myth, in my opinion. Not to say they aren’t out there but they are far rarer than is popularly understood and are more of the exception than the rule.

I liken it to the “Web Master” title of the dot-com bubble – these supposed people who could do full stack programming, front end development, marketing, everything. All of those roles/skills were always specialized and remain so today.

“Data Scientists” are supposed to be database architects, understand distributed computing, have a deep understanding of statistics AND some area of business or field expertise. That’s asking a lot when any one of those skill sets can take a career to build.

data_scientist_role


The Data Scientists I’ve worked with typically have a Ph.D. in A.I. or Machine learning and are effective communicators, which gives them the ability to direct the analysts, DevOps people, programmers and DBA’s at their disposal to solve problems with data-driven solutions. They outline the desired solution and leave it to their teams to fill in the gaps.

 

Machine Learning Engineer Working on NLP Tasks – Evan Pete Walsh

Let’s drill down into a particular specialization of machine learning. One of my favorites – Natural Language Processing (NLP)! I wanted to bring out a machine learning engineer’s view here (a role every data scientist should become familiar with). Check out Evan’s full response:

 

Currently working on NLP, for the most part, including intent classification and entity extraction. Here’s a typical day for me:

  • Get to work, pull up GitHub and check on the ZenHub board (kind of like Jira, except way cooler). I had some models that were training last night on our servers and I should have gotten an email that they finished. I did!
  • I’ll probably spend a few minutes testing those new models and then tweak some parameters, then restart the training process
  • The rest of the day I’m usually head-down coding, either working on a back-end Python application that will supply the AI for one of our products, or implementing a new algorithm that I want to try out
  • For example, recently I read a paper on coupled simulated annealing (CSA), and I wanted to try it out on tuning the parameters for XGBoost as an alternative to a grid search. CSA is a generalized form of simulated annealing (SA), which is an algorithm for optimizing a function that doesn’t use any information on the derivative of the function
  • Unfortunately, I couldn’t find an implementation in Python, so I decided to write my own. Two days later, I had submitted my first package to PyPI!

 

End Notes

The data scientist role is truly multi-faceted, isn’t it? A LOT of aspiring data scientists assume that they will primarily be building models all day long but that simply isn’t the case.

There are all sorts of tasks involved in a typical data science project which you’ll find yourself working on day-to-day. I quite like that because it opens up avenues to learn new concepts and apply them in the real world.

I’ll be posting some more career-related articles on Analytics Vidhya, so stay tuned and keep learning!

Shubham singh
Shubham singh

A Data Science Enthusiast who loves reading & writing about Data Science and its applications. He has done many projects in this field and his recent work include concepts like Web Scraping, NLP etc. He is a Data Science Content Strategist Intern at Analytics Vidhya. And currently pursuing BTech in Computer Science from DIT University, Dehradun.

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Rutvij Bhutaiya
Rutvij Bhutaiya

Hi! Shubham, nice article, on collective views from experienced persons in the industry. It's true most of the Data Science related tasks involves Data Cleaning. Here are my views on the Data Cleaning part. After completion of data collection, I store it in excel file. I love working on MS Excel, so here what I do, I clean 50%-60% data through MS Excel tool and then load the file on R platform - now, on R Studio I again start with data cleaning and mainly on data normalization. Then I do EDA and chart analysis, If I see there are outliers [depends on the project objective] and all, Then I again check on data normalization task. Then all the following tasks like modeling and prediction .. Hope this help!

123

Shubham Singh
Shubham Singh

Thank you so much for sharing your views. This would surely help the community.

123
Jyoti Kulkarni
Jyoti Kulkarni

Hi Rutvij, is that all a Data Scientist does? Data cleansing, outlier removal, and then data normalization? Then what is the difference between a data analyst and a data scientist? Being a data scientist, why one would end up doing the data cleansing activities? So, in case you work on a test data and implement the model on the rest of the data, what's the guarantee that the effort you have put would work correctly?

123
Brij Bhushan
Brij Bhushan

I read your blog and I like very much the roles and responsibilities of the data scientist part. There is no doubt that data science is the top trending career option and especially for those who have python skills. Thanks for sharing best career guidance.

123

Anitha
Anitha

I like your post. I appreciate your blogs because they are really good. Please go to this website for the Data Science Course In Mumbai. These courses are wonderful for professionals.

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details