Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

How to Launch First Amazon Elastic MapReduce (EMR)?

P
Pavan Kumar Last Updated : 12 Jan, 2023
5 min read

Introduction

Amazon Elastic MapReduce (EMR) is a fully managed service that makes it easy to process large amounts of data using the popular open-source framework Apache Hadoop. EMR enables you to run petabyte-scale data warehouses and analytics workloads using the Apache Spark, Presto, and Hadoop ecosystems.

AWS EMR

Amazon Elastic MapReduce (EMR) is designed to be flexible and easy to use. It lets you quickly set up and scale a big data environment without worrying about infrastructure and maintenance. EMR can be used to process data stored in Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon Redshift, as well as on your on-premises data sources.

EMR provides many powerful tools and features to help you process and analyze your data, including running custom scripts, integrating with other AWS services, and setting up automatic scaling. With EMR, you can efficiently perform many big data tasks, such as data transformation, machine learning, real-time processing, and more.

Advantages of Using Amazon Elastic MapReduce (EMR)

AWS EMR Benefits
There are several reasons why you might choose to use Amazon Elastic MapReduce (EMR) for big data processing:

  • Fully managed service: EMR is a fully managed service that takes care of your underlying infrastructure and maintenance. This means you can focus on processing and analyzing your data rather than worrying about setting up and maintaining a big data environment.
  • Scalability: EMR makes it easy to scale your big data processing needs up or down as needed. You can easily add or remove instances from your cluster to meet changing demands.
  • Integration with other AWS services: EMR integrates seamlessly with other AWS services, such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon Redshift. This makes it easy to process and analyze data stored in these services.
  • Wide range of tools and frameworks: EMR provides tools and frameworks for big data processing and analysis, including Apache Spark, Presto, and Hadoop. This gives you much flexibility and choice when processing and analyzing your data.
  • Customization options: EMR allows you to customize your cluster by installing additional applications or libraries or running custom scripts. This allows you to tailor your big data environment to your specific needs.

Stepwise Process to Launch Amazon Elastic MapReduce (EMR) on AWS

To launch an Amazon Elastic MapReduce (EMR) cluster, you will need to follow these steps:

Step 1. Sign in to the AWS Management Console and navigate the EMR service page.

Step 2. Click the “Create cluster” button to create a new EMR cluster.

Amazon Elastic MapReduce (EMR)

Step 3. On the “Select Configuration” page, choose the software and instance types you want to use for your cluster. You can also specify the number of instances and the instance sizes.

Amazon Elastic MapReduce (EMR)

Also, if you want to attach your cluster to jupyter notebook, you need to check the “JupyterEnterpriseGateway” (See the snapshot below)

Amazon Elastic MapReduce (EMR)

Step 4. On the “Hardware Configuration” page, choose the type of hardware you want to use for your cluster. You can choose between on-demand instances or spot instances.

Amazon Elastic MapReduce (EMR)

Amazon Elastic MapReduce (EMR)

 

 

Step 5. On the “General Cluster Settings” page, specify your cluster’s name and logging options. You can specify any additional applications or libraries you want to install on your cluster.

Amazon Elastic MapReduce (EMR)

Step 6. On the “Security and Access” page, specify the security settings for your cluster. You can use an existing security group or create a new one. Also, specify the EC2 key pair to be used. After completing this step, click on create a cluster.

Amazon Elastic MapReduce (EMR)

 

After your cluster is launched, you can access it through the EMR console or using the AWS CLI or SDKs. You can then use the tools and frameworks provided by EMR to process and analyze your data.

Executing a Pyspark Script on Amazon Elastic MapReduce (EMR)

Now we will see how to execute a sample pyspark script on EMR. Here is our sample pyspark script:

Executing a pyspark script on EMR

Firstly, you need to connect to the master node using SSH. Open a terminal where your EC2 key pair is located. Now click on “Connect to the master node using SSH

Executing a pyspark script on EMR

Now copy the command and paste it into the terminal.

Executing a pyspark script on EMR

If the master node is successfully connected, you will see something like this:

Executing a pyspark script on EMR

Now to upload the pyspark script that you have in your local machine to EMR, open another terminal and run the following command:

scp -i ./my_ec2_key_pair.pem samplePysparkScript.py hadoop@<master_public_dns>:~/

Executing a pyspark script on EMR

Now your file has been uploaded on EMR; you can confirm this by running this command in the terminal:

Executing a pyspark script on EMR

Now to run the script on EMR, simply run "spark-submit samplePysparkScript.py"

Executing a pyspark script on EMR

Here is the output:

Executing a pyspark script on EMR

 

When you finish your cluster, you can terminate it to stop incurring charges. If you want to use this cluster again, you can select the cluster from the “Clusters” list and click on “Clone” with or without modifying the previously chosen settings.

Conclusion

To summarize everything that has been stated so far, Amazon Elastic MapReduce (EMR) is a powerful and easy-to-use big data processing service that can help you quickly and efficiently process and analyze large amounts of data in the cloud. With its wide range of tools and frameworks, scalability, and integration with other AWS services, EMR is an excellent choice for businesses of all sizes that need to process and analyze large amounts of data.

P
Pavan Kumar

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details