Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

A Step by Step Guide to Create a CI/CD Pipeline with AWS Services

Deepak
Deepak Last Updated : 15 Jul, 2021
8 min read
This article was published as a part of the Data Science Blogathon

Overview

In this article, we will learn about how to create a CI/CD Pipeline using AWS Services: AWS CodeCommit, AWS CodeBuild, AWS Pipeline, AWS ECS & Fargate.

Prerequisites

  • Basic AWS Cloud Knowledge

Refer to these blogs, as we will use the same application and workflow

Introduction

Once we have developed and deployed the application/product, It needs to be continuously updated based on user feedback or the addition of new features. This process should be automated, as without automation we have to run the same development and deployment steps/commands again and again for every change to the application.

With Continuous Integration and Continuous Delivery Pipeline, we can automate the complete workflow from building, testing, packaging, and deploying, which will be triggered when there are any changes to an existing application or we can say if there is any new commit to an existing code repository.

Implementing QA in a CI/CD Pipeline AWS | Parasoft

Source: Parasoft

Advantages of a CI/CD Pipeline

  • Frequent releases: With CI/CD we can release the changes based on customer feedback or monitoring input
  • Low risk: As the process will be automated, there won’t be any manual intervention and configuration setup
  • Increased Productivity: With a structured process, a product will be released independently of other components, as in the case of multiple microservices we can release changes independently thus increasing developer productivity

Now, let’s start building the pipeline.

Application Load Balancer

First, we will create an Application load balancer, which will make our application scalable and highly available.

  • In AWS console, search for EC2 and then click on Load balancers
  • Choose Application Load Balancer and click Create
  • Next, Configure it by giving a name and select the VPC and availability zones
Application Load balancer | CI/CD pipeline AWS

Source: Author

  • Click Next, select Create a new security group and then click Next
  • Give a name to Target group, for Target type select IP and then click Next
create new security group

Source: Author

  • Click Next, Review it and click Create
  • Once created note down the DNS name, which is the public address for the service

AWS Fargate Service

In the previous blog, we have learned how to create a Task definition in ECS which is needed to run Fargate jobs. We will use the same task definition to create a Fargate Service. Please refer to the previous blog to understand the concepts and to follow along.

  • Go to Task Definitions in Amazon ECS, tick the radio button corresponding to the existing Task definition and click Actions and Create Service.
aws farget | CI/CD pipeline AWS

Source: Author

  • Choose Fargate as launch type, give it a name, do not change the Deployment type (Rolling update), and click Next.
configure service

Source: Author

  • Choose the subnets that we have configured in the load balancer
choose subnets

Source: Author

  • Choose Application load balancer for the load balancer type, and then click Add to load balancer
choose application load balancer

Source: Author

  • Select the Target group name that we have created in the Application load balancer and then click Next, Review it and then click Create Service.
  • Now we need to configure the Application load balancer security group that we have created earlier, go to the created Application load balancer and click on security groups and then click Edit inbound rules and add a Custom TCP rule with port 5000, as this is the internal port our application is configured in the flask application
create service | CI/CD pipeline AWS

Source: Author

  • Now, we can check our application is running by visiting the load balancer DNS name in a browser
application running | CI/CD pipeline AWS

Source: Author

Our Service is up and running! Now we can build a CI/CD Pipeline.

AWS CodeCommit

It is a version control service similar to Github, hosted by AWS. For Demo, we will use a CodeCommit repository and push our application code to it (If you want you can use Github also).

  • In AWS console search CodeCommit, create a repository, and give a name to it.
aws codecommit

Source: Author

  • Once the repository is created we need to add specific permission and create Git credentials to access the CodeCommit repository.
  • Go to the IAM console, choose Users and select which User you want to configure for CodeCommit, and attach AWSCodeCommitPowerUser policy from the policies list and Review and then click Add Permission.
code commit add permissions

 

Source: Author

  • Now, to generate Git credentials select the User for which you have added CodeCommit Permission and search for HTTPS Git credentials for AWS CodeCommit section and then click Generate credentials and download it.

 

generate git credentials

Source: Author

  • Now copy the repository URL and clone it on the local system using git clone
copy repository URL

Source: Author

git clone URL
  • For username and password, use from credentials file downloaded earlier.
  • Now we can push our application to this repository, I am using the same application from my previous post, you can download the source code from my Github or use your own application.
git add .
git commit -m 'app:v1.0'
git push

Once you have pushed the code, your CodeCommit repository will look as follows

push the code | CI/CD pipeline AWS

Source: Author

AWS CodeBuild

The next step is to set up Continuous Integration functionality on the CodeCommit repository.

AWS CodeBuild is managed CI Service which compiles source code, runs tests, and packages the source code which can be used for deployment.

To use CodeBuild, we need a buildspec.yml file that contains commands used to compile, test, and package our code. For our purpose, in the buildspec.yml file, we will write commands to containerized our application using docker and push it to ECR. It will look as follows

#Required mapping. Represents the buildspec version.
version: 0.2

#Required sequence. Represents the commands CodeBuild runs during each phase of the build. 
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - CODEBUILD_RESOLVED_SOURCE_VERSION="${CODEBUILD_RESOLVED_SOURCE_VERSION:-$IMAGE_TAG}"
      - IMAGE_TAG=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - echo image_tag $IMAGE_TAG
      - REPO="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
      - IMAGE_URI="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG"
      - echo Repository $REPO
      - docker login -u AWS -p $(aws ecr get-login-password --region $AWS_DEFAULT_REGION) $REPO
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $IMAGE_URI .
  post_build:
    commands:
      - bash -c "if [ /"$CODEBUILD_BUILD_SUCCEEDING/" == /"0/" ]; then exit 1; fi"
      - echo Build stage successfully completed on `date`
      - echo Pushing the Docker image...
      - docker push $IMAGE_URI
      - printf '[{"name":"sentiment-analysis-container","imageUri":"%s"}]' "$IMAGE_URI" > images.json
artifacts:
  files: images.json

I won’t go much into details of it as most of the commands are docker related and syntax such as pre_build, build, etc are self-explanatory. If you want to deep dive into it refer to AWS documentation.

One thing to highlight is artifacts files, here we are saving an ‘images.json‘ file containing the container name that we have created earlier in ECS(Task definition container name) and the Image URI of the container that we will push to ECR. This artifacts file will be needed in the CodePipeline stage.

AWS CodePipeline

CodePipeline will automatically build and deploy our application when there are any changes in the code repository.

  • Go to CodePipeline console, click Get started and configure the pipeline
  • Next, select CodeCommit for source provider (or Github if you are using it, for this, you have to log in to Github) and select the repository and branch to configure for CI/CD
aws codepipeline

Source: Author

  • Next, for the build provider select CodeBuild and create a new build project, give it a name and configure it as follows
codebuild
code build | CI/CD pipeline AWS

Source: Author

  • It will create a New service role, to which we have to add ECRContainerBuilds permission. For that, open the IAM console in a new tab, go to Roles and search and select the above-created role, and click Attach policies. Search for ECR and select the policy as below and click Attach policy
attach policy | CI/CD pipeline AWS

 

  • Now come back to the CodeBuild tab, Expand Additional information, and Add Environment variables, AWS_ACCOUNT_ID and IMAGE_REPO_NAME as we are using these two variables in the buildspec.yml file. You can tick CloudWatch logs and give it a name as it will help to debug the CI process, after that click Continue to CodePipeline and then click Next
add environment variables

Source: Author

  • For Deploy provider, select Amazon ECS, cluster, and service name. Also, add the name of the image definitions file as ‘images.json’ that we will create during the build process
add deploy stage

Source: Author

  • Click Next, Review it and click Create pipeline
  • Once Pipeline is created successfully, it will launch a new container as a Service

Now, let’s test the complete Pipeline by changing the source code and pushing the changes to the Code commit repository.

Make some changes(highlighted text in below image) in the ‘home.html’ file in the templates folder

home.html template | CI/CD pipeline AWS

Source: Author

Now push the changes to the repository

push the changes | CI/CD pipeline AWS

Source: Author

Once the changes are pushed, CodePipeline will trigger the CI/CD process and creates a new deployment of the AWS Fargate Service with the new image build. You can verify it by going to Fargate Services and check the Deployments tab.

deployments tab

Source: Author

Now, to verify our changes are made and deployed successfully, visit the DNS name in the browser as we did earlier

visit DNS |CI/CD pipeline AWS
predict output | CI/CD pipeline AWS

Source: Author

Hooray! We have successfully created a CI/CD Pipeline leveraging AWS services.

Don’t Forget

To clean up the AWS resources, you can delete the Service and cluster created (if you don’t have free credits).

References

https://aws.amazon.com/blogs/compute/building-deploying-and-operating-containerized-applications-with-aws-fargate/

https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html

About Author

Machine Learning Engineer, Solving challenging business problems through Data, Machine Learning, and Cloud.

Connect @ Linkedin

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Deepak
Deepak

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details