Introduction
As AI technologies continue to grow and the software development landscape constantly changes, it is critical that software be developed quickly, reliably, and of high quality. In order to satisfy these requirements, continuous integration (CI) and continuous deployment (CD), have become critical methods. While CD automates the deployment process, CI automates the integration and testing of code modifications. This detailed manual will go over the CI/CD with Jenkins, significance of CI/CD, its advantages, and the problems it solves.
Using Jenkins, we will then develop CI/CD for Flask application deployment, expanding on the work we did in our previous article. You can follow the previous articles here to learn about it. The project we are using is on Github.
Table of contents
Understanding CI/CD
Automating the software delivery pipeline from code commit to deployment is the objective of CI/CD. By guaranteeing that code updates are merged and tested, continuous integration, or CI, lowers integration issues and ensures code quality. Continuous Integration (CI) is enhanced by CD by automating the deployment process and making it easier to distribute updates, adjustments, and new features in a reliable and timely manner.
How CI/CD Works ?
CI/CD reduces human error, automates repetitive operations, and gives engineers quick feedback to optimize the software development lifecycle. Updates and new features are released more quickly, enabling teams to react swiftly to market and customer demands. Moreover, CI/CD improves code quality overall, team member contributions, and code transparency.
Need of CI/CD
CI/CD is crucial in modern software development for several reasons:
- Faster Time to Market: These solutions automate software delivery, enabling developers to deploy updates more reliably, providing a competitive advantage and swift market response.
- Improved Software Quality: CI ensures regular testing and integration of code updates into a shared repository, identifying bugs early in development and reducing the risk of breaking changes or introducing bugs.
- Reduced Human Effort and Errors: CI/CD automates software delivery phases, reducing human involvement, error risk, and development time, allowing developers to focus on other projects and customer needs.
- Enhanced Collaboration and Communication: CI/CD tools facilitate collaboration among development, testing, and operations teams by providing visibility into the entire software delivery pipeline, enhancing communication, coordination, and alignment of efforts.
- Scalability and Flexibility: CI/CD practices are adaptable to various project sizes and methodologies, offering flexibility through tools that support programming languages, platforms, and cloud environments.
- Continuous Feedback Loop: CI/CD facilitates a continuous feedback loop by collecting metrics and performance data throughout the software delivery process, enabling teams to identify bottlenecks, optimize workflows, and make data-driven decisions.
Overall, CI/CD is essential for modern software development practices as it enables organizations to deliver high-quality software faster, more reliably, and with greater efficiency, ultimately driving business success and customer satisfaction.
Challenges without CI/CD
Without CI/CD, software development teams and process may encounter several challenges, including:
- Manual Processes prone to error: The time-consuming and human error-prone nature of manual integration, testing, and deployment procedures causes delays and inconsistencies.
- Ineffective release cycles: In the absence of automation, release cycles are hard and slow, making it more difficult to provide updates and new features on time.
- Lack of visibility and feedback: Developers are unable to receive fast feedback on the functionality and quality of their code in the absence of automated testing and deployment, which could result in problems in the real world.
Understanding Jenkins Jobs and Pipeline
Jenkins Pipeline is a collection of plugins designed to facilitate the setup and implementation of CI/CD processes. With the help of a domain-specific language (DSL) based on the programming language, users may define and automate Jenkins operations. This Pipeline allows teams to version manage build, test, and deployment pipelines, as well as application code, promoting collaboration, reliability, and speed in software delivery. Jenkins offers various types of jobs, including freestyle, pipeline, and multi-configuration projects, each tailored to different use cases in the CI/CD process.
- Jobs: Jenkins can handle a wide range of projects, including Multibranch Pipeline, Freestyle, and Maven projects. Every kind of job has unique capabilities and configuration options.
- Configuration: A job can have various configurations, including parameters, build triggers, source code repositories, build stages, and post-build actions.
- Execution: Jenkins assigns resources and executes job configuration instructions, tracking progress and output through the web interface.
- Integration: Jenkins jobs can be integrated with various build tools, testing frameworks, version control systems, and deployment platforms like Docker and Kubernetes for efficient delivery processes.
Jenkins uses projects (also known as “jobs”) to perform its work. Projects are defined and run by Jenkins users. Jenkins offers several types of projects, including:
- Pipeline
- Multibranch Pipeline
- Organization folders
- Freestyle
- Multi-configuration (matrix)
- Maven
- External job
We will be working with the multi branch pipeline for GitHub.
Implementing CI/CD with Jenkins for Flask Application Deployment
Jenkins is an open-source automation server that facilitates CI/CD processes. It provides a platform for automating building, testing, and deployment tasks, making it an essential tool for modern software development pipelines. Jenkins integrates seamlessly with version control systems like Git, enabling developers to trigger builds and deployments automatically based on code changes.
Steps to Implement CI/CD with Jenkins
- Set up Jenkins: Install Jenkins on a server and configure it according to your requirements. Jenkins can be installed on-premises or on a cloud server.
- Connect Jenkins to GitHub: Configure Jenkins to connect to your GitHub repository where the Flask application code is hosted. This allows Jenkins to pull code changes and trigger builds automatically.
- Configure Jenkins Pipeline: Create a Jenkins pipeline to define the stages of the CI/CD process. A typical pipeline includes stages such as building, testing, and deployment.
- Implement Testing: Define automated tests to ensure the quality and functionality of the Flask application. These tests can include unit tests, integration tests, and end-to-end tests.
- Build and Test Stage: Set up Jenkins to automatically build the Flask application whenever there’s a new code commit. After building, execute the automated tests to verify the application’s functionality.
- Deployment Stage: Once the tests pass successfully, deploy the Flask application using Gunicorn and Nginx. Jenkins can execute deployment scripts or commands to upload the application to the cloud server.
- Monitor and Iterate: Monitor the CI/CD pipeline for any failures or issues. Continuously improve the pipeline by incorporating feedback and making optimizations.
Here in this article we are performing CI, in the continuation to this article, we will perform CICD with Docker.
Step1 : Set-up ec2 User and Install Dependencies
Create an Ec2 instance and choose OS ubuntu. Also, don’t forget to add security groups with rule to allow port 8080, as port 8080 is for serving Jenkins.
Then SSH login into your ec2 and remote use it and install the dependencies.
- “sudo apt install openjdk-17-jdk -y“: This command installs the OpenJDK 17 JDK package. Jenkins requires Java to run, and OpenJDK is a popular choice for Java development environments.
- “wget -q -O – https://pkg.jenkins.io/debian-stable/jenkins.io.key |sudo gpg –dearmor -o/ubuntu/share/keyrings/jenkins.gpg“: This command downloads the Jenkins GPG key and stores it in /ubuntu/share/keyrings/jenkins.gpg. This key is used to verify the authenticity of Jenkins packages during installation.
- “curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null” : This command downloads the Jenkins GPG key and stores it in /usr/share/keyrings/jenkins-keyring.asc. This is an alternative way to download the key.
- “echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null” : This command adds the Jenkins repository to the system’s package sources list. It specifies the location of Jenkins packages and the key used for package signing.
sudo apt update
sudo apt upgrade
sudo apt install openjdk-17-jdk -y
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key |sudo
gpg --dearmor -o /ubuntu/share/keyrings/jenkins.gpg
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io.key |
sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]
https://pkg.jenkins.io/debian-stable binary/ |
sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt install python3-pipNow Put in the jenkins commands :
sudo apt install jenkins -y
sudo systemctl status jenkins
sudo systemctl enable --now jenkinsOutput:
Now Open your public IP and add “:8080” to it in http and then in the ssh terminal put the command
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Output:
Copy this token and paste it on the block given on the URL for Jenkins.
Then after login ans setting up the option as “Install suggested plugins”.
Ouput:
Let us begin by building our first pipeline for the project which we used earlier to deploy on cloud and now to put under continuous watch/integration.
Step2: Building Pipeline
- Click on new item
- Enter a name and select Pipeline – MultiBranch here
- Then configure the first pipeline after adding a description and adding the URL of your repo and then select the branch as master and save it.
The repo we are using is here:
You can simply try on the flask model which we used previously. However, if you want to see the changes, then you should use your repo, so that next time you add something, it automatically gets updated in the pipeline.
Also, for the implementation, there must be a Jenkins file in your repo in order for Jenkins to recognize it and read the pipeline saved in it.
Create a file name as “jenkinsfile” in the same directory. You can create while doing it in your local and then put all of the files and folders in a directory and on Github.
The Jenkins file we used:
pipeline {
agent any
environment {
VENV_PATH = 'myprojectenv'
FLASK_APP = 'myproject.py'
}
stages {
stage('Checkout') {
steps {
// Checkout code from a source control
//management system (e.g., Git)
git url: 'https://github.com/Geek-shikha
/Flask_model_sentiment_analysis.git', branch: 'master'
}
}
stage('Setup Virtual Environment') {
steps {
script {
// Check for the virtual environment,
// create it if it doesn't exist
sh 'bash -c "python3 -m venv $VENV_PATH"'
// Activate the virtual environment
sh 'bash -c "source $VENV_PATH/bin/activate"'
}
}
}
stage('Install dependencies') {
steps {
// Install any dependencies listed in requirements.txt
sh 'bash -c "source $VENV_PATH/bin/activate &&
pip install -r requirements.txt"'
}
}
stage('Test') {
steps {
// Run your tests here. This is just a placeholder.
// For example, if you had tests, you might run: pytest
echo "Assuming tests are run here. Please replace
this with actual test commands."
// sh "source $VENV_PATH/bin/activate && pytest"
}
}
stage('Deploy') {
steps {
script {
// Deploy your Flask app
// This step greatly depends on where and
// how you're deploying your app
// For example, if you're deploying to a server you control,
// you might use scp, rsync, or SSH commands
// If you're using a PaaS (Platform as a Service),
//you might use a specific CLI tool for that platform
echo 'Deploying application...'
// Example: sh 'scp -r . user@your_server:/path/to/deploy'
}
}
}
}
post {
always {
// Clean up after the pipeline runs
echo 'Cleaning up...'
sh 'rm -rf ${VIRTUAL_ENV_DIR}'
}
}
}How to Write Jenkins File ?
This Jenkins pipeline script defines a continuous integration and deployment (CI/CD) process for a Flask web application hosted on GitHub. Lets understand important components and concepts:
Pipeline Declaration
- pipeline { … }: This defines the entire Jenkins pipeline.
Agent
- agent any: This tells Jenkins to execute the pipeline on any available agent (Jenkins worker node).
Environment
- environment { … }: This block defines environment variables used throughout the pipeline.
- VENV_PATH: Specifies the path for the virtual environment.
- FLASK_APP: Specifies the name of the Flask application file.
Stages
- stages { … }: This block contains multiple stages representing different steps of the pipeline.
- Checkout: Clones the source code repository from GitHub.
- Setup Virtual Environment: Creates and activates a virtual environment for the project.
- Install Dependencies: Installs Python dependencies listed in requirements.txt.
- Test: Placeholder for running tests. Here, it only echoes a message.
- Deploy: Placeholder for deploying the Flask application. Currently, it only echoes a message. we will complete this in the upcoming article with docker and CICD
Steps
- steps { … }: This block contains individual steps within each stage.
- sh ‘…’: Executes shell commands. Used for various tasks such as creating a virtual environment, activating it, installing dependencies, etc.
- script { … }: Allows for the execution of Groovy script code.
Post-build Actions
- post { … }: This block contains actions to be executed after the pipeline completes, regardless of success or failure.
- always { … }: This block executes its contents regardless of the pipeline status. In this case, it echoes a message indicating cleanup.
Start building by clicking on Build Now :
Now we can put a trigger so that it automatically checks for any github updates and integrate it in the pipeline, we are setting the trigger for 2 minutes.
Triggers
- Go to configure in the Jenkins UI and build Triggers.
- Select poll SCM
- Now add H/2**** and this is will trigger and check for the updates. You can check the website for syntax of timing trigger https://crontab.guru/
To check this, we added a new wsgi.py and this will automatically trigger jenkins to identify the changes.
Here the repo is my created repo, You can do the same with your own created repo, remember it must have a jenkins file to identify. you can add or edit the existing file in your repo and then if you have to set the trigger it will identify and auto update it. otherwise you might have to click on build now everytime you want jenkins to identify the changes.
For continuous deployment, follow the next article on Dockerisation along with CICD pipeline.
Conclusion
Deploying Flask applications using CI/CD with Jenkins is a critical step in improving the effectiveness, dependability, and flexibility of software development processes. These processes help teams to quickly produce high-quality software in response to customer input and market demands by automating the integration, testing, and deployment of code changes. In this comprehensive guide we have examined the value of continuous integration and continuous development, its advantages and difficulties. We’ve also covered the practical side of configuring Jenkins CI/CD pipelines, covering everything from constructing pipelines and comprehending Jenkins jobs to managing the full deployment process.
involves a blend of statistics, data analysis, machine learning, and their related methods to understand and analyze actual phenomena with data. It encompasses a wide range of skills and activities, including but not limited.
