Choosing the Right Python Environment Tool for Your Next Project

Introduction

Setting up an environment is the first step in Python development, and it’s crucial because package management can be challenging with Python. And also Python is a flexible language that can be applied in various domains, including scientific programming, DevOps, automation, and web development. Given the length and breadth of third-party applications, your global environment will become a bloated mess in no time.

It is a well-known fact in the developer community that Python is bad at managing packages. If there is a meme to sum it up, this would be it;

So, what is the solution to it? Well, to avoid any dependency management-related pitfalls, virtual environments are used. Virtual environments are a must if you are doing any development in Python.

There are a lot of virtual environment tools, each with unique features. In this article, we will discuss some of the most popular Python environments, their unique features, and shortcomings, and also learn how to set up virtual environments using those tools.

Depending on your preference, you may use any of these tools to set up your development environment. But before that, let’s see some benefits of using virtual environments.

This article was published as a part of the Data Science Blogathon.

Table of Contents

1. Why Use Virtual Environments?
2. Tools for managing Virtual Environments
3. Python venv
4. Pyenv + direnv
5. Configure the Direnv
6. Pipenv
7. Poetry

Why Use Virtual Environments?

There are several benefits of using virtual environments:

• Dependency Management: Managing packages while building anything can be trickier. Dev environments can make it easier to manage packages. We can explicitly mention the versions of libraries, and the dev environment will ensure the dependencies are installed and configured.
• Reproducibility: Virtual environments make it easier to recreate the same development environment on multiple machines. It ensures that all project collaborators use the same dependencies and tools. Thus, reducing the risks of code-breaking and bugs.
• Debugging: By isolating the environment, you can more easily identify which packages or configurations are causing issues and address them without affecting other parts of the system.
• Security: Isolating the dev environments will ensure that any security vulnerability is not affecting other parts of the system.
• Isolation: Your code is isolated from the global environment. Whatever packages you install will stay inside the virtual environment. Thus, not polluting your global Python environment. Linux and Mac come with a default Python installation and related packages. installing packages directly could result in problems that you don’t want to deal with. Hence, it is crucial to use virtual dependencies to isolate different installations.

Overall it is crucial to use virtual environments to create a consistent, stable, and secure space for software development.

Tools for Managing Python Virtual Environments

There are a lot of tools for managing virtual environments in Python. In this article, we will discuss some of the widely used virtual environment tools, such as,

1. Python venv
2. Pipenv
3. Pyenv
4. Poetry

Venv Python Environment

Python venv is the most popular virtual environment tool out there. It comes with the Python3 installation. So, if you are using Python3, you do not need to install anything. Venv helps create lightweight virtual environments. Venv makes creating and managing environments simple.

To create a virtual environment type,

python -m venv <env name>

Activate environment

source <path to env folder>/bin/activate

Once activated, you can install any package using pip. To see the installed packages and their versions type,

pip freeze

Save it as a text file type,

pip freeze > requirements.txt

To close the environment, type the deactivate command.

Pyenv

There are times when you require multiple Python installations in your system. For example, when you need to test if a feature works in different versions of Python. But installing multiple Python versions might break your existing code, which nobody wants. The best solution is to use a tool such as Pyenv. Pyenv lets you install different Python versions in your system without causing any quarrels.

Installing and configuring Pyenv might be a bit tricky. Pyenv requires some dependencies to be installed as it builds from the source. If you are on Ubuntu/Debian, execute the below command

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl

For CentOS/Fedora/RHEL,

sudo yum install gcc zlib-devel bzip2 bzip2-devel readline-devel sqlite \
sqlite-devel openssl-devel xz xz-devel libffi-devel

Now, you are ready to install Pyenv.

curl pyenv.run | bash

Now, add the Pyenv to the path in the .bashrc file. Open the file using any editor you wish. We will use “vi” to open and edit the file.

vi ~/.bashrc

Type the letter “o” to enter INSERT mode. Now, copy and paste the following code and restart the shell.

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Now you can install any specific version of Python in your system.

pyenv install python 3.8.15

Checkout Python versions in your system

$ pyenv versions

* system (set by /home/sunil/.pyenv/version)
  3.8.15
  3.10.4

You can see I have two different versions of Python installed. You can also set your global and local Python environments by running the following command.

$ pyenv global 3.8.15
$ python --version
Python 3.8.15

Configure the Direnv

Direnv is one such tool that can improve your productivity as a Python developer. Remember, How often have you forgotten to activate the virtual environment and start executing codes? Well, I think it has happened to most of us. This is where Direnv comes into play. With Pyenv and Direnv configured, you do not need to activate your virtual environment explicitly Direnv will handle this on your behalf.

So, how do we set up Direnv for our environment?

Let’s start by installing Direnv. To install in a Debian/Ubuntu distro type,

sudo apt-get direnv

If you use Fedora/CentOS/RedHat-based distro type,

sudo dnf install direnv

Now, add this file to your .bashrc file.

eval "$(direnv hook bash)"

Add a new file called .direnvrc into your home (~) folder with this content:

use_python() {
  local python_root=$(pyenv root)/versions/$1
  load_prefix "$python_root"
  if [[ -x "$python_root/bin/python" ]]; then
    layout python "$python_root/bin/python"
  else
    echo "Error: $python_root/bin/python can't be executed."
    exit
  fi
}

Create a virtual environment for your project by adding a file named .envrc into the parent folder of the project repository with this content:

use python 3.10.4 #or any version you have installed in pyenv

Now type

direnv allow

Restart or create a new shell and “cd” into it, and see the magic. Your virtual environment will automatically turn on. You will see something like this.

source "/home/sunil/Django/Feed App/.direnv/python-3.10.4/bin/activate"
direnv: loading ~/Django/Feed App/.envrc
direnv: using python 3.10.4
direnv: export +CPATH +LD_LIBRARY_PATH +LIBRARY_PATH +MANPATH +PKG_CONFIG_PATH +VIRTUAL_ENV ~PATH

To deactivate the environment move out of your project directory. So convenient.

Pipenv Python Environment

The next candidate on this list is Pipenv. Pipenv tries to bring the best of all package managers like yarn, npm, cargo, composer, etc. One of the pros of using Pipenv is that you do not need to use pip and venv separately. Apart from this, Pipenv aims to solve several problems using requirements.txt files, such as nested dependencies or packages with multiple sub-dependencies, which might cause errors later while re-creating the code environment.

Pipenv uses Pipfile and a Pipfile. lock file to resolve issues related to complex dependencies. A pipfile is generated automatically when you create a virtual environment with Pipenv. It is similar to a requirements.txt file. It is a high-level specification file for the project and updates in real time. The pipfile lock file specifies the exact versions of the dependencies. The lock file ensures the packages installed are of the same versions across systems.

The setup is easy, install Pipenv through pip,

pip install pipenv

If you are re-creating an environment, type

pipenv install -

This will install all the dependencies listed in the lock file in a virtual environment. To activate the shell type:

pipenv shell

To exit the shell, type exit. Another feature that makes Pipenv a better option is that you can see the dependency graph of installed packages. To view the dependency graph type, “pipenv graph”.

(temp) [sunil@fedora temp]$ pipenv graph
numpy==1.24.2
requests==2.28.2
  - certifi [required: >=2017.4.17, installed: 2022.12.7]
  - charset-normalizer [required: >=2,<4, installed: 3.0.1]
  - idna [required: >=2.5,<4, installed: 3.4]
  - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.14]

To know more about Pipenv, check out their official documentation.

Poetry Python Environment

Poetry is another modern Python tool for dependency management and packaging Python file. It provides a unified way for managing, versioning, packaging, and publishing Python packages. Just as Venv uses requirements.txt and Pipenv uses Pipfile. lock, Poetry uses pyproject. toml to pin dependencies. The Poetry will use the pyproject. toml file to resolve dependencies.

To install Poetry, run the below script;

curl -sSL https://install.python-poetry.org | POETRY_HOME=/etc/poetry python3 -

To start a new project with the Poetry, run the below script;

poetry new project

This will create the project directory with the following folder structure,

.
├── project
│   └── __init__.py
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py

As you can see, it has the project folder with __init__.py, a pyproject. toml file to build dependencies from, a README.md file for specifying information regarding the project, and a test directory for unit testing.

A Py project file (see below) consists of three sections the first section ([tool. poetry]) holds some general information. The second part ([tool. poetry. dependencies]) pins the Python version and dependencies. And the final section ([build-system]) specifies the build system. Refer to their official documentation for information regarding this.

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = ""
authors = ["sunilkumardash9 <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
requests = "^2.28.2"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Installing a library is straightforward. If you are building from a repository with a py project file, poetry will install the dependencies referring to the py project file.

#installing a single library
poetry add requests.py

#installing dependencies from pyproject.toml
poetry install -

This will add an auto-generated poetry lock file to the project directory. Once there is a lock file, from next time Poetry will install dependencies referring to the lock file.

To run a script, you can use the following poetry command;

poetry run python main.py

For example, create a main.py file in your project directory and add these lines to it.

import requests

# Make an HTTP GET request to the API
response = requests.get('https://api.ipify.org')

# Get the IP address from the response text
ip_address = response.text.strip()

print(f"External IP Address: {ip_address}")

Now, run the file using the above command to get the output.

If you do not want to use the Poetry command to run files, spin up a virtual environment with the poetry shell command. This will start a virtual environment, and you can work in it as usual.

Conclusion

Developing inside a virtual environment will make your code stable, consistent, and reliable across the product development life cycle. It is one of the most integral parts of a Python developer’s life. If a project already uses a virtual environment tool, it would be better to stick with that. But if you are starting a new project, it will be better to go with Pipenv and Poetry.

To sum it up, here are the key takeaways from the article,

• We learned the benefits of using virtual environments.
• We learned about the distinctive features of popular python environment tools like Venv, pyenv, Pipenv, and Poetry.
• We also learned how to create and manage virtual environments using these tools.

So, this was all about different open-source Python environments to streamline your Python projects.

I hope you found the article helpful. Follow me on Twitter for more things related to Development and Machine learning.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion. 

Sunil Kumar

Meet your author Sunil kumar Dash, a developer and a writer. Has diverse interests in tech, pop culture, wellness, philosophy and Anime. Exploring underrated music is his hobby. And loves to doom scroll Twitter when bored.