10.2. Use Virtual Environments for Isolated and Reproducible Dependencies¶
Building larger and more complex programs often leads you to rely on various packages from the Python community (see Item 82: “Know Where to Find Community-Built Modules”). You’ll find yourself running the python3 -m pip command-line tool to install packages like pytz, numpy, and many others.
The problem is that, by default, pip installs new packages in a global location. That causes all Python programs on your system to be affected by these installed modules. In theory, this shouldn’t be an issue. If you install a package and never import it, how could it affect your programs?
The trouble comes from transitive dependencies: the packages that the packages you install depend on. For example, you can see what the Sphinx package depends on after installing it by asking pip:
$ python3 -m pip show Sphinx Name: Sphinx Version: 2.1.2 Summary: Python documentation generator Location: /usr/local/lib/python3.8/site-packages Requires: alabaster, imagesize, requests, ➥ sphinxcontrib-applehelp, sphinxcontrib-qthelp, ➥ Jinja2, setuptools, sphinxcontrib-jsmath, ➥ sphinxcontrib-serializinghtml, Pygments, snowballstemmer, ➥ packaging, sphinxcontrib-devhelp, sphinxcontrib-htmlhelp, ➥ babel, docutils Required-by:
If you install another package like flask, you can see that it, too, depends on the Jinja2 package:
$ python3 -m pip show flask Name: Flask Version: 1.0.3 Summary: A simple framework for building complex web applications. Location: /usr/local/lib/python3.8/site-packages Requires: itsdangerous, click, Jinja2, Werkzeug Required-by:
A dependency conflict can arise as Sphinx and flask diverge over time. Perhaps right now they both require the same version of Jinja2, and everything is fine. But six months or a year from now, Jinja2 may release a new version that makes breaking changes to users of the library. If you update your global version of Jinja2 with python3 -m pip install –upgrade Jinja2, you may find that Sphinx breaks, while flask keeps working.
The cause of such breakage is that Python can have only a single global version of a module installed at a time. If one of your installed packages must use the new version and another package must use the old version, your system isn’t going to work properly; this situation is often called dependency hell.
Such breakage can even happen when package maintainers try their best to preserve API compatibility between releases (see Item 85: “Use Packages to Organize Modules and Provide Stable APIs”). New versions of a library can subtly change behaviors that API-consuming code relies on. Users on a system may upgrade one package to a new version but not others, which could break dependencies. If you’re not careful there’s a constant risk of the ground moving beneath your feet.
These difficulties are magnified when you collaborate with other developers who do their work on separate computers. It’s best to assume the worst: that the versions of Python and global packages that they have installed on their machines will be slightly different from yours. This can cause frustrating situations such as a codebase working perfectly on one programmer’s machine and being completely broken on another’s.
The solution to all of these problems is using a tool called venv, which provides virtual environments. Since Python 3.4, pip and the venv module have been available by default along with the Python installation (accessible with python -m venv).
venv allows you to create isolated versions of the Python environment. Using venv, you can have many different versions of the same package installed on the same system at the same time without conflicts. This means you can work on many different projects and use many different tools on the same computer. venv does this by installing explicit versions of packages and their dependencies into completely separate directory structures. This makes it possible to reproduce a Python environment that you know will work with your code. It’s a reliable way to avoid surprising breakages.
10.2.1. Using venv on the Command Line¶
Here’s a quick tutorial on how to use venv effectively. Before using the tool, it’s important to note the meaning of the python3 command line on your system. On my computer, python3 is located in the /usr/local/bin directory and evaluates to version 3.8.0 (see Item 1: “Know Which Version of Python You’re Using”):
$ which python3 /usr/local/bin/python3 $ python3 --version Python 3.8.0
To demonstrate the setup of my environment, I can test that running a command to import the pytz module doesn’t cause an error. This works because I already have the pytz package installed as a global module:
$ python3 -c 'import pytz' $
Now, I use venv to create a new virtual environment called myproject. Each virtual environment must live in its own unique directory. The result of the command is a tree of directories and files that are used to manage the virtual environment:
$ python3 -m venv myproject $ cd myproject $ ls bin include lib pyvenv.cfg
To start using the virtual environment, I use the source command from my shell on the bin/activate script. activate modifies all of my environment variables to match the virtual environment. It also updates my command-line prompt to include the virtual environment name (“myproject”) to make it extremely clear what I’m working on:
$ source bin/activate (myproject)$
On Windows the same script is available as:
C:> myprojectScriptsactivate.bat (myproject) C:>
Or with PowerShell as:
PS C:> myprojectScriptsactivate.ps1 (myproject) PS C:>
After activation, the path to the python3 command-line tool has moved to within the virtual environment directory:
(myproject)$ which python3 /tmp/myproject/bin/python3 (myproject)$ ls -l /tmp/myproject/bin/python3 ... -> /usr/local/bin/python3.8
This ensures that changes to the outside system will not affect the virtual environment. Even if the outer system upgrades its default python3 to version 3.9, my virtual environment will still explicitly point to version 3.8.
The virtual environment I created with venv starts with no packages installed except for pip and setuptools. Trying to use the pytz package that was installed as a global module in the outside system will fail because it’s unknown to the virtual environment:
(myproject)$ python3 -c 'import pytz' Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pytz'
I can use the pip command-line tool to install the pytz module into my virtual environment:
(myproject)$ python3 -m pip install pytz Collecting pytz
Downloading ...
Installing collected packages: pytz Successfully installed pytz-2019.1
Once it’s installed, I can verify that it’s working by using the same test import command:
(myproject)$ python3 -c 'import pytz' (myproject)$
When I’m done with a virtual environment and want to go back to my default system, I use the deactivate command. This restores my environment to the system defaults, including the location of the python3 command-line tool:
(myproject)$ which python3 /tmp/myproject/bin/python3 (myproject)$ deactivate $ which python3 /usr/local/bin/python3
If I ever want to work in the myproject environment again, I can just run source bin/activate in the directory as before.
Reproducing Dependencies Once you are in a virtual environment, you can continue installing packages in it with pip as you need them. Eventually, you might want to copy your environment somewhere else. For example, say that I want to reproduce the development environment from my workstation on a server in a datacenter. Or maybe I want to clone someone else’s environment on my own machine so I can help debug their code.
venv makes such tasks easy. I can use the python3 -m pip freeze command to save all of my explicit package dependencies into a file (which, by convention, is named requirements.txt):
(myproject)$ python3 -m pip freeze > requirements.txt (myproject)$ cat requirements.txt certifi==2019.3.9 chardet==3.0.4 idna==2.8 numpy==1.16.2 pytz==2018.9 requests==2.21.0 urllib3==1.24.1
Now, imagine that I’d like to have another virtual environment that matches the myproject environment. I can create a new directory as before by using venv and activate it:
$ python3 -m venv otherproject $ cd otherproject $ source bin/activate (otherproject)$
The new environment will have no extra packages installed:
(otherproject)$ python3 -m pip list Package Version ---------- ------- pip 10.0.1 setuptools 39.0.1
I can install all of the packages from the first environment by running python3 -m pip install on the requirements.txt that I generated with the python3 -m pip freeze command:
(otherproject)$ python3 -m pip install -r /tmp/myproject/ ➥ requirements.txt
This command cranks along for a little while as it retrieves and installs all of the packages required to reproduce the first environment. When it’s done, I can list the set of installed packages in the second virtual environment and should see the same list of dependencies found in the first virtual environment:
(otherproject)$ python3 -m pip list Package Version ---------- -------- certifi 2019.3.9 chardet 3.0.4 idna 2.8 numpy 1.16.2 pip 10.0.1 pytz 2018.9 requests 2.21.0 setuptools 39.0.1 urllib3 1.24.1
Using a requirements.txt file is ideal for collaborating with others through a revision control system. You can commit changes to your code at the same time you update your list of package dependencies, ensuring that they move in lockstep. However, it’s important to note that the specific version of Python you’re using is not included in the requirements.txt file, so that must be managed separately.
The gotcha with virtual environments is that moving them breaks everything because all of the paths, like the python3 command-line tool, are hard-coded to the environment’s install directory. But ultimately this limitation doesn’t matter. The whole purpose of virtual environments is to make it easy to reproduce a setup. Instead of moving a virtual environment directory, just use python3 -m pip freeze on the old one, create a new virtual environment somewhere else, and reinstall everything from the requirements.txt file.
10.2.2. Things to Remember¶
✦ Virtual environments allow you to use pip to install many different versions of the same package on the same machine without conflicts.
✦ Virtual environments are created with python -m venv, enabled with source bin/activate, and disabled with deactivate.
✦ You can dump all of the requirements of an environment with python3 -m pip freeze. You can reproduce an environment by running python3 -m pip install -r requirements.txt.