Installing Python Modules and Virtual Environments

Installing Python modules

You can install additional Python libraries on Flux into your own Python library directory (or directories). In addition to installing into library directories, you can also create Python virtual environments that can make managing libraries easier. Information on installing and using virtual environments appears after this section on installing Python libraries.

Much of the value of Python is in its extensibility by using libraries. Most libraries give the option to install into a non-system directory, so you can create and maintain a library collection of your own. This example will use SymPy as the module to install, and we’ll use the directory ${HOME}/pythonlibs

The first step is to download the source code for the module that you want. In this case, the SymPy web site is

http://sympy.org/en/index.html

and from there, we follow links to the Downloads page and find the URL to download the source, which can be done from a Flux login node with the wget program.

$ wget https://github.com/sympy/sympy/releases/download/sympy-0.7.5/sympy-0.7.5.tar.gz

Once the source is downloaded, it needs to be expanded. Most of the time, a directory will be created that is the filename minus the .tar.gz. In this case, it is sympy-0.7.5, and you need to be in that directory to build the source code. We also need to create the target directory.

$ tar xzvf sympy-0.7.5.tar.gz
$ cd sympy-0.7.5
$ mkdir ${HOME}/pythonlibs

Most Python libraries will come with a setup.py file that is used to configure, build, and install the library. Some help can be got from

$ python setup.py --help
$ python setup.py build --help
$ python setup.py install --help

The SymPy build and install is straightforward, so the only option you really need to worry about is the --prefix= option, which tells it where to install the library.

$ python setup.py build
$ python setup.py install --prefix=${HOME}/pythonlibs

Once SymPy is installed, Python needs to be made aware of its existence. This is done by exporting an environment variable containing the library directory prior to starting Python.

$ export PYTHONUSERBASE=${HOME}/pythonlibs/lib/python2.7/site-packages

Finally, we want to test it. You want to be in a directory other than the one in which you built the library to test it.

$ cd
$ python
>>> import sympy
>>> x = sympy.symbols('x')
>>> sympy.diff(x**2/2, x)
x

Note that you will need to export the PYTHONUSERBASE variable prior to running Python the first time in each login session.

Using the Python virtual environment

In addition to installing Python libraries in separate directories, you can set up a ‘virtual environment’. A virtual environment is a local directory that acts as if it were the main Python installation. The virtual environment creates a new directory, which will contain a setup file that needs to be sourced to make the virtual environment active. Once the virtual environment is active, its copy of the python executable will be at the front of the path.

Here is what it would look like to create a virtual environment in the demo directory of your home directory, assuming you want to use the python/2.7.5 module.

$ module load python/2.7.5
$ virtualenv --system-site-packages demo
New python executable in demo/bin/python
Installing setuptools............done.
Installing pip...............done.

Note that we recommend the --system-site-package option, which makes the modules installed with the system available. If for some reason you do not want those, leave that option off. You then need to source the activate script, which will prepend the name of the virtual environment to your prompt.

$ source demo/bin/activate
(demo)$ python
>>> import numpy as np
>>> x = np.array([1,3,5,7])
>>> x
array([1, 3, 5, 7])

This makes it possible to use pip to install additional libraries without having administrative access because pip will install into the virtual environment not into the system environment. Here is an example of installing SymPy.

$ pip install sympy
Downloading/unpacking sympy
  Downloading sympy-0.7.5.tar.gz (6.0MB): 6.0MB downloaded
  Running setup.py egg_info for package sympy

Installing collected packages: sympy
  Running setup.py install for sympy
    changing mode of build/scripts-2.7/isympy from 644 to 755

    changing mode of /home/bennet/demo/bin/isympy to 755
Successfully installed sympy
Cleaning up...
$ python
>>> from sympy import symbols
>>> x, y = symbols('x y')
>>> expr = x + 2*y
>>> z = 2 * expr
>>> z
2*x + 4*y