Installing Python modules
You can install additional Python libraries on Flux into your own Python library directory (or directories). In addition to installing into library directories, you can also create Python virtual environments that can make managing libraries easier. Information on installing and using virtual environments appears after this section on installing Python libraries.
Much of the value of Python is in its extensibility by using libraries. Most libraries give the option to install into a non-system directory, so you can create and maintain a library collection of your own. This example will use SymPy as the module to install, and we’ll use the directory ${HOME}/pythonlibs
The first step is to download the source code for the module that you want. In this case, the SymPy web site is
http://sympy.org/en/index.html
and from there, we follow links to the Downloads page and find the URL to download the source, which can be done from a Flux login node with the wget program.
$ wget https://github.com/sympy/sympy/releases/download/sympy-0.7.5/sympy-0.7.5.tar.gz
Once the source is downloaded, it needs to be expanded. Most of the time, a directory will be created that is the filename minus the .tar.gz. In this case, it is sympy-0.7.5, and you need to be in that directory to build the source code. We also need to create the target directory.
$ tar xzvf sympy-0.7.5.tar.gz $ cd sympy-0.7.5 $ mkdir ${HOME}/pythonlibs
Most Python libraries will come with a setup.py file that is used to configure, build, and install the library. Some help can be got from
$ python setup.py --help $ python setup.py build --help $ python setup.py install --help
The SymPy build and install is straightforward, so the only option you really need to worry about is the --prefix= option, which tells it where to install the library.
$ python setup.py build $ python setup.py install --prefix=${HOME}/pythonlibs
Once SymPy is installed, Python needs to be made aware of its existence. This is done by exporting an environment variable containing the library directory prior to starting Python.
$ export PYTHONUSERBASE=${HOME}/pythonlibs/lib/python2.7/site-packages
Finally, we want to test it. You want to be in a directory other than the one in which you built the library to test it.
$ cd $ python >>> import sympy >>> x = sympy.symbols('x') >>> sympy.diff(x**2/2, x) x
Note that you will need to export the PYTHONUSERBASE variable prior to running Python the first time in each login session.
Using the Python virtual environment
In addition to installing Python libraries in separate directories, you can set up a ‘virtual environment’. A virtual environment is a local directory that acts as if it were the main Python installation. The virtual environment creates a new directory, which will contain a setup file that needs to be sourced to make the virtual environment active. Once the virtual environment is active, its copy of the python executable will be at the front of the path.
Here is what it would look like to create a virtual environment in the demo directory of your home directory, assuming you want to use the python/2.7.5 module.
$ module load python/2.7.5 $ virtualenv --system-site-packages demo New python executable in demo/bin/python Installing setuptools............done. Installing pip...............done.
Note that we recommend the --system-site-package option, which makes the modules installed with the system available. If for some reason you do not want those, leave that option off. You then need to source the activate script, which will prepend the name of the virtual environment to your prompt.
$ source demo/bin/activate (demo)$ python >>> import numpy as np >>> x = np.array([1,3,5,7]) >>> x array([1, 3, 5, 7])
This makes it possible to use pip to install additional libraries without having administrative access because pip will install into the virtual environment not into the system environment. Here is an example of installing SymPy.
$ pip install sympy Downloading/unpacking sympy Downloading sympy-0.7.5.tar.gz (6.0MB): 6.0MB downloaded Running setup.py egg_info for package sympy Installing collected packages: sympy Running setup.py install for sympy changing mode of build/scripts-2.7/isympy from 644 to 755 changing mode of /home/bennet/demo/bin/isympy to 755 Successfully installed sympy Cleaning up... $ python >>> from sympy import symbols >>> x, y = symbols('x y') >>> expr = x + 2*y >>> z = 2 * expr >>> z 2*x + 4*y