Singularity containers on Flux

Singularity is a container technology that enables users to include portions of the operating system along with an application so that an application designed to run on one version of an operating system can be run on a different one. For example, this enables running applications from CentOS 6 on a CentOS 7 computer when there are system library incompatibilities; or to run an application from Ubuntu on CentOS.

What is a Singularity container?

A Singularity container is designed to hold an application, or set of applications, and all the operating system libraries that are needed to run that application.

Outline of building a container

A container is a file, and it contains some operating system components and some application components. The procedure for creating a functional container is

  • Create the container file
  • Run Singularity to put the needed componenets of the operating system into the container (this required privileged access on the computer creating the container)
  • Copy or otherwise install the application or applications into the container
  • Copy the file containing the container to the computer on which you wish to run it
  • Run the application in the container using Singularity

You will not be able to create containers on Flux. You should create your container elsewhere and copy it to Flux, or use one of the containers that has been precreated. Precreated containers can be found in /scratch/data/examples/singularity.

NOTE: This description is for an older version of Singularity. Please see the Singularity web site for more up-to-date instructions and examples.

If you are creating a container specifically to use on Flux, you should insure that the container contains directories for /scratch and, if you have persistent NFS storage, /nfs. If those exist within the container, the corresponding directories on Flux can be made available there. If you have an existing container, you should add those directories from a system on which you have root privileges and create them.

We will use the centos6.img container there for the examples that follow.

We will work backward from an existing container to creating a container, as the supported activity on Flux is running an existing Singularity container.

Running an existing Singularity container

The first step is to copy the Singularity container you wish to run. Let’s assume that you have copied it to the current directory, and that it is called centos6.img. The most basic thing you can do is to run a shell inside the container, which is done with

$ cp /scratch/data/examples/singularity/centos6.img ./
$ singularity shell centos6.img

The prompt will change, and you will have a shell inside the container. Here is an example of a short session to get information about what is in the container.

Singularity.centos6.img> uname -r

Singularity.centos6.img> cat /etc/redhat-release
CentOS release 6.8 (Final)

Singularity.centos6.img> echo 'Hello from Singularity!' > hello.out

Singularity.centos6.img> exit

The uname -r command requests the Linux kernel information, and the redhat-release file contains the operating system release name and version. Compare the output above, from the Singularity container, to the output below, from the Flux operating system.

$ uname -r

$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

The kernel is the same, because the kernel runs only on the host system. However, the release file contains different information.

You will also notice that the Singularity container has mapped some directories inside the container to the corresponding directories outside the container. We echoed some text into hello.txt, and it was created in the folder from which the container was run.

$ ls
centos6.img  hello.out

$ cat hello.out
Hello from Singularity!

If you will need access to data in your project’s /scratch folder, then make sure that /scratch exists inside your container, and use the bind option, -B, to bind it to your container. For example, to run a that expects data to be in /scratch, you would use

$ singularity run -B /scratch centos6.img

Similarly, if you also have data in /nfs, you could use

$ singularity run -B /scratch -B /nfs centos6.img

to make them both accessible from inside the container.

Creating a Singularity container

We cannot cover all the possibilities for creating a Singularity container, and you will not be able to create on Flux. Nonetheless, this is the procedure followed to create an example container that could be run on Flux.

In this case, we had a machine on which we could obtain administrative (root) permissions that was running the operating system we wanted the container to provide, CentOS 6. We followed the instructions for bootstrapping a container found on the Singularity web site

using the following Singularity definition file. which we named CentOS6.def.

# Bootstrap definition Centos-6

BootStrap: yum
OSVersion: 6
Include: yum

    echo "Looking in directory '$SINGULARITY_ROOTFS' for /bin/sh"
    if [ ! -x "$SINGULARITY_ROOTFS/bin/sh" ]; then
        echo "Hrmm, this container does not have /bin/sh installed..."
        exit 1
    exit 0

    echo "Installing Development Tools YUM group"
    yum -y groupinstall "Development Tools"
    yum -y install bc
    mkdir /scratch /nfs

    echo "Arguments received: $*"
    exec /bin/bash "$@"

We found by trial and error that the program bc was needed by our application, so we added that to the definition. This container will run whatever shell script is passed to it as an argument. You can put any command that exists inside the container here. See the Singularity web site for more details.

Next we need to create a container in which to run the bootstrap procedure. We also show here how to set the size of the resulting container.

# size is given in MiB, 1024*1024 bytes, so 10000 is about 10gb
$ sudo singularity create --size 10000 test.img
$ sudo singularity bootstrap centos6.img CentOS6.def

Again, note, that you will not be able to run these commands on Flux.

You should determine how big your container needs to be ahead of time and try to size it appropriately. One procedure for doing so is to prepare an image that is larger than needed, install your application, determine how much disk space is taken, then remake the container using that figure as a guide. For example,

$ singularity shell centos6.img
Singularity.centos6.img> df -h /usr
Filesystem      Size  Used Avail Use% Mounted on
singularity     740M  318M  384M  46% /

So, you could go back and recreate an image that is, say, 325 MB, and it would probably be fine. Note, also, that by default /tmp maps to the /tmp on the host system, so you do not need to leave room inside the container for it.