using_singularity.md 8.77 KB
Newer Older
Adam Caprez's avatar
Adam Caprez committed
1
2
3
4
+++
title = "Using Singularity"
description = "How to use the Singularity containerization software on HCC resources."
+++
5

npavlovikj's avatar
npavlovikj committed
6
[Singularity](https://www.sylabs.io/singularity/)
7
8
9
10
11
12
13
14
is a containerization solution designed for high-performance computing
cluster environments.  It allows a user on an HPC resource to run an
application using a different operating system than the one provided by
the cluster.  For example, the application may require Ubuntu but the
cluster OS is CentOS.  Conceptually, it is similar to other container
software such as Docker, but is designed with several important
differences that make it more suited for HPC environments.  

Adam Caprez's avatar
Adam Caprez committed
15
16
17
18
- Encapsulation of the environment
- Containers are image based
- No user contextual changes or root escalation allowed
- No root owned daemon processes
19
20

To use Singularity on HCC machines, first load the `singularity `module.
Adam Caprez's avatar
Adam Caprez committed
21
22
Singularity provides a few different ways to access the container.
Most common is to use the `exec` command to run a specific command
23
24
25
26
27
28
29
within the container; alternatively, the `shell` command is used to
launch a bash shell and work interactively.  Both commands take the
source of the image to run as the first argument.  The `exec` command
takes an additional argument for the command within the container to
run.  Singularity can run images from a variety of sources, including
both a flat image file or a Docker image from Docker Hub.  For
convenience, HCC provides a set of images on
Adam Caprez's avatar
Adam Caprez committed
30
[Docker Hub](https://hub.docker.com/u/unlhcc/)
31
32
33
known to work on HCC resources.  Finally, pass any arguments for the
program itself in the same manner as you would if running it directly.
 For example, the Spades Assembler software is run using the Docker
Adam Caprez's avatar
Adam Caprez committed
34
35
image `unlhcc/spades` and via the command `spades.py`.
To run the software using Singularity, the commands are:
36

Adam Caprez's avatar
Adam Caprez committed
37
38
{{% panel theme="info" header="Run Spades using Singularity" %}}
{{< highlight bash >}}
39
40
module load singularity
singularity exec docker://unlhcc/spades spades.py <spades arguments>
Adam Caprez's avatar
Adam Caprez committed
41
42
{{< /highlight >}}
{{% /panel %}}
43
44
45

Using Singularity in a SLURM job is the same as any other software.

Adam Caprez's avatar
Adam Caprez committed
46
47
{{% panel theme="info" header="Example Singularity SLURM script" %}}
{{< highlight bash >}}
48
49
50
51
52
53
54
55
56
#!/bin/sh
#SBATCH --time=03:15:00          # Run time in hh:mm:ss
#SBATCH --mem-per-cpu=4096       # Maximum memory required per CPU (in megabytes)
#SBATCH --job-name=singularity-test
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out

module load singularity
singularity exec docker://unlhcc/spades spades.py <spades arguments>
Adam Caprez's avatar
Adam Caprez committed
57
58
{{< /highlight >}}
{{% /panel %}}
59
60
61
62
63
64

### Available Images

The following table lists the currently available images and the command
to run the software.

Adam Caprez's avatar
Adam Caprez committed
65
{{% notice note %}}
66
If you would like to request an image to be added, please fill out the
Adam Caprez's avatar
Adam Caprez committed
67
HCC [Software Request Form](http://hcc.unl.edu/software-installation-request)
68
and indicate you would like to use Singularity.
Adam Caprez's avatar
Adam Caprez committed
69
{{% /notice %}}
70

Adam Caprez's avatar
Adam Caprez committed
71
{{< readfile file="static/markdown/singularity-images.md" markdown="true" >}}
72
73

### What if I need other Python packages not in the image?
Adam Caprez's avatar
Adam Caprez committed
74
75
{{% notice info %}}
An alternative to the steps below is to create your own custom image as
npavlovikj's avatar
npavlovikj committed
76
[described farther down](#what-if-i-want-to-build-a-custom-image-to-use-on-the-hcc-clusters).
Adam Caprez's avatar
Adam Caprez committed
77
78
79
Start with an HCC-provided image as the base for your Dockerfile (i.e. `FROM unlhcc/spades`)
and add any additional packages you desire.
{{% /notice %}}
80
81
82
83
84
85
86
87
88
89
90
91
92

Unfortunately it's not possible to create one image that has every
available Python package installed for logistical reasons.  Images are
created with a small set of the most commonly-used scientific packages,
but you may need others.  If so, you can install them in a location in
your `$WORK` directory and set the `PYTHONPATH` variable to that
location in your submit script.  The extra packages will then be "seen"
by the Python interpreter within the image.  To ensure the packages will
work, the install must be done from within the container via
the `singularity shell` command.  For example, suppose you are using
the `tensorflow-gpu` image and need the packages `nibabel` and `tables`.
 First, run an interactive SLURM job to get a shell on a worker node.

Adam Caprez's avatar
Adam Caprez committed
93
94
{{% panel theme="info" header="Run an interactive SLURM job" %}}
{{< highlight bash >}}
95
srun --pty --mem=4gb --qos=short $SHELL
Adam Caprez's avatar
Adam Caprez committed
96
97
{{< /highlight >}}
{{% /panel %}}
98
99
100
101

After the job starts, the prompt will change to indicate you're on a
worker node.  Next, start an interactive session in the container.

Adam Caprez's avatar
Adam Caprez committed
102
103
{{% panel theme="info" header="Start a shell in the container" %}}
{{< highlight bash >}}
104
105
module load singularity
singularity shell docker://unlhcc/tensorflow-gpu
Adam Caprez's avatar
Adam Caprez committed
106
107
{{< /highlight >}}
{{% /panel %}}
108
109
110
111
112
113
114
115

This may take a few minutes to start.  Again, the prompt will change and
begin with `Singularity` to indicate you're within the container.

Next, install the needed packages via `pip` to a location somewhere in
your `work` directory.  For example, `$WORK/tf-gpu-pkgs`.  (If you are
using Python 3, use `pip3` instead of `pip`).

Adam Caprez's avatar
Adam Caprez committed
116
117
{{% panel theme="info" header="Install needed Python packages with pip" %}}
{{< highlight bash >}}
118
119
export LC_ALL=C
pip install --system --target=$WORK/tf-gpu-pkgs --install-option="--install-scripts=$WORK/tf-gpu-pkgs/bin" nibabel tables
Adam Caprez's avatar
Adam Caprez committed
120
121
{{< /highlight >}}
{{% /panel %}}
122
123
124
125
126
127
128
129
130
131
132
133

You should see some progress indicators, and a
"`Successfully installed..."` message at the end.  Exit both the
container and the interactive SLURM job by typing `exit` twice.  The
above steps only need to be done once per each image you need additional
packages for.   Be sure to use a separate location for each image's
extra packages.

To make the packages visible within the container, you'll need to add a
line to the submit script used for your Singularity job.  Before the
lines to load the `singularity `module and run the script, add a line
setting the `PYTHONPATH` variable to the `$WORK/tf-gpu-pkgs` directory.
Adam Caprez's avatar
Adam Caprez committed
134
For example,
135

Adam Caprez's avatar
Adam Caprez committed
136
137
{{% panel theme="info" header="Example SLURM script" %}}
{{< highlight bash >}}
138
139
140
141
142
143
144
145
146
147
148
149
#!/bin/sh
#SBATCH --time=03:15:00          # Run time in hh:mm:ss
#SBATCH --mem-per-cpu=4096       # Maximum memory required per CPU (in megabytes)
#SBATCH --job-name=singularity-test
#SBATCH --partition=gpu
#SBATCH --gres=gpu
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
 
export PYTHONPATH=$WORK/tf-gpu-pkgs
module load singularity
singularity exec docker://unlhcc/tensorflow-gpu python /path/to/my_tf_code.py
Adam Caprez's avatar
Adam Caprez committed
150
151
{{< /highlight >}}
{{% /panel %}}
152
153
154
155
156
157
158
159
160
161
162

The additional packages should then be available for use by your Python
code running within the container.

### What if I need a specific software version of the Singularity image?

You can see all the available versions of the software built with
Singularity in the table above. If you don't specify a specific sofware
version, Singulariy will use the latest one. If you want to use a
specific version instead, you can append the version number from the
table to the image. For example, if you want to use the Singularity
Adam Caprez's avatar
Adam Caprez committed
163
image for Spades version 3.11.0, run:
164

Adam Caprez's avatar
Adam Caprez committed
165
{{< highlight bash >}}
166
singularity exec docker://unlhcc/spades:3.11.0 spades.py
Adam Caprez's avatar
Adam Caprez committed
167
{{< /highlight >}}
168
169
170

### What if I want to build a custom image to use on the HCC clusters?

Adam Caprez's avatar
Adam Caprez committed
171
172
You can create custom Docker image and use it with Singularity on our clusters.
Singularity can run images directly from Docker Hub, so you don't need to upload anything to HCC.
173
174
For this purpose, you just need to have a Docker Hub account and upload
your image there. Then, if you want to run the command "*mycommand*"
Adam Caprez's avatar
Adam Caprez committed
175
from the image "*myimage*", type:
176

Adam Caprez's avatar
Adam Caprez committed
177
{{< highlight bash >}}
178
179
module load singularity
singularity exec docker://myaccount/myimage mycommand
Adam Caprez's avatar
Adam Caprez committed
180
{{< /highlight >}}
181

Adam Caprez's avatar
Adam Caprez committed
182
where "*myaccount*" is your Docker Hub account.
183

Adam Caprez's avatar
Adam Caprez committed
184
185
In case you see the error `ERROR MANIFEST_INVALID: manifest invalid`
when running the command above, try:
186

Adam Caprez's avatar
Adam Caprez committed
187
{{< highlight bash >}}
188
189
190
module load singularity
unset REGISTRY
singularity exec docker://myaccount/myimage mycommand
Adam Caprez's avatar
Adam Caprez committed
191
192
{{< /highlight >}}

npavlovikj's avatar
npavlovikj committed
193
194
195
196
197
{{% notice info %}}
If you get the error `FATAL: kernel too old` when using your Singularity image on the HCC clusters, that means the *glibc* version in your image is too new for the kernel on the cluster. One way to solve this is to use lower version of your base image (for example, if you have used Ubuntu:18.04 please use Ubuntu:16.04 instead).
{{% /notice %}}


Adam Caprez's avatar
Adam Caprez committed
198
199
200
201
202
203
204
205
206
All the Dockerfiles of the images we host on HCC are
[publicly available here](https://github.com/unlhcc/singularity-dockerfiles)
You can use them as an example when creating your own image. The only thing you need to note
when creating custom Docker images you want to use on HCC is to add the line:
{{< highlight batch >}}
RUN mkdir -p /work
{{< /highlight >}}
at the end of your Dockerfile. This creates a `/work` directory inside your image so
your `$WORK` directory on Crane/Tusker is available.