Skip to content
Snippets Groups Projects

Changes to Singulary document to elaborate on how to find images and create custom ones\

Merged Carrie A Brown requested to merge singularity-docker into master
1 file
+ 72
61
Compare changes
  • Side-by-side
  • Inline
+++
title = "Using Singularity"
title = "Using Singularity and Docker Containers"
description = "How to use the Singularity containerization software on HCC resources."
+++
## What is Singularity
[Singularity](https://www.sylabs.io/singularity/)
is a containerization solution designed for high-performance computing
cluster environments. It allows a user on an HPC resource to run an
@@ -17,6 +19,34 @@ differences that make it more suited for HPC environments.  
- No user contextual changes or root escalation allowed
- No root owned daemon processes
## Finding Images
Singularity can run images from a variety of sources, including
both a flat image file or a Docker image from Docker Hub.
### Docker Hub
Publically available Docker images can be found at [Docker Hub](http://hub.docker.com). For
convenience, HCC also provides a set of images on [Docker Hub](https://hub.docker.com/u/unlhcc/)
known to work on HCC resources.
### Available Images at HCC
The following table lists the currently available images and the command
to run the software.
{{< readfile file="static/markdown/singularity-images.md" markdown="true" >}}
{{% notice note %}}
If you would like to request an image to be added, please fill out the
HCC [Software Request Form](http://hcc.unl.edu/software-installation-request)
and indicate you would like to use Singularity.
{{% /notice %}}
## Use images on HCC resources
To use Singularity on HCC machines, first load the `singularity `module.
Singularity provides a few different ways to access the container.
Most common is to use the `exec` command to run a specific command
@@ -24,12 +54,9 @@ within the container; alternatively, the `shell` command is used to
launch a bash shell and work interactively. Both commands take the
source of the image to run as the first argument. The `exec` command
takes an additional argument for the command within the container to
run. Singularity can run images from a variety of sources, including
both a flat image file or a Docker image from Docker Hub. For
convenience, HCC provides a set of images on
[Docker Hub](https://hub.docker.com/u/unlhcc/)
known to work on HCC resources. Finally, pass any arguments for the
program itself in the same manner as you would if running it directly.
run.
Finally, pass any arguments for the program itself in the same manner as you would if running it directly.
For example, the Spades Assembler software is run using the Docker
image `unlhcc/spades` and via the command `spades.py`.
To run the software using Singularity, the commands are:
@@ -41,7 +68,9 @@ singularity exec docker://unlhcc/spades spades.py <spades arguments>
{{< /highlight >}}
{{% /panel %}}
Using Singularity in a SLURM job is the same as any other software.
### Use images within a SLURM job
Using Singularity in a SLURM job is similar to how you would use any other software within a job. Load the module, then execute your image:
{{% panel theme="info" header="Example Singularity SLURM script" %}}
{{< highlight bash >}}
@@ -57,27 +86,48 @@ singularity exec docker://unlhcc/spades spades.py <spades arguments>
{{< /highlight >}}
{{% /panel %}}
### Available Images
## Create a custom image
The following table lists the currently available images and the command
to run the software.
Custom images can be created locally on your personal machine and added to Docker Hub for use
on HCC clusters. More information on creating custom Docker images can be found in the [Docker documentation](https://docs.docker.com/develop/develop-images/baseimages/).
{{% notice note %}}
If you would like to request an image to be added, please fill out the
HCC [Software Request Form](http://hcc.unl.edu/software-installation-request)
and indicate you would like to use Singularity.
{{% /notice %}}
You can create custom Docker image and use it with Singularity on our clusters.
Singularity can run images directly from Docker Hub, so you don't need to upload anything to HCC.
For this purpose, you just need to have a Docker Hub account and upload
your image there. Then, if you want to run the command "*mycommand*"
from the image "*myimage*", type:
{{< readfile file="static/markdown/singularity-images.md" markdown="true" >}}
{{< highlight bash >}}
module load singularity
singularity exec docker://myaccount/myimage mycommand
{{< /highlight >}}
where "*myaccount*" is your Docker Hub account.
In case you see the error `ERROR MANIFEST_INVALID: manifest invalid`
when running the command above, try:
{{< highlight bash >}}
module load singularity
unset REGISTRY
singularity exec docker://myaccount/myimage mycommand
{{< /highlight >}}
### What if I need other Python packages not in the image?
{{% notice info %}}
An alternative to the steps below is to create your own custom image as
[described farther down](#what-if-i-want-to-build-a-custom-image-to-use-on-the-hcc-clusters).
Start with an HCC-provided image as the base for your Dockerfile (i.e. `FROM unlhcc/spades`)
and add any additional packages you desire.
If you get the error `FATAL: kernel too old` when using your Singularity image on the HCC clusters, that means the *glibc* version in your image is too new for the kernel on the cluster. One way to solve this is to use lower version of your base image (for example, if you have used Ubuntu:18.04 please use Ubuntu:16.04 instead).
{{% /notice %}}
All the Dockerfiles of the images we host on HCC are
[publicly available here](https://github.com/unlhcc/singularity-dockerfiles).
You can use them as an example when creating your own image.
### Add packages to an existing image
Alternatively, instead of building an image from scratch, you can start with an HCC-provided
image as the base for your Dockerfile (i.e. `FROM unlhcc/spades`)
and add any additional packages you desire.
Unfortunately it's not possible to create one image that has every
available Python package installed for logistical reasons. Images are
created with a small set of the most commonly-used scientific packages,
@@ -165,42 +215,3 @@ image for Spades version 3.11.0, run:
{{< highlight bash >}}
singularity exec docker://unlhcc/spades:3.11.0 spades.py
{{< /highlight >}}
### What if I want to build a custom image to use on the HCC clusters?
You can create custom Docker image and use it with Singularity on our clusters.
Singularity can run images directly from Docker Hub, so you don't need to upload anything to HCC.
For this purpose, you just need to have a Docker Hub account and upload
your image there. Then, if you want to run the command "*mycommand*"
from the image "*myimage*", type:
{{< highlight bash >}}
module load singularity
singularity exec docker://myaccount/myimage mycommand
{{< /highlight >}}
where "*myaccount*" is your Docker Hub account.
In case you see the error `ERROR MANIFEST_INVALID: manifest invalid`
when running the command above, try:
{{< highlight bash >}}
module load singularity
unset REGISTRY
singularity exec docker://myaccount/myimage mycommand
{{< /highlight >}}
{{% notice info %}}
If you get the error `FATAL: kernel too old` when using your Singularity image on the HCC clusters, that means the *glibc* version in your image is too new for the kernel on the cluster. One way to solve this is to use lower version of your base image (for example, if you have used Ubuntu:18.04 please use Ubuntu:16.04 instead).
{{% /notice %}}
All the Dockerfiles of the images we host on HCC are
[publicly available here](https://github.com/unlhcc/singularity-dockerfiles)
You can use them as an example when creating your own image. The only thing you need to note
when creating custom Docker images you want to use on HCC is to add the line:
{{< highlight batch >}}
RUN mkdir -p /work
{{< /highlight >}}
at the end of your Dockerfile. This creates a `/work` directory inside your image so
your `$WORK` directory on Crane/Tusker is available.
Loading