From 5dc0b28488781c8d170ca6912e2f261733fb3ce8 Mon Sep 17 00:00:00 2001
From: Carrie A Brown <cbrown58@unl.edu>
Date: Mon, 6 May 2019 14:52:35 -0500
Subject: [PATCH] Changes to Singulary document to elaborate on how to find
images and create custom ones\ \ fixes issue #7
---
.../running_applications/using_singularity.md | 139 ++++++++++--------
1 file changed, 78 insertions(+), 61 deletions(-)
diff --git a/content/guides/running_applications/using_singularity.md b/content/guides/running_applications/using_singularity.md
index bc38021a..692858f7 100644
--- a/content/guides/running_applications/using_singularity.md
+++ b/content/guides/running_applications/using_singularity.md
@@ -1,8 +1,10 @@
+++
-title = "Using Singularity"
+title = "Using Singularity and Docker Containers"
description = "How to use the Singularity containerization software on HCC resources."
+++
+## What is Singularity
+
[Singularity](https://www.sylabs.io/singularity/)
is a containerization solution designed for high-performance computing
cluster environments. It allows a user on an HPC resource to run an
@@ -17,6 +19,34 @@ differences that make it more suited for HPC environments.
- No user contextual changes or root escalation allowed
- No root owned daemon processes
+## Finding Images
+
+Singularity can run images from a variety of sources, including
+both a flat image file or a Docker image from Docker Hub.
+
+### Docker Hub
+
+Publically available Docker images can be found at [Docker Hub](http://hub.docker.com). For
+convenience, HCC also provides a set of images on [Docker Hub](https://hub.docker.com/u/unlhcc/)
+known to work on HCC resources.
+
+
+### Available Images at HCC
+
+The following table lists the currently available images and the command
+to run the software.
+
+{{< readfile file="static/markdown/singularity-images.md" markdown="true" >}}
+
+{{% notice note %}}
+If you would like to request an image to be added, please fill out the
+HCC [Software Request Form](http://hcc.unl.edu/software-installation-request)
+and indicate you would like to use Singularity.
+{{% /notice %}}
+
+
+## Use images on HCC resources
+
To use Singularity on HCC machines, first load the `singularity `module.
Singularity provides a few different ways to access the container.
Most common is to use the `exec` command to run a specific command
@@ -24,12 +54,9 @@ within the container; alternatively, the `shell` command is used to
launch a bash shell and work interactively. Both commands take the
source of the image to run as the first argument. The `exec` command
takes an additional argument for the command within the container to
-run. Singularity can run images from a variety of sources, including
-both a flat image file or a Docker image from Docker Hub. For
-convenience, HCC provides a set of images on
-[Docker Hub](https://hub.docker.com/u/unlhcc/)
-known to work on HCC resources. Finally, pass any arguments for the
-program itself in the same manner as you would if running it directly.
+run.
+
+Finally, pass any arguments for the program itself in the same manner as you would if running it directly.
For example, the Spades Assembler software is run using the Docker
image `unlhcc/spades` and via the command `spades.py`.
To run the software using Singularity, the commands are:
@@ -41,7 +68,9 @@ singularity exec docker://unlhcc/spades spades.py <spades arguments>
{{< /highlight >}}
{{% /panel %}}
-Using Singularity in a SLURM job is the same as any other software.
+### Use images within a SLURM job
+
+Using Singularity in a SLURM job is similar to how you would use any other software within a job. Load the module, then execute your image:
{{% panel theme="info" header="Example Singularity SLURM script" %}}
{{< highlight bash >}}
@@ -57,27 +86,54 @@ singularity exec docker://unlhcc/spades spades.py <spades arguments>
{{< /highlight >}}
{{% /panel %}}
-### Available Images
+## Create a custom image
-The following table lists the currently available images and the command
-to run the software.
+Custom images can be created locally on your personal machine and added to Docker Hub for use
+on HCC clusters. More information on creating custom Docker images can be found in the [Docker documentation](https://docs.docker.com/develop/develop-images/baseimages/).
-{{% notice note %}}
-If you would like to request an image to be added, please fill out the
-HCC [Software Request Form](http://hcc.unl.edu/software-installation-request)
-and indicate you would like to use Singularity.
-{{% /notice %}}
+You can create custom Docker image and use it with Singularity on our clusters.
+Singularity can run images directly from Docker Hub, so you don't need to upload anything to HCC.
+For this purpose, you just need to have a Docker Hub account and upload
+your image there. Then, if you want to run the command "*mycommand*"
+from the image "*myimage*", type:
-{{< readfile file="static/markdown/singularity-images.md" markdown="true" >}}
+{{< highlight bash >}}
+module load singularity
+singularity exec docker://myaccount/myimage mycommand
+{{< /highlight >}}
+
+where "*myaccount*" is your Docker Hub account.
+
+In case you see the error `ERROR MANIFEST_INVALID: manifest invalid`
+when running the command above, try:
+
+{{< highlight bash >}}
+module load singularity
+unset REGISTRY
+singularity exec docker://myaccount/myimage mycommand
+{{< /highlight >}}
-### What if I need other Python packages not in the image?
{{% notice info %}}
-An alternative to the steps below is to create your own custom image as
-[described farther down](#what-if-i-want-to-build-a-custom-image-to-use-on-the-hcc-clusters).
-Start with an HCC-provided image as the base for your Dockerfile (i.e. `FROM unlhcc/spades`)
-and add any additional packages you desire.
+If you get the error `FATAL: kernel too old` when using your Singularity image on the HCC clusters, that means the *glibc* version in your image is too new for the kernel on the cluster. One way to solve this is to use lower version of your base image (for example, if you have used Ubuntu:18.04 please use Ubuntu:16.04 instead).
{{% /notice %}}
+
+All the Dockerfiles of the images we host on HCC are
+[publicly available here](https://github.com/unlhcc/singularity-dockerfiles)
+You can use them as an example when creating your own image. The only thing you need to note
+when creating custom Docker images you want to use on HCC is to add the line:
+{{< highlight batch >}}
+RUN mkdir -p /work
+{{< /highlight >}}
+at the end of your Dockerfile. This creates a `/work` directory inside your image so
+your `$WORK` directory is available.
+
+### Add packages to an existing image
+
+Alternatively, instead of building an image from scratch, you can start with an HCC-provided
+image as the base for your Dockerfile (i.e. `FROM unlhcc/spades`)
+and add any additional packages you desire.
+
Unfortunately it's not possible to create one image that has every
available Python package installed for logistical reasons. Images are
created with a small set of the most commonly-used scientific packages,
@@ -165,42 +221,3 @@ image for Spades version 3.11.0, run:
{{< highlight bash >}}
singularity exec docker://unlhcc/spades:3.11.0 spades.py
{{< /highlight >}}
-
-### What if I want to build a custom image to use on the HCC clusters?
-
-You can create custom Docker image and use it with Singularity on our clusters.
-Singularity can run images directly from Docker Hub, so you don't need to upload anything to HCC.
-For this purpose, you just need to have a Docker Hub account and upload
-your image there. Then, if you want to run the command "*mycommand*"
-from the image "*myimage*", type:
-
-{{< highlight bash >}}
-module load singularity
-singularity exec docker://myaccount/myimage mycommand
-{{< /highlight >}}
-
-where "*myaccount*" is your Docker Hub account.
-
-In case you see the error `ERROR MANIFEST_INVALID: manifest invalid`
-when running the command above, try:
-
-{{< highlight bash >}}
-module load singularity
-unset REGISTRY
-singularity exec docker://myaccount/myimage mycommand
-{{< /highlight >}}
-
-{{% notice info %}}
-If you get the error `FATAL: kernel too old` when using your Singularity image on the HCC clusters, that means the *glibc* version in your image is too new for the kernel on the cluster. One way to solve this is to use lower version of your base image (for example, if you have used Ubuntu:18.04 please use Ubuntu:16.04 instead).
-{{% /notice %}}
-
-
-All the Dockerfiles of the images we host on HCC are
-[publicly available here](https://github.com/unlhcc/singularity-dockerfiles)
-You can use them as an example when creating your own image. The only thing you need to note
-when creating custom Docker images you want to use on HCC is to add the line:
-{{< highlight batch >}}
-RUN mkdir -p /work
-{{< /highlight >}}
-at the end of your Dockerfile. This creates a `/work` directory inside your image so
-your `$WORK` directory on Crane/Tusker is available.
--
GitLab