Verified Commit d93c94e8 authored by Adam Caprez's avatar Adam Caprez
Browse files

Add GPU memm docs.

parent d7771cf2
......@@ -18,6 +18,19 @@ of GPU in your job resource requirements if necessary.
| Tesla V100, with 10GbE | gpu_v100 | 1 node - 4 GPUs with 16 GB per node |
| Tesla V100, with OPA | gpu_v100 | 21 nodes - 2 GPUs with 32GB per node |
### Specifying GPU memory (optional)
You may optionally specify a GPU memory amount via the use of an additional feature statement.
The available memory specifcations are:
| Description | SLURM Feature |
| -------------- | ------------- |
| 12 GB RAM | gpu_12gb |
| 16 GB RAM | gpu_16gb |
| 32 GB RAM | gpu_32gb |
### Requesting GPU resources in your SLURM script
To run your job on the next available GPU regardless of type, add the
following options to your srun or sbatch command:
......@@ -33,6 +46,23 @@ a feature. To run on K40 GPUs for example:
--partition=gpu --gres=gpu --constraint=gpu_k40
{{< /highlight >}}
The GPU memory feature may be used to specify a GPU RAM amount either independent of architecture, or in combination with it.
For example, using
{{< highlight batch >}}
--partition=gpu --gres=gpu --constraint=gpu_16gb
{{< /highlight >}}
will request a GPU with 16GB of RAM, independent of the type of card (K20, K40, P100, etc.). You may also
request both a GPU type _and_ memory amount using the `&` operator. For example,
{{< highlight batch >}}
--partition=gpu --gres=gpu --constraint=gpu_32gb&gpu_v100
{{< /highlight >}}
will request a V100 GPU with 32GB RAM.
{{% notice info %}}
You may request multiple GPUs by changing the` --gres` value to
-`-gres=gpu:2`. Note that this value is **per node**. For example,
......@@ -44,14 +74,14 @@ total of 4 GPUs.
Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes.
Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}})
to compile. An example command to compile in the **gpu** partition could be:
to compile. An example command to compile in the `gpu` partition could be:
{{< highlight batch >}}
$ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL
$ srun --partition=gpu --gres=gpu --mem=4gb --ntasks-per-node=2 --nodes=1 --pty $SHELL
{{< /highlight >}}
The above command will start a shell on a GPU node with 6 cores and 6GB
of ram in order to compile a GPU job.  The above command could also be
The above command will start a shell on a GPU node with 2 cores and 4GB
of RAM in order to compile a GPU job.  The above command could also be
useful if you want to run a test GPU job interactively.
### Submitting Jobs
......@@ -69,7 +99,7 @@ CUDA and OpenACC submissions require running on GPU nodes.
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load cuda/8.0
module load cuda
./cuda-app.exe
{{< /highlight >}}
{{% /panel %}}
......@@ -110,4 +140,3 @@ To submit jobs to these resources, add the following to your srun or sbatch comm
**In order to properly utilize pre-emptable resources, your job must be able to support
some type of checkpoint/resume functionality.**
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment