From d93c94e8ddc983a75b8a67f77c4644983a584713 Mon Sep 17 00:00:00 2001
From: Adam Caprez <acaprez2@unl.edu>
Date: Mon, 18 Nov 2019 21:35:51 +0000
Subject: [PATCH] Add GPU memm docs.
---
.../submitting_cuda_or_openacc_jobs.md | 41 ++++++++++++++++---
1 file changed, 35 insertions(+), 6 deletions(-)
diff --git a/content/Submitting_Jobs/submitting_cuda_or_openacc_jobs.md b/content/Submitting_Jobs/submitting_cuda_or_openacc_jobs.md
index f35bc762..f564e5de 100644
--- a/content/Submitting_Jobs/submitting_cuda_or_openacc_jobs.md
+++ b/content/Submitting_Jobs/submitting_cuda_or_openacc_jobs.md
@@ -18,6 +18,19 @@ of GPU in your job resource requirements if necessary.
| Tesla V100, with 10GbE | gpu_v100 | 1 node - 4 GPUs with 16 GB per node |
| Tesla V100, with OPA | gpu_v100 | 21 nodes - 2 GPUs with 32GB per node |
+### Specifying GPU memory (optional)
+
+You may optionally specify a GPU memory amount via the use of an additional feature statement.
+The available memory specifcations are:
+
+| Description | SLURM Feature |
+| -------------- | ------------- |
+| 12 GB RAM | gpu_12gb |
+| 16 GB RAM | gpu_16gb |
+| 32 GB RAM | gpu_32gb |
+
+
+### Requesting GPU resources in your SLURM script
To run your job on the next available GPU regardless of type, add the
following options to your srun or sbatch command:
@@ -33,6 +46,23 @@ a feature. To run on K40 GPUs for example:
--partition=gpu --gres=gpu --constraint=gpu_k40
{{< /highlight >}}
+The GPU memory feature may be used to specify a GPU RAM amount either independent of architecture, or in combination with it.
+For example, using
+
+{{< highlight batch >}}
+--partition=gpu --gres=gpu --constraint=gpu_16gb
+{{< /highlight >}}
+
+will request a GPU with 16GB of RAM, independent of the type of card (K20, K40, P100, etc.). You may also
+request both a GPU type _and_ memory amount using the `&` operator. For example,
+
+{{< highlight batch >}}
+--partition=gpu --gres=gpu --constraint=gpu_32gb&gpu_v100
+{{< /highlight >}}
+
+will request a V100 GPU with 32GB RAM.
+
+
{{% notice info %}}
You may request multiple GPUs by changing the` --gres` value to
-`-gres=gpu:2`. Note that this value is **per node**. For example,
@@ -44,14 +74,14 @@ total of 4 GPUs.
Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes.
Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}})
-to compile. An example command to compile in the **gpu** partition could be:
+to compile. An example command to compile in the `gpu` partition could be:
{{< highlight batch >}}
-$ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL
+$ srun --partition=gpu --gres=gpu --mem=4gb --ntasks-per-node=2 --nodes=1 --pty $SHELL
{{< /highlight >}}
-The above command will start a shell on a GPU node with 6 cores and 6GB
-of ram in order to compile a GPU job. The above command could also be
+The above command will start a shell on a GPU node with 2 cores and 4GB
+of RAM in order to compile a GPU job. The above command could also be
useful if you want to run a test GPU job interactively.
### Submitting Jobs
@@ -69,7 +99,7 @@ CUDA and OpenACC submissions require running on GPU nodes.
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
-module load cuda/8.0
+module load cuda
./cuda-app.exe
{{< /highlight >}}
{{% /panel %}}
@@ -110,4 +140,3 @@ To submit jobs to these resources, add the following to your srun or sbatch comm
**In order to properly utilize pre-emptable resources, your job must be able to support
some type of checkpoint/resume functionality.**
-
--
GitLab