diff --git a/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md b/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md index 566b43a8188e230a28e8e1c1a1b3744f4dbf1a7e..eab255f31fabd50e1f099492142220c12d9e363a 100644 --- a/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md +++ b/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md @@ -18,6 +18,19 @@ of GPU in your job resource requirements if necessary. | Tesla V100, with 10GbE | gpu_v100 | 1 node - 4 GPUs with 16 GB per node | | Tesla V100, with OPA | gpu_v100 | 21 nodes - 2 GPUs with 32GB per node | +### Specifying GPU memory (optional) + +You may optionally specify a GPU memory amount via the use of an additional feature statement. +The available memory specifcations are: + +| Description | SLURM Feature | +| -------------- | ------------- | +| 12 GB RAM | gpu_12gb | +| 16 GB RAM | gpu_16gb | +| 32 GB RAM | gpu_32gb | + + +### Requesting GPU resources in your SLURM script To run your job on the next available GPU regardless of type, add the following options to your srun or sbatch command: @@ -33,6 +46,23 @@ a feature. To run on K40 GPUs for example: --partition=gpu --gres=gpu --constraint=gpu_k40 {{< /highlight >}} +The GPU memory feature may be used to specify a GPU RAM amount either independent of architecture, or in combination with it. +For example, using + +{{< highlight batch >}} +--partition=gpu --gres=gpu --constraint=gpu_16gb +{{< /highlight >}} + +will request a GPU with 16GB of RAM, independent of the type of card (K20, K40, P100, etc.). You may also +request both a GPU type _and_ memory amount using the `&` operator. For example, + +{{< highlight batch >}} +--partition=gpu --gres=gpu --constraint=gpu_32gb&gpu_v100 +{{< /highlight >}} + +will request a V100 GPU with 32GB RAM. + + {{% notice info %}} You may request multiple GPUs by changing the` --gres` value to -`-gres=gpu:2`. Note that this value is **per node**. For example, @@ -44,14 +74,14 @@ total of 4 GPUs. Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes. Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}}) -to compile. An example command to compile in the **gpu** partition could be: +to compile. An example command to compile in the `gpu` partition could be: {{< highlight batch >}} -$ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL +$ srun --partition=gpu --gres=gpu --mem=4gb --ntasks-per-node=2 --nodes=1 --pty $SHELL {{< /highlight >}} -The above command will start a shell on a GPU node with 6 cores and 6GB -of ram in order to compile a GPU job. The above command could also be +The above command will start a shell on a GPU node with 2 cores and 4GB +of RAM in order to compile a GPU job. The above command could also be useful if you want to run a test GPU job interactively. ### Submitting Jobs @@ -69,7 +99,7 @@ CUDA and OpenACC submissions require running on GPU nodes. #SBATCH --error=/work/[groupname]/[username]/job.%J.err #SBATCH --output=/work/[groupname]/[username]/job.%J.out -module load cuda/8.0 +module load cuda ./cuda-app.exe {{< /highlight >}} {{% /panel %}}