submitting_cuda_or_openacc_jobs.md 3.09 KB
Newer Older
Adam Caprez's avatar
Adam Caprez committed
1
2
3
4
+++
title = "Submitting CUDA or OpenACC Jobs"
description =  "How to submit GPU (CUDA/OpenACC) jobs on HCC resources."
+++
5

Adam Caprez's avatar
Adam Caprez committed
6
### Available GPUs
7
8
9
10
11

Crane has four types of GPUs available in the **gpu** partition. The
type of GPU is configured as a SLURM feature, so you can specify a type
of GPU in your job resource requirements if necessary.

Adam Caprez's avatar
Adam Caprez committed
12
|    Description       | SLURM Feature |      Available Hardware      |
Adam Caprez's avatar
Adam Caprez committed
13
| -------------------- | ------------- | ---------------------------- |
Carrie A Brown's avatar
Carrie A Brown committed
14
15
16
17
| Tesla K20, non-IB    | gpu_k20       | 3 nodes - 2 GPUs with 4 GB mem per node  |
| Teska K20, with IB   | gpu_k20       | 3 nodes - 3 GPUs with 4 GB mem per node    |
| Tesla K40, with IB   | gpu_k40       | 5 nodes - 4 K40M GPUs with 11 GB mem per node<br> 1 node - 2 K40C GPUs |
| Tesla P100, with OPA | gpu_p100      | 2 nodes - 2 GPUs with 12 GB per node |
Adam Caprez's avatar
Adam Caprez committed
18
| Tesla V100, with 10GbE | gpu_v100      | 1 node - 4 GPUs with 16 GB per node |
Adam Caprez's avatar
Adam Caprez committed
19

20
21
22
23

To run your job on the next available GPU regardless of type, add the
following options to your srun or sbatch command:

Adam Caprez's avatar
Adam Caprez committed
24
{{< highlight batch >}}
25
--partition=gpu --gres=gpu
Adam Caprez's avatar
Adam Caprez committed
26
{{< /highlight >}}
27
28
29
30

To run on a specific type of GPU, you can constrain your job to require
a feature. To run on K40 GPUs for example:

Adam Caprez's avatar
Adam Caprez committed
31
{{< highlight batch >}}
32
--partition=gpu --gres=gpu --constraint=gpu_k40
Adam Caprez's avatar
Adam Caprez committed
33
{{< /highlight >}}
34

Adam Caprez's avatar
Adam Caprez committed
35
{{% notice info %}}
36
37
38
39
You may request multiple GPUs by changing the` --gres` value to
-`-gres=gpu:2`. Note that this value is **per node**. For example,
`--nodes=2 --gres=gpu:2 `will request 2 nodes with 2 GPUs each, for a
total of 4 GPUs.
Adam Caprez's avatar
Adam Caprez committed
40
{{% /notice %}}
41

Adam Caprez's avatar
Adam Caprez committed
42
### Compiling
43
44

Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes.
Adam Caprez's avatar
Adam Caprez committed
45
46
Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}})
to compile. An example command to compile in the **gpu** partition could be:
47

Adam Caprez's avatar
Adam Caprez committed
48
{{< highlight batch >}}
49
$ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL
Adam Caprez's avatar
Adam Caprez committed
50
{{< /highlight >}}
51
52
53
54
55

The above command will start a shell on a GPU node with 6 cores and 6GB
of ram in order to compile a GPU job.  The above command could also be
useful if you want to run a test GPU job interactively.

Adam Caprez's avatar
Adam Caprez committed
56
### Submitting Jobs
57
58
59

CUDA and OpenACC submissions require running on GPU nodes.

Adam Caprez's avatar
Adam Caprez committed
60
61
62
63
64
65
66
67
68
69
70
71
{{% panel theme="info" header="cuda.submit" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --time=03:15:00
#SBATCH --mem-per-cpu=1024
#SBATCH --job-name=cuda
#SBATCH --partition=gpu
#SBATCH --gres=gpu
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out

module load cuda/8.0
72
./cuda-app.exe
Adam Caprez's avatar
Adam Caprez committed
73
74
{{< /highlight >}}
{{% /panel %}}
75
76
77
78
79

OpenACC submissions require loading the PGI compiler (which is currently
required to compile as well).


Adam Caprez's avatar
Adam Caprez committed
80
81
82
83
84
85
86
87
88
89
{{% panel theme="info" header="openacc.submit" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --time=03:15:00
#SBATCH --mem-per-cpu=1024
#SBATCH --job-name=cuda-acc
#SBATCH --partition=gpu
#SBATCH --gres=gpu
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
90
91


Adam Caprez's avatar
Adam Caprez committed
92
module load cuda/8.0 compiler/pgi/16
93
./acc-app.exe
Adam Caprez's avatar
Adam Caprez committed
94
95
{{< /highlight >}}
{{% /panel %}}