Add GPU memm docs.

content/Submitting_Jobs/submitting_cuda_or_openacc_jobs.md

@@ -18,6 +18,19 @@ of GPU in your job resource requirements if necessary.
 | Tesla V100, with 10GbE | gpu_v100      | 1 node - 4 GPUs with 16 GB per node |
 | Tesla V100, with OPA   | gpu_v100      | 21 nodes - 2 GPUs with 32GB per node |
 
+### Specifying GPU memory (optional)
+
+You may optionally specify a GPU memory amount via the use of an additional feature statement.
+The available memory specifcations are:
+
+| Description    | SLURM Feature |
+| -------------- | ------------- |
+| 12 GB RAM      | gpu_12gb      |
+| 16 GB RAM      | gpu_16gb      |
+| 32 GB RAM      | gpu_32gb      |
+
+
+### Requesting GPU resources in your SLURM script
 
 To run your job on the next available GPU regardless of type, add the
 following options to your srun or sbatch command:
@@ -33,6 +46,23 @@ a feature. To run on K40 GPUs for example:
 --partition=gpu --gres=gpu --constraint=gpu_k40
 {{< /highlight >}}
 
+The GPU memory feature  may be used to specify a GPU RAM amount either independent of architecture, or in combination with it.
+For example, using
+
+{{< highlight batch >}}
+--partition=gpu --gres=gpu --constraint=gpu_16gb
+{{< /highlight >}}
+
+will request a GPU with 16GB of RAM, independent of the type of card (K20, K40, P100, etc.).  You may also
+request both a GPU type _and_ memory amount using the `&` operator.  For example,
+
+{{< highlight batch >}}
+--partition=gpu --gres=gpu --constraint=gpu_32gb&gpu_v100
+{{< /highlight >}}
+
+will request a V100 GPU with 32GB RAM.
+
+
 {{% notice info %}}
 You may request multiple GPUs by changing the` --gres` value to
 -`-gres=gpu:2`. Note that this value is **per node**. For example,
@@ -44,14 +74,14 @@ total of 4 GPUs.
 
 Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes.
 Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}})
-to compile. An example command to compile in the **gpu** partition could be:
+to compile. An example command to compile in the `gpu` partition could be:
 
 {{< highlight batch >}}
-$ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL
+$ srun --partition=gpu --gres=gpu --mem=4gb --ntasks-per-node=2 --nodes=1 --pty $SHELL
 {{< /highlight >}}
 
-The above command will start a shell on a GPU node with 6 cores and 6GB
-of ram in order to compile a GPU job.  The above command could also be
+The above command will start a shell on a GPU node with 2 cores and 4GB
+of RAM in order to compile a GPU job.  The above command could also be
 useful if you want to run a test GPU job interactively.
 
 ### Submitting Jobs
@@ -69,7 +99,7 @@ CUDA and OpenACC submissions require running on GPU nodes.
 #SBATCH --error=/work/[groupname]/[username]/job.%J.err
 #SBATCH --output=/work/[groupname]/[username]/job.%J.out
 
-module load cuda/8.0
+module load cuda
 ./cuda-app.exe
 {{< /highlight >}}
 {{% /panel %}}
@@ -110,4 +140,3 @@ To submit jobs to these resources, add the following to your srun or sbatch comm
 
 **In order to properly utilize pre-emptable resources, your job must be able to support
 some type of checkpoint/resume functionality.**
-