Commit 02bac6ae authored by Adam Caprez's avatar Adam Caprez
Browse files

Update Submitting Jobs section.

parent 2a8466e1
......@@ -4,62 +4,53 @@ description = "How to submit jobs to HCC resources"
weight = "10"
+++
HCC-DOCS : Submitting Jobs
=========================================================
Crane, Sandhills and Tusker are managed by
the <a href="https://slurm.schedmd.com" class="external-link">SLURM</a> resource
manager.  In order to run processing on Crane, Sandhills or Tusker, you
the [SLURM](https://slurm.schedmd.com) resource manager.  
In order to run processing on Crane, Sandhills or Tusker, you
must create a SLURM script that will run your processing. After
submitting the job, SLURM will schedule your processing on an available
worker node.
Before writing a submit file, you may need to [compile your
application](Compiling-Source-Code_332258.html).
Before writing a submit file, you may need to
[compile your application]({{< relref "/guides/running_applications/compiling_source_code" >}}).
- [Ensure proper working directory for job
output](#SubmittingJobs-Ensureproperworkingdirectoryforjoboutput)
- [Creating a SLURM Submit
File](#SubmittingJobs-CreatingaSLURMSubmitFile)
- [Submitting the job](#SubmittingJobs-Submittingthejob)
- [Checking Job Status](#SubmittingJobs-CheckingJobStatus)
- [Checking Job Start](#SubmittingJobs-CheckingJobStart)
- [Next Steps](#SubmittingJobs-NextSteps)
- [Ensure proper working directory for job output](#ensure-proper-working-directory-for-job-output)
- [Creating a SLURM Submit File](#creating-a-slurm-submit-file)
- [Submitting the job](#submitting-the-job)
- [Checking Job Status](#checking-job-status)
- [Checking Job Start](#checking-job-start)
- [Next Steps](#next-steps)
Ensure proper working directory for job output
----------------------------------------------
<span
class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span>
### Ensure proper working directory for job output
{{% notice info %}}
All SLURM job output should be directed to your /work path.
{{% /notice %}}
**manual specification of /work path**
``` syntaxhighlighter-pre
{{% panel theme="info" header="Manual specification of /work path" %}}
{{< highlight bash >}}
$ cd /work/[groupname]/[username]
```
The environment variable $WORK can also be used.
{{< /highlight >}}
{{% /panel %}}
**using environment for /work path**
``` syntaxhighlighter-pre
The environment variable `$WORK` can also be used.
{{% panel theme="info" header="Using environment variable for /work path" %}}
{{< highlight bash >}}
$ cd $WORK
$ pwd
/work/[groupname]/[username]
```
Review how /work differs from /home [here.](Handling-Data_332256.html)
{{< /highlight >}}
{{% /panel %}}
Creating a SLURM Submit File
----------------------------
Review how /work differs from /home [here.]({{< relref "/guides/handling_data/_index.md" >}})
<span
class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span>
### Creating a SLURM Submit File
{{% notice info %}}
The below example is for a serial job. For submitting MPI jobs, please
look at the [MPI Submission Guide.](Submitting-an-MPI-Job_332242.html)
look at the [MPI Submission Guide.]({{< relref "submitting_an_mpi_job" >}})
{{% /notice %}}
A SLURM submit file is broken into 2 sections, the job description and
the processing.  SLURM job description are prepended with `#SBATCH` in
......@@ -67,7 +58,7 @@ the submit file.
**SLURM Submit File**
``` syntaxhighlighter-pre
{{< highlight batch >}}
#!/bin/sh
#SBATCH --time=03:15:00 # Run time in hh:mm:ss
#SBATCH --mem-per-cpu=1024 # Maximum memory required per CPU (in megabytes)
......@@ -79,47 +70,41 @@ module load example/test
hostname
sleep 60
```
- **time**
Maximum walltime the job can run.  After this time has expired, the
job will be stopped.
- **mem-per-cpu**
Memory that is allocated per core for the job.  If you exceed this
memory limit, your job will be stopped.
- **mem**
Specify the real memory required per node in MegaBytes. If you
exceed this limit, your job will be stopped. Note that for you
should ask for less memory than each node actually has. For
instance, Tusker has 1TB, 512GB and 256GB of RAM per node. You may
only request 1000GB of RAM for the 1TB node, 500GB of RAM for the
512GB nodes, and 250GB of RAM for the 256GB nodes. For Crane, the
max is 500GB.
- **job-name
**The name of the job.  Will be reported in the job listing.
- **partition**
The partition the job should run in.  Partitions determine the job's
priority and on what nodes the partition can run on.  See [Available
Partitions on
Sandhills](Available-Partitions-on-Sandhills_332211.html), and
[Available Partitions on Crane and
Tusker](https://hcc-docs.unl.edu/display/HCCDOC/Available+Partitions+on+Crane+and+Tusker) for
a list of possible partitions. 
- **error**
Location of the stderr will be written for the job.  `[groupname]`
and `[username]` should be replaced your group name and username.
 Your username can be retrieved with the command `id -un` and your
group with `id -ng`.
- **output**
Location of the stdout will be written for the job.
More advanced submit commands can be found on the
<a href="https://slurm.schedmd.com/sbatch.html" class="external-link">SLURM Docs</a>.
 You can also find an example of a MPI submission on [Submitting an MPI
Job](Submitting-an-MPI-Job_332242.html).
Submitting the job
------------------
{{< /highlight >}}
- **time**
Maximum walltime the job can run.  After this time has expired, the
job will be stopped.
- **mem-per-cpu**
Memory that is allocated per core for the job.  If you exceed this
memory limit, your job will be stopped.
- **mem**
Specify the real memory required per node in MegaBytes. If you
exceed this limit, your job will be stopped. Note that for you
should ask for less memory than each node actually has. For
instance, Tusker has 1TB, 512GB and 256GB of RAM per node. You may
only request 1000GB of RAM for the 1TB node, 500GB of RAM for the
512GB nodes, and 250GB of RAM for the 256GB nodes. For Crane, the
max is 500GB.
- **job-name**
The name of the job.  Will be reported in the job listing.
- **partition**
The partition the job should run in.  Partitions determine the job's
priority and on what nodes the partition can run on.  See
[Available Partitions on Crane and Tusker]({{< relref "available_partitions_on_crane_and_tusker" >}}) 
for a list of possible partitions. 
- **error**
Location of the stderr will be written for the job.  `[groupname]`
and `[username]` should be replaced your group name and username.
Your username can be retrieved with the command `id -un` and your
group with `id -ng`.
- **output**
Location of the stdout will be written for the job.
More advanced submit commands can be found on the [SLURM Docs](https://slurm.schedmd.com/sbatch.html).
You can also find an example of a MPI submission on [Submitting an MPI Job]({{< relref "submitting_an_mpi_job" >}}).
### Submitting the job
Submitting the SLURM job is done by command `sbatch`.  SLURM will read
the submit file, and schedule the job according to the description in
......@@ -127,64 +112,59 @@ the submit file.
Submitting the job described above is:
**SLURM Submission**
``` syntaxhighlighter-pre
$ sbatch example.slurm
{{% panel theme="info" header="SLURM Submission" %}}
{{< highlight batch >}}
$ sbatch example.slurm
Submitted batch job 24603
```
{{< /highlight >}}
{{% /panel %}}
The job was successfully submitted.
Checking Job Status
-------------------
### Checking Job Status
Job status is found with the command `squeue`.  It will provide
information such as:
- The State of the job: 
- **R** - Running
- **PD** - Pending - <span style="color: rgb(0,0,0);">Job is
awaiting resource allocation.</span>
- <span style="color: rgb(0,0,0);">Additional codes are available
on the
<a href="http://slurm.schedmd.com/squeue.html" class="external-link">squeue</a>
page.</span><span style="color: rgb(0,0,0);">
</span>
- <span style="color: rgb(0,0,0);">Job Name</span>
- <span style="color: rgb(0,0,0);">Run Time</span>
- <span style="color: rgb(0,0,0);">Nodes running the job</span>
- The State of the job: 
- **R** - Running
- **PD** - Pending - Job is awaiting resource allocation.
- Additional codes are available
on the [squeue](http://slurm.schedmd.com/squeue.html)
page.
- Job Name
- Run Time
- Nodes running the job
Checking the status of the job is easiest by filtering by your username,
using the `-u` option to squeue.
``` syntaxhighlighter-pre
{{< highlight batch >}}
$ squeue -u <username>
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
24605 batch hello-wo <username> R 0:56 1 b01
```
{{< /highlight >}}
Additionally, if you want to see the status of a specific partition, for
example if you are part of a
[partition](Available-Partitions-on-Sandhills_332211.html), you can use
the `-p` option to `squeue`:
example if you are part of a [partition]({{< relref "available_partitions_on_crane_and_tusker" >}}),
you can use the `-p` option to `squeue`:
``` syntaxhighlighter-pre
{{< highlight batch >}}
$ squeue -p esquared
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
73435 esquared MyRandom tingting R 10:35:20 1 ri19n10
73436 esquared MyRandom tingting R 10:35:20 1 ri19n12
73735 esquared SW2_driv hroehr R 10:14:11 1 ri20n07
73736 esquared SW2_driv hroehr R 10:14:11 1 ri20n07
```
{{< /highlight >}}
### Checking Job Start
#### Checking Job Start
You may view the start time of your job with the
command `squeue --start`.  The output of the command will show the
expected start time of the jobs.
``` syntaxhighlighter-pre
{{< highlight batch >}}
$ squeue --start --user lypeng
JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON)
5822 batch Starace lypeng PD 2013-06-08T00:05:09 3 (Priority)
......@@ -198,29 +178,22 @@ $ squeue --start --user lypeng
5830 batch Starace lypeng PD 2013-06-08T00:13:09 3 (Priority)
5831 batch Starace lypeng PD 2013-06-08T00:14:09 3 (Priority)
5832 batch Starace lypeng PD N/A 3 (Priority)
```
{{< /highlight >}}
The output shows the expected start time of the jobs, as well as the
reason that the jobs are currently idle (in this case, low priority of
the user due to running numerous jobs already).
 
<span
style="color: rgb(0,0,0);font-size: 20.0px;line-height: 1.5;">Removing
the Job</span>
#### Removing the Job
Removing the job is done with the `scancel` command.  The only argument
to the `scancel` command is the job id.  For the job above, the command
is:
``` syntaxhighlighter-pre
{{< highlight batch >}}
$ scancel 24605
```
Next Steps
----------
 
{{< /highlight >}}
### Next Steps
{{% children %}} 
1. [HCC-DOCS](index.html)
2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html)
3. [HCC Documentation](HCC-Documentation_332651.html)
4. [Submitting Jobs](Submitting-Jobs_332222.html)
<span id="title-text"> HCC-DOCS : Available Partitions on Sandhills </span>
===========================================================================
Created by <span class="author"> Confluence Admin</span>, last modified
by <span class="editor"> Josh Samuelson</span> on Oct 19, 2017
Partitions are used in Sandhills to enforce ownership and priority for
owned resources.  You can view the partitions with the command `sinfo`.
*last generated Wed Oct 24 14:47:57 2018*
<table style="width:100%;">
<colgroup>
<col style="width: 8%" />
<col style="width: 11%" />
<col style="width: 18%" />
<col style="width: 18%" />
<col style="width: 19%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
</colgroup>
<thead>
<tr class="header">
<th>Partition</th>
<th>Owner</th>
<th>Node total (NODExCPU/MEM/FEATURE)</th>
<th>Description</th>
<th>SLURM Specification</th>
<th style="text-align: center;">Max Job Run Time</th>
<th style="text-align: center;">Max CPUs Per User</th>
<th style="text-align: center;">Max Jobs Per User</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>batch</td>
<td>shared</td>
<td>92 total (49x64/192GB/ib,2300Mhz; 41x32/128GB/ib,2000Mhz; 2x48/256GB/none)</td>
<td>(default, no specification)</td>
<td>#SBATCH --partition=batch</td>
<td style="text-align: center;">10-00:00:00</td>
<td style="text-align: center;">2000Mhz:1312 2300Mhz:3136</td>
<td style="text-align: center;">1000</td>
</tr>
<tr class="even">
<td>sabirianov</td>
<td>sabirianov</td>
<td>4 total (4x64/192GB/ib,2300Mhz)</td>
<td>Dedicated resources with ib or 2300Mhz</td>
<td>#SBATCH --partition=sabirianov</td>
<td style="text-align: center;">10-00:00:00</td>
<td style="text-align: center;">2300Mhz:256</td>
<td style="text-align: center;">1000</td>
</tr>
</tbody>
</table>
Specific CPU Frequency Selection
--------------------------------
Sandhills is a hetereogenous cluster in terms of individual node
resources, even when 'owned' by a research group and allocated to a
specific partition with limited access.  So that researchers can
constrain their jobs' deployment to nodes with specific CPU frequencies,
HCC has added a 'feature' on each node that can be used to filter as
part of submission requests.  Currently there are nodes with processors
operating at 2300Mhz and 2000Mhz.  In order to select only nodes of a
specific frequency one must add '–contraint=2000Mhz' to the SLURM
specification (for example).  One can see partitions, node names,
features, and allocation status by using a command such as 'sinfo -N -o
"%P %f %t %N". 
<span style="color: rgb(0,0,0);">Limitations of Jobs</span>
-----------------------------------------------------------
<span style="color: rgb(0,0,0);">There are no special QoS settings for
Sandhills currently. All jobs are subject to the maximum 10 day run time
and 1000 jobs per user.</span>
Guest Partition
---------------
The `guest` partition can be used by users and groups that do not own
resources on Sandhills but still want to run parallel applications.
 Jobs running in the `guest` partition will run on the owned resources
with a high performance interconnect optimized for parallel
applications.  The jobs are preempted when the resources are needed by
the resource owners and are restarted on another node.
Owned Partitions
----------------
Partitions marked as owned by a group means only specific groups are
allowed to submit jobs to that partition.  Groups are manually added to
the list allowed to submit jobs to the partition.  If you are unable to
submit jobs to a partition, and you feel that you should be, please
contact <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a>.
 
1. [HCC-DOCS](index.html)
2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html)
3. [HCC Documentation](HCC-Documentation_332651.html)
4. [Submitting Jobs](Submitting-Jobs_332222.html)
<span id="title-text"> HCC-DOCS : HCC Acknowledgment Credit </span>
===================================================================
Created by <span class="author"> Josh Samuelson</span>, last modified on
Sep 27, 2017
<span
class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span>
+++
title = "HCC Acknowledgment Credit"
description = "Details on the Acknowledgment Credit system."
+++
{{% notice note %}}
To submit an acknowledgement and receive the credit, please use the form
here:
<a href="https://hcc.unl.edu/acknowledgement-submission" class="external-link">https://hcc.unl.edu/acknowledgement-submission</a>
here: https://hcc.unl.edu/acknowledgement-submission.
{{% /notice %}}
### What is HCC Acknowledgment Credit?
......@@ -29,12 +21,10 @@ machine, it may not start immediately like a Priority Access (owned
partition) job will.  This will, however, likely shorten the start time
of the job considerably, depending on job requirements.
Different types of research activities are awarded different amounts of
time.
The following table lists research activities and awarded time:
......@@ -48,22 +38,17 @@ The following table lists research activities and awarded time:
| undergrad thesis | 1 year |
| unfunded grant | 6 months |
Time is awarded evenly for compute and memory resources at a ratio of
1CPU to 4GB of memory.  The QoS remains active until either resource is
exhausted. 
exhausted.
Why this ratio?
**Why this ratio?**
All nodes in the Crane batch partition can meet this CPU to memory
ratio.
Why have this ratio?
**Why have this ratio?**
In short: fairness.  All programs require CPU and memory to complete the
tasks they were written to perform.  Dividing the memory amount by the
......@@ -76,191 +61,148 @@ job.  Similarly, a high memory job may use all the memory on a node but
only use a single CPU.  Using the ratio will result in fair accounting
of CPU and memory resources awarded.
Column description of the hcc-ac utility
| hcc-ac header | Description |
|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Slurm qos | The qos name that must be provided with the Slurm '--qos=ac\_&lt;name&gt;' argument |
| Slurm qos | The qos name that must be provided with the Slurm `--qos=ac_<name>` argument |
| CPUx1 time | Time remaining for a single CPU |
| MEMx4GB time | Time remaining for 4GB of memory |
| per-CPU AvgMEM | The per-CPU average memory size available for the CPU time remaining in the qos.  If CPU time is consumed faster than memory time, this value will increase.  If memory time is consumed faster than CPU time, this value will decrease. |
Example of how to use the awarded time for the 'demo' group.
##### Example of how to use the awarded time for the 'demo' group.
The awarded time is reduced down to 10 minutes to show consumption
changes with differing job resource requirements:
All times are in days-hours:minutes:seconds as used in Slurm's '--time='
argument.
**Default output**
``` syntaxhighlighter-pre
{{% panel theme="info" header="Default output" %}}
{{< highlight batch >}}
[demo01@login.hcc_cluster ~]$ hcc-ac
+-----------+--------------+--------------+----------------+
| Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM |
+-----------+--------------+--------------+----------------+
| ac_demo | 0-00:10:00 | 0-00:10:00 | 4.0GB |
+-----------+--------------+--------------+----------------+
```
{{< /highlight >}}
{{% /panel %}}
Use the Slurm quality of service argument '--qos' to gain access to the
awarded time with increased priority:
**--qos=ac\_demo**
``` syntaxhighlighter-pre
{{% panel theme="info" header="**--qos=ac_demo**" %}}
{{< highlight batch >}}
[demo01@login.hcc_cluster ~]$ srun --qos=ac_demo --ntasks=1 --mem=8g --time=1:00 /bin/sleep 60
```
{{< /highlight >}}
{{% /panel %}}
**\*\*job runs for 60 seconds\*\***
\*\***job runs for 60 seconds**\*\*
**After 60 second job**
``` syntaxhighlighter-pre
{{% panel theme="info" header="**After 60 second job**" %}}
{{< highlight batch >}}
[demo01@login.hcc_cluster ~]$ hcc-ac
+-----------+--------------+--------------+----------------+
| Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM |
+-----------+--------------+--------------+----------------+
| ac_demo | 0-00:09:00 | 0-00:08:00 | 3.556GB |
+-----------+--------------+--------------+----------------+
```
{{< /highlight >}}
{{% /panel %}}
1 CPU minute and 2 4GB memory minutes were consumed by the prior srun
job.
1 CPU minute  ie, (--ntasks=1 --time=1:00)
2 4GB minutes ie, (--ntasks=1 --mem=8G --time=1:00 \~= 1 8GB minute \~=
2 4GB minutes)
The remaining per-CPU average memory can be found with the following
equation:
(memory time remaining) / (cpu time remaining) \* 4
ie, 8 / 9 \* 4 == 3.556
Multiplying the remaining cpu time against the per-CPU average memory
will give the same value as multiplying the remaining memory time
against 4GB:
ie, 9 \* 3.556 \~= 8 \* 4
**--ntasks=4**
``` syntaxhighlighter-pre
{{% panel theme="info" header="**--ntasks=4**" %}}
{{< highlight batch >}}
[demo01@login.hcc_cluster ~]$ srun --qos=ac_demo --ntasks=4 --mem-per-cpu=2G --time=1:00 /bin/sleep 60
```
{{< /highlight >}}
{{% /panel %}}
**\*\*job runs for 60 seconds\*\***
\*\***job runs for 60 seconds**\*\*
**After 60 second job**
``` syntaxhighlighter-pre
{{% panel theme="info" header="**After 60 second job**" %}}