diff --git a/content/guides/running_applications/compiling_source_code.md b/content/guides/running_applications/compiling_source_code/_index.md similarity index 100% rename from content/guides/running_applications/compiling_source_code.md rename to content/guides/running_applications/compiling_source_code/_index.md diff --git a/content/guides/running_applications/compiling_an_openmp_application.md b/content/guides/running_applications/compiling_source_code/compiling_an_openmp_application.md similarity index 100% rename from content/guides/running_applications/compiling_an_openmp_application.md rename to content/guides/running_applications/compiling_source_code/compiling_an_openmp_application.md diff --git a/content/guides/submitting_jobs/_index.md b/content/guides/submitting_jobs/_index.md index 9abc7d071d841f95f0100c61bfb2b241874dfa58..e4629049f977259295e850abccd5564f6bc17c57 100644 --- a/content/guides/submitting_jobs/_index.md +++ b/content/guides/submitting_jobs/_index.md @@ -4,62 +4,53 @@ description = "How to submit jobs to HCC resources" weight = "10" +++ -HCC-DOCS : Submitting Jobs -========================================================= - Crane, Sandhills and Tusker are managed by -the <a href="https://slurm.schedmd.com" class="external-link">SLURM</a> resource -manager. In order to run processing on Crane, Sandhills or Tusker, you +the [SLURM](https://slurm.schedmd.com) resource manager. +In order to run processing on Crane, Sandhills or Tusker, you must create a SLURM script that will run your processing. After submitting the job, SLURM will schedule your processing on an available worker node. -Before writing a submit file, you may need to [compile your -application](Compiling-Source-Code_332258.html). +Before writing a submit file, you may need to +[compile your application]({{< relref "/guides/running_applications/compiling_source_code" >}}). -- [Ensure proper working directory for job - output](#SubmittingJobs-Ensureproperworkingdirectoryforjoboutput) -- [Creating a SLURM Submit - File](#SubmittingJobs-CreatingaSLURMSubmitFile) -- [Submitting the job](#SubmittingJobs-Submittingthejob) -- [Checking Job Status](#SubmittingJobs-CheckingJobStatus) - - [Checking Job Start](#SubmittingJobs-CheckingJobStart) -- [Next Steps](#SubmittingJobs-NextSteps) +- [Ensure proper working directory for job output](#ensure-proper-working-directory-for-job-output) +- [Creating a SLURM Submit File](#creating-a-slurm-submit-file) +- [Submitting the job](#submitting-the-job) +- [Checking Job Status](#checking-job-status) + - [Checking Job Start](#checking-job-start) +- [Next Steps](#next-steps) -Ensure proper working directory for job output ----------------------------------------------- -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> +### Ensure proper working directory for job output +{{% notice info %}} All SLURM job output should be directed to your /work path. +{{% /notice %}} -**manual specification of /work path** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="Manual specification of /work path" %}} +{{< highlight bash >}} $ cd /work/[groupname]/[username] -``` - -The environment variable $WORK can also be used. +{{< /highlight >}} +{{% /panel %}} -**using environment for /work path** - -``` syntaxhighlighter-pre +The environment variable `$WORK` can also be used. +{{% panel theme="info" header="Using environment variable for /work path" %}} +{{< highlight bash >}} $ cd $WORK $ pwd /work/[groupname]/[username] -``` - -Review how /work differs from /home [here.](Handling-Data_332256.html) +{{< /highlight >}} +{{% /panel %}} -Creating a SLURM Submit File ----------------------------- +Review how /work differs from /home [here.]({{< relref "/guides/handling_data/_index.md" >}}) -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> +### Creating a SLURM Submit File +{{% notice info %}} The below example is for a serial job. For submitting MPI jobs, please -look at the [MPI Submission Guide.](Submitting-an-MPI-Job_332242.html) +look at the [MPI Submission Guide.]({{< relref "submitting_an_mpi_job" >}}) +{{% /notice %}} A SLURM submit file is broken into 2 sections, the job description and the processing. SLURM job description are prepended with `#SBATCH` in @@ -67,7 +58,7 @@ the submit file. **SLURM Submit File** -``` syntaxhighlighter-pre +{{< highlight batch >}} #!/bin/sh #SBATCH --time=03:15:00 # Run time in hh:mm:ss #SBATCH --mem-per-cpu=1024 # Maximum memory required per CPU (in megabytes) @@ -79,47 +70,41 @@ module load example/test hostname sleep 60 -``` - -- **time** - Maximum walltime the job can run. After this time has expired, the - job will be stopped. -- **mem-per-cpu** - Memory that is allocated per core for the job. If you exceed this - memory limit, your job will be stopped. -- **mem** - Specify the real memory required per node in MegaBytes. If you - exceed this limit, your job will be stopped. Note that for you - should ask for less memory than each node actually has. For - instance, Tusker has 1TB, 512GB and 256GB of RAM per node. You may - only request 1000GB of RAM for the 1TB node, 500GB of RAM for the - 512GB nodes, and 250GB of RAM for the 256GB nodes. For Crane, the - max is 500GB. -- **job-name - **The name of the job. Will be reported in the job listing. -- **partition** - The partition the job should run in. Partitions determine the job's - priority and on what nodes the partition can run on. See [Available - Partitions on - Sandhills](Available-Partitions-on-Sandhills_332211.html), and - [Available Partitions on Crane and - Tusker](https://hcc-docs.unl.edu/display/HCCDOC/Available+Partitions+on+Crane+and+Tusker) for - a list of possible partitions. -- **error** - Location of the stderr will be written for the job. `[groupname]` - and `[username]` should be replaced your group name and username. - Your username can be retrieved with the command `id -un` and your - group with `id -ng`. -- **output** - Location of the stdout will be written for the job. - -More advanced submit commands can be found on the -<a href="https://slurm.schedmd.com/sbatch.html" class="external-link">SLURM Docs</a>. - You can also find an example of a MPI submission on [Submitting an MPI -Job](Submitting-an-MPI-Job_332242.html). - -Submitting the job ------------------- +{{< /highlight >}} + +- **time** + Maximum walltime the job can run. After this time has expired, the + job will be stopped. +- **mem-per-cpu** + Memory that is allocated per core for the job. If you exceed this + memory limit, your job will be stopped. +- **mem** + Specify the real memory required per node in MegaBytes. If you + exceed this limit, your job will be stopped. Note that for you + should ask for less memory than each node actually has. For + instance, Tusker has 1TB, 512GB and 256GB of RAM per node. You may + only request 1000GB of RAM for the 1TB node, 500GB of RAM for the + 512GB nodes, and 250GB of RAM for the 256GB nodes. For Crane, the + max is 500GB. +- **job-name** + The name of the job. Will be reported in the job listing. +- **partition** + The partition the job should run in. Partitions determine the job's + priority and on what nodes the partition can run on. See + [Available Partitions on Crane and Tusker]({{< relref "available_partitions_on_crane_and_tusker" >}}) + for a list of possible partitions. +- **error** + Location of the stderr will be written for the job. `[groupname]` + and `[username]` should be replaced your group name and username. + Your username can be retrieved with the command `id -un` and your + group with `id -ng`. +- **output** + Location of the stdout will be written for the job. + +More advanced submit commands can be found on the [SLURM Docs](https://slurm.schedmd.com/sbatch.html). +You can also find an example of a MPI submission on [Submitting an MPI Job]({{< relref "submitting_an_mpi_job" >}}). + +### Submitting the job Submitting the SLURM job is done by command `sbatch`. SLURM will read the submit file, and schedule the job according to the description in @@ -127,64 +112,59 @@ the submit file. Submitting the job described above is: -**SLURM Submission** - -``` syntaxhighlighter-pre -$ sbatch example.slurm +{{% panel theme="info" header="SLURM Submission" %}} +{{< highlight batch >}} +$ sbatch example.slurm Submitted batch job 24603 -``` +{{< /highlight >}} +{{% /panel %}} The job was successfully submitted. -Checking Job Status -------------------- +### Checking Job Status Job status is found with the command `squeue`. It will provide information such as: -- The State of the job: - - **R** - Running - - **PD** - Pending - <span style="color: rgb(0,0,0);">Job is - awaiting resource allocation.</span> - - <span style="color: rgb(0,0,0);">Additional codes are available - on the - <a href="http://slurm.schedmd.com/squeue.html" class="external-link">squeue</a> - page.</span><span style="color: rgb(0,0,0);"> - </span> -- <span style="color: rgb(0,0,0);">Job Name</span> -- <span style="color: rgb(0,0,0);">Run Time</span> -- <span style="color: rgb(0,0,0);">Nodes running the job</span> +- The State of the job: + - **R** - Running + - **PD** - Pending - Job is awaiting resource allocation. + - Additional codes are available + on the [squeue](http://slurm.schedmd.com/squeue.html) + page. +- Job Name +- Run Time +- Nodes running the job Checking the status of the job is easiest by filtering by your username, using the `-u` option to squeue. -``` syntaxhighlighter-pre +{{< highlight batch >}} $ squeue -u <username> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 24605 batch hello-wo <username> R 0:56 1 b01 -``` +{{< /highlight >}} Additionally, if you want to see the status of a specific partition, for -example if you are part of a -[partition](Available-Partitions-on-Sandhills_332211.html), you can use -the `-p` option to `squeue`: +example if you are part of a [partition]({{< relref "available_partitions_on_crane_and_tusker" >}}), +you can use the `-p` option to `squeue`: -``` syntaxhighlighter-pre +{{< highlight batch >}} $ squeue -p esquared JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 73435 esquared MyRandom tingting R 10:35:20 1 ri19n10 73436 esquared MyRandom tingting R 10:35:20 1 ri19n12 73735 esquared SW2_driv hroehr R 10:14:11 1 ri20n07 73736 esquared SW2_driv hroehr R 10:14:11 1 ri20n07 -``` +{{< /highlight >}} -### Checking Job Start +#### Checking Job Start You may view the start time of your job with the command `squeue --start`. The output of the command will show the expected start time of the jobs. -``` syntaxhighlighter-pre +{{< highlight batch >}} $ squeue --start --user lypeng JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON) 5822 batch Starace lypeng PD 2013-06-08T00:05:09 3 (Priority) @@ -198,29 +178,22 @@ $ squeue --start --user lypeng 5830 batch Starace lypeng PD 2013-06-08T00:13:09 3 (Priority) 5831 batch Starace lypeng PD 2013-06-08T00:14:09 3 (Priority) 5832 batch Starace lypeng PD N/A 3 (Priority) -``` +{{< /highlight >}} The output shows the expected start time of the jobs, as well as the reason that the jobs are currently idle (in this case, low priority of the user due to running numerous jobs already). - - -<span -style="color: rgb(0,0,0);font-size: 20.0px;line-height: 1.5;">Removing -the Job</span> +#### Removing the Job Removing the job is done with the `scancel` command. The only argument to the `scancel` command is the job id. For the job above, the command is: -``` syntaxhighlighter-pre +{{< highlight batch >}} $ scancel 24605 -``` - -Next Steps ----------- - - +{{< /highlight >}} +### Next Steps +{{% children %}} diff --git a/content/guides/submitting_jobs/available_partitions_on_crane_and_tusker.md b/content/guides/submitting_jobs/available_partitions_on_crane_and_tusker.md index 05bcf5068601bf5942d6b6591ac2fe0f7ea09bc4..3a9c67ca0c1a46670ec87b0649a302279d269126 100644 --- a/content/guides/submitting_jobs/available_partitions_on_crane_and_tusker.md +++ b/content/guides/submitting_jobs/available_partitions_on_crane_and_tusker.md @@ -1,576 +1,77 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Available Partitions on Crane and Tusker </span> -================================================================================== - -Created by <span class="author"> John Thiltges</span>, last modified by -<span class="editor"> Adam Caprez</span> on Sep 14, 2018 ++++ +title = "Available partitions on Tusker and Crane" +description = "Listing of partitions on Tusker and Crane." ++++ Partitions are used in Crane and Tusker to distinguish different resources. You can view the partitions with the command `sinfo`. -Crane: ------- - -*last generated Wed Oct 24 14:47:56 2018* +### Crane: -<table style="width:100%;"> -<colgroup> -<col style="width: 8%" /> -<col style="width: 11%" /> -<col style="width: 18%" /> -<col style="width: 18%" /> -<col style="width: 19%" /> -<col style="width: 7%" /> -<col style="width: 7%" /> -<col style="width: 7%" /> -</colgroup> -<thead> -<tr class="header"> -<th>Partition</th> -<th>Owner</th> -<th>Node total (NODExCPU/MEM/FEATURE)</th> -<th>Description</th> -<th>SLURM Specification</th> -<th style="text-align: center;">Max Job Run Time</th> -<th style="text-align: center;">Max CPUs Per User</th> -<th style="text-align: center;">Max Jobs Per User</th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td>batch</td> -<td>shared</td> -<td>404 total (6x36/256GB/opa; 382x16/64GB/ib; 16x36/512GB/opa)</td> -<td>(default, no specification)</td> -<td>#SBATCH --partition=batch</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:2000 opa:576</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>guest</td> -<td>shared</td> -<td>151 total (73x36/256GB/opa; 20x36/512GB/opa; 58x16/64GB/ib)</td> -<td>Preemtable partition with ib or opa</td> -<td>#SBATCH --partition=guest</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:928 opa:2000</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>alexandrov</td> -<td>alexandrov</td> -<td>32 total (32x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=alexandrov</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:512</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>tmp_anvil</td> -<td>shared</td> -<td>21 total (21x20/256GB/none)</td> -<td>Shared resources</td> -<td>#SBATCH --partition=tmp_anvil</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">420</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>gfcluster</td> -<td>gladich francisco</td> -<td>20 total (10x36/256GB/opa; 10x36/512GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=gfcluster</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:360</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>zeng</td> -<td>zeng</td> -<td>16 total (14x36/256GB/opa; 2x36/512GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=zeng</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:504</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>tsymbal</td> -<td>tsymbal</td> -<td>14 total (14x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=tsymbal</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:504</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>benson</td> -<td>benson</td> -<td>12 total (12x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=benson</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:432</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>gpu</td> -<td>shared</td> -<td>11 total (2x24/128GB/ib,gpu_k20; 1x24/120GB/ib,gpu_k20; 1x8/64GB/gpu_k40; 5x16/64GB/gpu_k40; 2x8/64GB/opa,gpu_p100)</td> -<td>Shared resources with gpu_p100 or gpu_k20 or ib or gpu_k40 or opa</td> -<td>#SBATCH --partition=gpu</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:48 opa:16</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>jclarke</td> -<td>jclarke schnablelab</td> -<td>10 total (4x36/256GB/opa; 6x36/512GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=jclarke</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:216</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>gladich</td> -<td>gladich</td> -<td>10 total (10x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=gladich</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:360</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>francisco</td> -<td>francisco</td> -<td>10 total (10x36/512GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=francisco</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:360</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>ahouston</td> -<td>ahouston</td> -<td>9 total (2x16/64GB/ib; 7x36/256GB/opa)</td> -<td>Dedicated resources with ib or opa</td> -<td>#SBATCH --partition=ahouston</td> -<td style="text-align: center;">28-00:00:00</td> -<td style="text-align: center;">ib:32 opa:252</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>scott</td> -<td>scott vinod samal</td> -<td>6 total (1x16/15GB/gpu_1060; 3x16/48GB/gpu_k20; 1x8/95GB/gpu_p100; 1x20/256GB/gpu_titanx)</td> -<td>Dedicated resources with gpu_p100 or gpu_k20 or gpu_1060 or gpu_titanx</td> -<td>#SBATCH --partition=scott</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">48</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>larios</td> -<td>larios</td> -<td>6 total (4x16/64GB/ib; 2x36/256GB/opa)</td> -<td>Dedicated resources with ib or opa</td> -<td>#SBATCH --partition=larios</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:64 opa:72</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>qinghuilab</td> -<td>qinghuilab</td> -<td>5 total (5x8/192GB/gpu_p100)</td> -<td>Dedicated resources with gpu_p100</td> -<td>#SBATCH --partition=qinghuilab</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">40</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>choueiry</td> -<td>choueiry</td> -<td>4 total (4x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=choueiry</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:144</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>sabirianov</td> -<td>sabirianov</td> -<td>4 total (4x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=sabirianov</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:64</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>jyanglab</td> -<td>jyanglab</td> -<td>4 total (4x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=jyanglab</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:64</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>jaturner</td> -<td>jaturner</td> -<td>4 total (4x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=jaturner</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:64</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>riethoven</td> -<td>riethoven</td> -<td>3 total (3x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=riethoven</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:108</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>bigweather</td> -<td>bigweather</td> -<td>3 total (3x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=bigweather</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:48</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>starace</td> -<td>starace</td> -<td>2 total (1x36/512GB/opa; 1x16/64GB/ib)</td> -<td>Dedicated resources with ib or opa</td> -<td>#SBATCH --partition=starace</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:16 opa:36</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>stats</td> -<td>qizhangstat yzstat</td> -<td>2 total (2x32/256GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=stats</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:64</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>dwyer</td> -<td>dwyer</td> -<td>2 total (2x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=dwyer</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:72</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>esquared</td> -<td>esquared dwyer vuran cohen nimbus</td> -<td>2 total (2x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=esquared</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:72</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>unsmgenomics</td> -<td>unsmgenomics</td> -<td>1 total (1x20/256GB/none)</td> -<td>Dedicated resources</td> -<td>#SBATCH --partition=unsmgenomics</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">20</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>statsgeneral</td> -<td>statsgeneral</td> -<td>1 total (1x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=statsgeneral</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:16</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>abg</td> -<td>anscgenomics rmlewis</td> -<td>1 total (1x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=abg</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:16</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>juancui</td> -<td>juancui</td> -<td>1 total (1x36/512GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=juancui</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:36</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>gpu_v100</td> -<td></td> -<td>1 total (1x36/384GB/gpu_v100)</td> -<td>Dedicated resources with gpu_v100</td> -<td>#SBATCH --partition=gpu_v100</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">36</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>guda</td> -<td>unmc_cbsb unmc_gudalab</td> -<td>1 total (1x36/256GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=guda</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:36</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>waterforfood</td> -<td>waterforfood</td> -<td>1 total (1x36/512GB/opa)</td> -<td>Dedicated resources with opa</td> -<td>#SBATCH --partition=waterforfood</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">opa:36</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>npod</td> -<td>chunglab juancui ramertait samodha unmc_ngs wtp zempleni</td> -<td>1 total (1x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=npod</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:16</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>thinklab</td> -<td>thinklab</td> -<td>1 total (1x16/64GB/ib)</td> -<td>Dedicated resources with ib</td> -<td>#SBATCH --partition=thinklab</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">ib:16</td> -<td style="text-align: center;">1000</td> -</tr> -</tbody> -</table> +{{< readfile file="/static/html/crane.html" >}} - +### Tusker: -Tusker: -------- - -*last generated Wed Oct 24 14:47:57 2018* - -<table style="width:100%;"> -<colgroup> -<col style="width: 8%" /> -<col style="width: 11%" /> -<col style="width: 18%" /> -<col style="width: 18%" /> -<col style="width: 19%" /> -<col style="width: 7%" /> -<col style="width: 7%" /> -<col style="width: 7%" /> -</colgroup> -<thead> -<tr class="header"> -<th>Partition</th> -<th>Owner</th> -<th>Node total (NODExCPU/MEM/FEATURE)</th> -<th>Description</th> -<th>SLURM Specification</th> -<th style="text-align: center;">Max Job Run Time</th> -<th style="text-align: center;">Max CPUs Per User</th> -<th style="text-align: center;">Max Jobs Per User</th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td>batch</td> -<td>shared</td> -<td>38 total (38x64/256GB/none)</td> -<td>(default, no specification)</td> -<td>#SBATCH --partition=batch</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">2000</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>highmem</td> -<td>shared</td> -<td>4 total (2x64/1020GB/none; 2x64/512GB/none)</td> -<td>Shared resources</td> -<td>#SBATCH --partition=highmem</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">128</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="odd"> -<td>guda</td> -<td>unmc_cbsb unmc_gudalab</td> -<td>1 total (1x64/256GB/none)</td> -<td>Dedicated resources</td> -<td>#SBATCH --partition=guda</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">64</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>reid</td> -<td>reid</td> -<td>1 total (1x64/256GB/none)</td> -<td>Dedicated resources</td> -<td>#SBATCH --partition=reid</td> -<td style="text-align: center;">7-00:00:00</td> -<td style="text-align: center;">64</td> -<td style="text-align: center;">1000</td> -</tr> -</tbody> -</table> +{{< readfile file="/static/html/tusker.html" >}} Two nodes have 512GB of memory instead of 256GB (Max Request = 500GB), and two have 1024GB of memory (Max Request = 1000GB). -Priority for short jobs ------------------------ +#### Priority for short jobs To run short jobs for testing and development work, a job can specify a different quality of service (QoS). The *short* QoS increases a jobs priority so it will run as soon as possible. -| <span style="color: rgb(0,51,102);">SLURM Specification</span> | -|----------------------------------------------------------------| -| `#SBATCH --qos=short` | - -Limits per user for 'short' QoS +| SLURM Specification | +|----------------------- | +| `#SBATCH --qos=short` | -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> +{{% panel theme="warning" header="Limits per user for 'short' QoS" %}} +- 6 hour job run time +- 2 jobs of 16 CPUs or fewer +- No more than 256 CPUs in use for *short* jobs from all users +{{% /panel %}} -- 6 hour job run time -- 2 jobs of 16 CPUs or fewer -- No more than 256 CPUs in use for *short* jobs from all users -<span style="color: rgb(0,0,0);">Limitations of Jobs</span> ------------------------------------------------------------ +### Limitations of Jobs -<span style="color: rgb(0,0,0);">Overall limitations of maximum job wall -time. CPUs, etc. are set for all jobs with the default setting (when the -"–qos=" section is omitted) and "short" jobs (described as above) on -Tusker and Crane. The limitations are shown in the following -form.</span> +Overall limitations of maximum job wall time. CPUs, etc. are set for +all jobs with the default setting (when thea "–qos=" section is omitted) +and "short" jobs (described as above) on Tusker and Crane. +The limitations are shown in the following form. -<span style="color: rgb(0,0,0);"> </span> +| | SLURM Specification | Max Job Run Time | Max CPUs per User | Max Jobs per User | +| ------- | -------------------- | ---------------- | ----------------- | ----------------- | +| Default | Leave blank | 7 days | 2000 | 1000 | +| Short | #SBATCH --qos=short | 6 hours | 16 | 2 | -<table> -<thead> -<tr class="header"> -<th><div class="tablesorter-header-inner"> -<span style="color: rgb(0,0,0);"> </span> -</div></th> -<th><div class="tablesorter-header-inner"> -<span style="color: rgb(0,0,0);">SLURM Specification</span> -</div></th> -<th><div class="tablesorter-header-inner"> -<span style="color: rgb(0,0,0);">Max Job Run Time</span> -</div></th> -<th><div class="tablesorter-header-inner"> -<span style="color: rgb(0,0,0);">Max CPUs per User</span> -</div></th> -<th><div class="tablesorter-header-inner"> -<span style="color: rgb(0,0,0);">Max Jobs per User</span> -</div></th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td><span style="color: rgb(0,0,0);">Default</span></td> -<td><span style="color: rgb(0,0,0);">Leave blank</span></td> -<td><span style="color: rgb(0,0,0);">7 days</span></td> -<td><span style="color: rgb(0,0,0);">2000</span></td> -<td><span style="color: rgb(0,0,0);">1000</span></td> -</tr> -<tr class="even"> -<td><span style="color: rgb(0,0,0);">Short</span></td> -<td><span style="color: rgb(0,0,0);"><code>#SBATCH --qos=short</code></span></td> -<td><span style="color: rgb(0,0,0);">6 hours</span></td> -<td><span style="color: rgb(0,0,0);">16</span></td> -<td><span style="color: rgb(0,0,0);">2</span></td> -</tr> -</tbody> -</table> -<span style="color: rgb(0,0,0);"> </span> - -<span style="color: rgb(0,0,0);">Please also note that the memory and +Please also note that the memory and local hard drive limits are subject to the physical limitations of the -nodes, described in the resources capabilities section of [<span -style="color: rgb(0,0,0);">HCC -Documentation</span>](https://hcc-docs.unl.edu/display/HCCDOC/HCC+Documentation) and -the partition sections above.</span> +nodes, described in the resources capabilities section of the +[HCC Documentation]({{< relref "/#resource-capabilities" >}}) +and the partition sections above. -Owned Partitions ----------------- +### Owned Partitions Partitions marked as owned by a group means only specific groups are allowed to submit jobs to that partition. Groups are manually added to the list allowed to submit jobs to the partition. If you are unable to submit jobs to a partition, and you feel that you should be, please -contact <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a>. +contact {{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu). -Guest Partition ---------------- +### Guest Partition The `guest` partition can be used by users and groups that do not own dedicated resources on Crane. Jobs running in the `guest` partition -will run on the owned resources with Intel OPA interconnect. The jobs +will run on the owned resources with Intel OPA interconnect. The jobs are preempted when the resources are needed by the resource owners and are restarted on another node. -tmp\_anvil Partition --------------------- +### tmp_anvil Partition -We have put 27 Anvil nodes which are not running Openstack in this +We have put Anvil nodes which are not running Openstack in this partition. They have Intel Xeon E5-2650 v3 2.30GHz 2 CPU/20 cores and 256GB memory per node. However, they don't have Infiniband or OPA interconnect. They are suitable for serial or single node parallel jobs. @@ -578,8 +79,7 @@ The nodes in this partition are subjected to be drained and move to our Openstack cloud when more cloud resources are needed without notice in advance. -Use of Infiniband or OPA ------------------------- +### Use of Infiniband or OPA Crane nodes use either Infiniband or Intel Omni-Path interconnects in the batch partition. Most users don't need to worry about which one to @@ -588,34 +88,14 @@ scheduler. However, if the user wants to use one of the interconnects exclusively, the SLURM constraint keyword is available. Here are the examples: -<table> -<thead> -<tr class="header"> -<th><span style="color: rgb(0,51,102);">SLURM Specification Omni-Path</span></th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td><p>#SBATCH --constraint=opa</p> -<code></code></td> -</tr> -</tbody> -</table> - - - -<table> -<thead> -<tr class="header"> -<th><span style="color: rgb(0,51,102);">SLURM Specification Infiniband</span></th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td><p>#SBATCH --constraint=ib</p> -<code></code></td> -</tr> -</tbody> -</table> - - +{{% panel theme="info" header="SLURM Specification: Omni-Path" %}} +{{< highlight bash >}} +#SBATCH --constraint=opa +{{< /highlight >}} +{{% /panel %}} + +{{% panel theme="info" header="SLURM Specification: Infiniband" %}} +{{< highlight bash >}} +#SBATCH --constraint=ib +{{< /highlight >}} +{{% /panel %}} diff --git a/content/guides/submitting_jobs/available_partitions_on_sandhills.md b/content/guides/submitting_jobs/available_partitions_on_sandhills.md deleted file mode 100644 index c8193bd9afeeca6be454e41b1d36ad7cd915e9ff..0000000000000000000000000000000000000000 --- a/content/guides/submitting_jobs/available_partitions_on_sandhills.md +++ /dev/null @@ -1,107 +0,0 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Available Partitions on Sandhills </span> -=========================================================================== - -Created by <span class="author"> Confluence Admin</span>, last modified -by <span class="editor"> Josh Samuelson</span> on Oct 19, 2017 - -Partitions are used in Sandhills to enforce ownership and priority for -owned resources. You can view the partitions with the command `sinfo`. - -*last generated Wed Oct 24 14:47:57 2018* - -<table style="width:100%;"> -<colgroup> -<col style="width: 8%" /> -<col style="width: 11%" /> -<col style="width: 18%" /> -<col style="width: 18%" /> -<col style="width: 19%" /> -<col style="width: 7%" /> -<col style="width: 7%" /> -<col style="width: 7%" /> -</colgroup> -<thead> -<tr class="header"> -<th>Partition</th> -<th>Owner</th> -<th>Node total (NODExCPU/MEM/FEATURE)</th> -<th>Description</th> -<th>SLURM Specification</th> -<th style="text-align: center;">Max Job Run Time</th> -<th style="text-align: center;">Max CPUs Per User</th> -<th style="text-align: center;">Max Jobs Per User</th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td>batch</td> -<td>shared</td> -<td>92 total (49x64/192GB/ib,2300Mhz; 41x32/128GB/ib,2000Mhz; 2x48/256GB/none)</td> -<td>(default, no specification)</td> -<td>#SBATCH --partition=batch</td> -<td style="text-align: center;">10-00:00:00</td> -<td style="text-align: center;">2000Mhz:1312 2300Mhz:3136</td> -<td style="text-align: center;">1000</td> -</tr> -<tr class="even"> -<td>sabirianov</td> -<td>sabirianov</td> -<td>4 total (4x64/192GB/ib,2300Mhz)</td> -<td>Dedicated resources with ib or 2300Mhz</td> -<td>#SBATCH --partition=sabirianov</td> -<td style="text-align: center;">10-00:00:00</td> -<td style="text-align: center;">2300Mhz:256</td> -<td style="text-align: center;">1000</td> -</tr> -</tbody> -</table> - -Specific CPU Frequency Selection --------------------------------- - -Sandhills is a hetereogenous cluster in terms of individual node -resources, even when 'owned' by a research group and allocated to a -specific partition with limited access. So that researchers can -constrain their jobs' deployment to nodes with specific CPU frequencies, -HCC has added a 'feature' on each node that can be used to filter as -part of submission requests. Currently there are nodes with processors -operating at 2300Mhz and 2000Mhz. In order to select only nodes of a -specific frequency one must add '–contraint=2000Mhz' to the SLURM -specification (for example). One can see partitions, node names, -features, and allocation status by using a command such as 'sinfo -N -o -"%P %f %t %N". - -<span style="color: rgb(0,0,0);">Limitations of Jobs</span> ------------------------------------------------------------ - -<span style="color: rgb(0,0,0);">There are no special QoS settings for -Sandhills currently. All jobs are subject to the maximum 10 day run time -and 1000 jobs per user.</span> - -Guest Partition ---------------- - -The `guest` partition can be used by users and groups that do not own -resources on Sandhills but still want to run parallel applications. - Jobs running in the `guest` partition will run on the owned resources -with a high performance interconnect optimized for parallel -applications. The jobs are preempted when the resources are needed by -the resource owners and are restarted on another node. - -Owned Partitions ----------------- - -Partitions marked as owned by a group means only specific groups are -allowed to submit jobs to that partition. Groups are manually added to -the list allowed to submit jobs to the partition. If you are unable to -submit jobs to a partition, and you feel that you should be, please -contact <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a>. - - - - diff --git a/content/guides/submitting_jobs/hcc_acknowledgment_credit.md b/content/guides/submitting_jobs/hcc_acknowledgment_credit.md index 0f5b19ec923f69a5853991c130f504f61ee1949c..d04c68342486a900504f7988a4cf4f015039ae17 100644 --- a/content/guides/submitting_jobs/hcc_acknowledgment_credit.md +++ b/content/guides/submitting_jobs/hcc_acknowledgment_credit.md @@ -1,20 +1,12 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : HCC Acknowledgment Credit </span> -=================================================================== - -Created by <span class="author"> Josh Samuelson</span>, last modified on -Sep 27, 2017 - -<span -class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span> ++++ +title = "HCC Acknowledgment Credit" +description = "Details on the Acknowledgment Credit system." ++++ +{{% notice note %}} To submit an acknowledgement and receive the credit, please use the form -here: -<a href="https://hcc.unl.edu/acknowledgement-submission" class="external-link">https://hcc.unl.edu/acknowledgement-submission</a> +here: https://hcc.unl.edu/acknowledgement-submission. +{{% /notice %}} ### What is HCC Acknowledgment Credit? @@ -29,12 +21,10 @@ machine, it may not start immediately like a Priority Access (owned partition) job will. This will, however, likely shorten the start time of the job considerably, depending on job requirements. - Different types of research activities are awarded different amounts of time. - The following table lists research activities and awarded time: @@ -48,22 +38,17 @@ The following table lists research activities and awarded time: | undergrad thesis | 1 year | | unfunded grant | 6 months | - Time is awarded evenly for compute and memory resources at a ratio of 1CPU to 4GB of memory. The QoS remains active until either resource is -exhausted. - - +exhausted. -Why this ratio? +**Why this ratio?** All nodes in the Crane batch partition can meet this CPU to memory ratio. - - -Why have this ratio? +**Why have this ratio?** In short: fairness. All programs require CPU and memory to complete the tasks they were written to perform. Dividing the memory amount by the @@ -76,191 +61,148 @@ job. Similarly, a high memory job may use all the memory on a node but only use a single CPU. Using the ratio will result in fair accounting of CPU and memory resources awarded. - Column description of the hcc-ac utility | hcc-ac header | Description | |----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Slurm qos | The qos name that must be provided with the Slurm '--qos=ac\_<name>' argument | +| Slurm qos | The qos name that must be provided with the Slurm `--qos=ac_<name>` argument | | CPUx1 time | Time remaining for a single CPU | | MEMx4GB time | Time remaining for 4GB of memory | | per-CPU AvgMEM | The per-CPU average memory size available for the CPU time remaining in the qos. If CPU time is consumed faster than memory time, this value will increase. If memory time is consumed faster than CPU time, this value will decrease. | - -Example of how to use the awarded time for the 'demo' group. - - +##### Example of how to use the awarded time for the 'demo' group. The awarded time is reduced down to 10 minutes to show consumption changes with differing job resource requirements: - - All times are in days-hours:minutes:seconds as used in Slurm's '--time=' argument. - - -**Default output** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="Default output" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ hcc-ac +-----------+--------------+--------------+----------------+ | Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM | +-----------+--------------+--------------+----------------+ | ac_demo | 0-00:10:00 | 0-00:10:00 | 4.0GB | +-----------+--------------+--------------+----------------+ -``` +{{< /highlight >}} +{{% /panel %}} Use the Slurm quality of service argument '--qos' to gain access to the awarded time with increased priority: - - -**--qos=ac\_demo** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="**--qos=ac_demo**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ srun --qos=ac_demo --ntasks=1 --mem=8g --time=1:00 /bin/sleep 60 -``` +{{< /highlight >}} +{{% /panel %}} -**\*\*job runs for 60 seconds\*\*** +\*\***job runs for 60 seconds**\*\* -**After 60 second job** -``` syntaxhighlighter-pre +{{% panel theme="info" header="**After 60 second job**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ hcc-ac +-----------+--------------+--------------+----------------+ | Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM | +-----------+--------------+--------------+----------------+ | ac_demo | 0-00:09:00 | 0-00:08:00 | 3.556GB | +-----------+--------------+--------------+----------------+ -``` +{{< /highlight >}} +{{% /panel %}} 1 CPU minute and 2 4GB memory minutes were consumed by the prior srun job. - - 1 CPU minute ie, (--ntasks=1 --time=1:00) 2 4GB minutes ie, (--ntasks=1 --mem=8G --time=1:00 \~= 1 8GB minute \~= 2 4GB minutes) - - The remaining per-CPU average memory can be found with the following equation: - - (memory time remaining) / (cpu time remaining) \* 4 - - ie, 8 / 9 \* 4 == 3.556 - - Multiplying the remaining cpu time against the per-CPU average memory will give the same value as multiplying the remaining memory time against 4GB: - - ie, 9 \* 3.556 \~= 8 \* 4 - - -**--ntasks=4** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="**--ntasks=4**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ srun --qos=ac_demo --ntasks=4 --mem-per-cpu=2G --time=1:00 /bin/sleep 60 -``` +{{< /highlight >}} +{{% /panel %}} -**\*\*job runs for 60 seconds\*\*** +\*\***job runs for 60 seconds**\*\* - - -**After 60 second job** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="**After 60 second job**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ hcc-ac +-----------+--------------+--------------+----------------+ | Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM | +-----------+--------------+--------------+----------------+ | ac_demo | 0-00:05:00 | 0-00:06:00 | 4.8GB | +-----------+--------------+--------------+----------------+ -``` - - +{{< /highlight >}} +{{% /panel %}} 4 CPU minutes and 2 4GB minutes were consumed by the prior srun job. - - 4 CPU minutes (--ntasks=4 --time=1:00) 2 4GB minutes (--ntasks=4 --mem-per-cpu=2G --time=1:00 \~= 1 8GB minute \~= 2 4GB minutes) - - 6 / 5 \* 4 == 4.8 - - -**Insufficient time** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="**Insufficient Time**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ srun --qos=ac_demo --ntasks=5 --mem-per-cpu=5000M --time=1:00 /bin/sleep 60 srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) -``` - - +{{< /highlight >}} +{{% /panel %}} An example of a job requesting more resources than what remains available in the qos. - - -**corrected memory requirement** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="**Corrected Memory Requirement**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ srun --qos=ac_demo --ntasks=5 --mem-per-cpu=4800M --time=1:00 /bin/sleep 60 -``` +{{< /highlight >}} +{{% /panel %}} -**\*\*job runs for 60 seconds\*\*** - - +\*\***job runs for 60 seconds**\*\* -**exhausted qos** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="**Exhausted QoS**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ hcc-ac +-----------+--------------+--------------+----------------+ | Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM | +-----------+--------------+--------------+----------------+ | ac_demo | exhausted | exhausted | 0.0GB | +-----------+--------------+--------------+----------------+ -``` +{{< /highlight >}} +{{% /panel %}} All remaining time was used. Any further submissions to the qos will be **denied at submission time**. - - All of the above **srun** arguments work the same with **sbatch** within the submit file header. - - -**submit file example** -``` syntaxhighlighter-pre +{{% panel theme="info" header="**Submit File Example**" %}} +{{< highlight batch >}} [demo01@login.hcc_cluster ~]$ cat submit_test.slurm -#!/bin/bash +#!/bin/batch #SBATCH --ntasks=4 #SBATCH --qos=ac_demo #SBATCH --ntasks=5 @@ -269,9 +211,8 @@ the submit file header. /bin/sleep 60 [demo01@login.hcc_cluster ~]$ sbatch ./submit_test.slurm -``` - - +{{< /highlight >}} +{{% /panel %}} CPU and memory time in the qos are only consumed when jobs run against the qos. Therefore it is possible for more jobs to be submitted @@ -280,31 +221,22 @@ exhausted all pending jobs attempting to use it will remain in pending state until more time is added to the qos or the job is modified to not consider the qos as a job requirement. - - HCC will run a script periodically (hourly) to scan for jobs pending in this state and modify them to utilize the cluster's default qos. - - -Sizing what can fit in the qos's time limits using Slurm's '--test-only' +Sizing what can fit in the qos's time limits using Slurm's `--test-only` argument. - - -You can attempt different combinations of --nodes, --ntasks, ---ntasks-per-node, --mem-per-cpu= and --mem utilizing the '--test-only' +You can attempt different combinations of `--nodes`, `--ntasks`, +`--ntasks-per-node`, `--mem-per-cpu` and `--mem` utilizing the `--test-only` argument to size the job against what time remains in the qos. - - For example, with the same 10 minute limit: - -**--test-only job to see if it fits within qos time limits** +{{% panel theme="info" header="**--test-only job to see if it fits within qos time limits**" %}} +{{< highlight batch >}} -``` syntaxhighlighter-pre [demo01@login.hcc_cluster ~]$ hcc-ac +-----------+--------------+--------------+----------------+ | Slurm qos | CPUx1 time | MEMx4GB time | per-CPU AvgMEM | @@ -334,6 +266,5 @@ allocation failure: Job violates accounting/QOS policy (job submit limit, user's [demo01@login.hcc_cluster ~]$ srun --test-only --qos=ac_demo --ntasks=1 --time=3:00 --mem-per-cpu=12G srun: Job <number> to start at YYYY-MM-DDTHH:MM:SS using 1 processors on compute_node -``` - - +{{< /highlight >}} +{{% /panel %}} diff --git a/content/guides/submitting_jobs/job_dependencies.md b/content/guides/submitting_jobs/job_dependencies.md index fa431e972282707f867fa1cc684c64f2db851883..488c03f95e3f7dd0f2cfe677d3f4c77f66adf531 100644 --- a/content/guides/submitting_jobs/job_dependencies.md +++ b/content/guides/submitting_jobs/job_dependencies.md @@ -1,13 +1,7 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Job Dependencies </span> -========================================================== - -Created by <span class="author"> Adam Caprez</span>, last modified by -<span class="editor"> Derek Weitzel</span> on Nov 05, 2013 ++++ +title = "Job Dependencies" +description = "How to use job dependencies with the SLURM scheduler." ++++ The job dependency feature of SLURM is useful when you need to run multiple jobs in a particular order. A standard example of this is a @@ -17,15 +11,14 @@ then manually submit the next, all the jobs in the workflow can be submitted at once. SLURM will then run them in the proper order based on the conditions supplied. -Syntax ------- +### Syntax -The basic syntax is to include the `-d `option with the `sbatch` command +The basic syntax is to include the `-d` option with the `sbatch` command for a new submission to indicate it depends on another job. You must also supply the condition and job id upon which it depends. SLURM supports several possible conditions; see the `sbatch` -<a href="http://slurm.schedmd.com/sbatch.html" class="external-link">man page</a> -for all the options. The example here uses `after``ok`, which instructs +[man page](http://slurm.schedmd.com/sbatch.html) +for all the options. The example here uses `afterok`, which instructs SLURM to only run the submitted job after the dependency job has terminated without error (exit code 0). @@ -34,18 +27,12 @@ This example is usually referred to as a "diamond" workflow. There are B and C both depend on Job A completing before they can run. Job D then depends on Jobs B and C completing. -<span -class="confluence-embedded-file-wrapper confluence-embedded-manual-size"><img src="assets/images/4849718/4980738.png" title="Simple Dependencies" alt="Simple Dependencies" class="confluence-embedded-image" width="300" height="300" /></span> +{{< figure src="/images/4980738.png" width="400" >}} The SLURM submit files for each step are below. -**JobA.submit** <span class="collapse-source expand-control" -style="display:none;"><span -class="expand-control-icon icon"> </span><span -class="expand-control-text">Expand source</span></span> <span -class="collapse-spinner-wrapper"></span> - -``` syntaxhighlighter-pre +{{%expand "JobA.submit" %}} +{{< highlight batch >}} #!/bin/sh #SBATCH --job-name=JobA #SBATCH --time=00:05:00 @@ -55,15 +42,12 @@ class="collapse-spinner-wrapper"></span> echo "I'm job A" echo "Sample job A output" > jobA.out sleep 120 -``` +{{< /highlight >}} +{{% /expand %}} -**JobB.submit** <span class="collapse-source expand-control" -style="display:none;"><span -class="expand-control-icon icon"> </span><span -class="expand-control-text">Expand source</span></span> <span -class="collapse-spinner-wrapper"></span> -``` syntaxhighlighter-pre +{{%expand "JobB.submit" %}} +{{< highlight batch >}} #!/bin/sh #SBATCH --job-name=JobB #SBATCH --time=00:05:00 @@ -76,15 +60,11 @@ cat jobA.out >> jobB.out echo "" >> jobB.out echo "Sample job B output" >> jobB.out sleep 120 -``` - -**JobC.submit** <span class="collapse-source expand-control" -style="display:none;"><span -class="expand-control-icon icon"> </span><span -class="expand-control-text">Expand source</span></span> <span -class="collapse-spinner-wrapper"></span> +{{< /highlight >}} +{{% /expand %}} -``` syntaxhighlighter-pre +{{%expand "JobC.submit" %}} +{{< highlight batch >}} #!/bin/sh #SBATCH --job-name=JobC #SBATCH --time=00:05:00 @@ -97,15 +77,11 @@ cat jobA.out >> jobC.out echo "" >> jobC.out echo "Sample job C output" >> jobC.out sleep 120 -``` +{{< /highlight >}} +{{% /expand %}} -**JobD.submit** <span class="collapse-source expand-control" -style="display:none;"><span -class="expand-control-icon icon"> </span><span -class="expand-control-text">Expand source</span></span> <span -class="collapse-spinner-wrapper"></span> - -``` syntaxhighlighter-pre +{{%expand "JobC.submit" %}} +{{< highlight batch >}} #!/bin/sh #SBATCH --job-name=JobD #SBATCH --time=00:05:00 @@ -120,60 +96,53 @@ cat jobC.out >> jobD.out echo "" >> jobD.out echo "Sample job D output" >> jobD.out sleep 120 -``` +{{< /highlight >}} +{{% /expand %}} To start the workflow, submit Job A first: -**Submit Job A** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="Submit Job A" %}} +{{< highlight batch >}} [demo01@login.tusker demo01]$ sbatch JobA.submit Submitted batch job 666898 -``` +{{< /highlight >}} +{{% /panel %}} Now submit jobs B and C, using the job id from Job A to indicate the dependency: -**Submit Jobs B and C** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="Submit Jobs B and C" %}} +{{< highlight batch >}} [demo01@login.tusker demo01]$ sbatch -d afterok:666898 JobB.submit Submitted batch job 666899 [demo01@login.tusker demo01]$ sbatch -d afterok:666898 JobC.submit Submitted batch job 666900 -``` +{{< /highlight >}} +{{% /panel %}} Finally, submit Job D as depending on both jobs B and C: -**Submit Job D** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="Submit Job D" %}} +{{< highlight batch >}} [demo01@login.tusker demo01]$ sbatch -d afterok:666899:666900 JobD.submit Submitted batch job 666901 -``` +{{< /highlight >}} +{{% /panel %}} Running `squeue` will now show all four jobs. The output from `squeue` will also indicate that Jobs B, C, and D are in a pending state because of the dependency. -**squeue output** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="Squeue Output" %}} +{{< highlight batch >}} [demo01@login.tusker demo01]$ squeue -u demo01 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 666899 batch JobB demo01 PD 0:00 1 (Dependency) 666900 batch JobC demo01 PD 0:00 1 (Dependency) 666901 batch JobD demo01 PD 0:00 1 (Dependency) 666898 batch JobA demo01 R 0:52 1 c2409 -``` +{{< /highlight >}} +{{% /panel %}} As the each job completes successfully, SLURM will run the job(s) in the workflow as resources become available. - -Attachments: ------------- - -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[SimpleDependencies.png](attachments/4849718/4980738.png) (image/png) - - diff --git a/content/guides/submitting_jobs/monitoring_jobs.md b/content/guides/submitting_jobs/monitoring_jobs.md index 8976f03c73dc9a7f2d71c3f660ed659c74f9aaab..2e5d2e581cf69530d46926adcd6b66da617bb048 100644 --- a/content/guides/submitting_jobs/monitoring_jobs.md +++ b/content/guides/submitting_jobs/monitoring_jobs.md @@ -1,13 +1,7 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Monitoring Jobs </span> -========================================================= - -Created by <span class="author"> Carrie Brown</span>, last modified by -<span class="editor"> Natasha Pavlovikj</span> on Sep 19, 2018 ++++ +title = "Monitoring Jobs" +description = "How to find out information about running and completed jobs." ++++ Careful examination of running times, memory usage and output files will allow you to ensure the job completed correctly and give you a good idea @@ -18,17 +12,14 @@ of what memory and time limits to request in the future. To see the runtime and memory usage of a job that has completed, use the sacct command: -``` syntaxhighlighter-pre +{{< highlight bash >}} sacct -``` +{{< /highlight >}} Lists all jobs by the current user and displays information such as JobID, JobName, State, and ExitCode. -<span -class="confluence-embedded-file-wrapper confluence-embedded-manual-size"><img src="assets/images/21070052/21070053.png" class="confluence-embedded-image" height="150" /></span> - - +{{< figure src="/images/21070053.png" height="150" >}} Coupling this command with the --format flag will allow you to see more than the default information about a job. Fields to display should be @@ -36,15 +27,14 @@ listed as a comma separated list after the --format flag (without spaces). For example, to see the Elapsed time and Maximum used memory by a job, this command can be used: -``` syntaxhighlighter-pre +{{< highlight bash >}} sacct --format JobID,JobName,Elapsed,MaxRSS -``` +{{< /highlight >}} -<span -class="confluence-embedded-file-wrapper confluence-embedded-manual-size"><img src="assets/images/21070052/21070054.png" class="confluence-embedded-image" height="150" /></span> +{{< figure src="/images/21070054.png" height="150" >}} Additional arguments and format field information can be found in -<a href="https://slurm.schedmd.com/sacct.html" class="external-link">the SLURM documentation.</a> +[the SLURM documentation](https://slurm.schedmd.com/sacct.html). ### Monitoring Running Jobs: @@ -54,46 +44,43 @@ multi-process jobs, whereas the cgroup files provide information on memory usage. Both of these tools require the use of an interactive job on the same node as the job to be monitored. -<span -class="aui-icon aui-icon-small aui-iconfont-error confluence-information-macro-icon"></span> - +{{% notice warning %}} If the job to be monitored is using all available resources for a node, the user will not be able to obtain a simultaneous interactive job. +{{% /notice %}} After the job to be monitored is submitted and has begun to run, request an interactive job on the same node using the srun command: -``` syntaxhighlighter-pre +{{< highlight bash >}} srun --jobid=<JOB_ID> --pty bash -``` +{{< /highlight >}} -Where <JOB\_ID> is replaced by the job id for the monitored job as +Where `<JOB_ID>` is replaced by the job id for the monitored job as assigned by SLURM. Alternately, you can request the interactive job by nodename as follows: -``` syntaxhighlighter-pre +{{< highlight bash >}} srun --nodelist=<NODE_ID> --pty bash -``` +{{< /highlight >}} -Where <NODE\_ID> is replaced by the node name that the monitored +Where `<NODE_ID>` is replaced by the node name that the monitored job is running. This information can be found out by looking at the -squeue output under the NODELIST column. +squeue output under the `NODELIST` column. -<span -class="confluence-embedded-file-wrapper confluence-embedded-manual-size"><img src="assets/images/21070052/21070055.png" class="confluence-embedded-image" width="700" /></span> +{{< figure src="/images/21070055.png" width="700" >}} Once the interactive job begins, you can run top to view the processes on the node you are on: -<span -class="confluence-embedded-file-wrapper confluence-embedded-manual-size"><img src="assets/images/21070052/21070056.png" class="confluence-embedded-image" height="400" /></span> +{{< figure src="/images/21070056.png" height="400" >}} Output for top displays each running process on the node. From the above image, we can see the various MATLAB processes being run by user -cathrine98. To filter the list of processes, you can type \`u\` followed +cathrine98. To filter the list of processes, you can type `u` followed by the username of the user who owns the processes. To exit this screen, -press \`q\`. +press `q`. During a running job, the cgroup folder is created which contains much of the information used by sacct. These files can provide a live @@ -104,43 +91,20 @@ following commands: ##### To view current memory usage: -``` syntaxhighlighter-pre +{{< highlight bash >}} less /cgroup/memory/slurm/uid_<UID>/job_<SLURM_JOB_ID>/memory.usage_in_bytes -``` +{{< /highlight >}} -Where <UID> is replaced by your UID and <SLURM\_JOB\_ID> is +Where `<UID>` is replaced by your UID and `<SLURM_JOB_ID>` is replaced by the monitored job's Job ID as assigned by Slurm. -<span -class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span> - -To find your uid, use the command \`id -u\`. Your UID never changes but -is cluster specific (ie, your UID on Crane will always be the same but -will differ from your UID on the other clusters). +{{% notice note %}} +To find your uid, use the command `id -u`. Your UID never changes and is +the same on all HCC clusters (*not* on Anvil, however!). +{{% /notice %}} ##### To view maximum memory usage from start of job to current point: -``` syntaxhighlighter-pre +{{< highlight bash >}} cat /cgroup/memory/slurm/uid_${UID}/job_${SLURM_JOBID}/memory.max_usage_in_bytes -``` - -Attachments: ------------- - -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[image2017-7-10\_11-16-5.png](attachments/21070052/21070053.png) -(image/png) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[image2017-7-10\_11-16-36.png](attachments/21070052/21070054.png) -(image/png) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[image2017-7-10\_12-14-1.png](attachments/21070052/21070055.png) -(image/png) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[image2017-7-10\_12-23-27.png](attachments/21070052/21070056.png) -(image/png) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[image2017-7-10\_12-27-45.png](attachments/21070052/21070057.png) -(image/png) - - +{{< /highlight >}} diff --git a/content/guides/submitting_jobs/submitting_a_job_array.md b/content/guides/submitting_jobs/submitting_a_job_array.md index 8772ed97b9cfc196fad893938d59f08f06fc462d..08245c8d7764c9d00d1abbeaf59ddcf3b93d47b3 100644 --- a/content/guides/submitting_jobs/submitting_a_job_array.md +++ b/content/guides/submitting_jobs/submitting_a_job_array.md @@ -1,30 +1,20 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Submitting a Job Array </span> -================================================================ - -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Carrie Brown</span> on Jun 29, 2017 ++++ +title = "Submitting a Job Array" +description = "How to use job arrays with the SLURM scheduler." ++++ A job array is a set of jobs that share the same submit file, but will run multiple copies with a environment variable incremented. These are useful when you need to process a series of files, or if you need to run the same application multiple times. -Creating a Array Submit File ----------------------------- +### Creating a Array Submit File An array submit file is very similar to the example submit files -in [Submitting Jobs](Submitting-Jobs_332222.html). - - - -**example.slurm** +in [Submitting Jobs]({{< relref "/guides/submitting_jobs/_index.md" >}}). -``` syntaxhighlighter-pre +{{% panel theme="info" header="example.slurm" %}} +{{< highlight batch >}} #!/bin/sh #SBATCH --array=0-31 #SBATCH --time=03:15:00 # Run time in hh:mm:ss @@ -37,18 +27,12 @@ module load example/test echo "I am task $SLURM_ARRAY_TASK_ID on node `hostname`" sleep 60 -``` +{{< /highlight >}} +{{% /panel %}} The submit file above will output the `$SLURM_ARRAY_TASK_ID`, which will be different for every one of the 32 (0-31) jobs, to the output files. Job Arrays in SLURM have more features than this brief description, -please visit -the <a href="https://slurm.schedmd.com/job_array.html" class="external-link">SLURM Documentation</a> for -more detailed information. - - - - - - +please visit the [SLURM Documentation](https://slurm.schedmd.com/job_array.html) +for more detailed information. diff --git a/content/guides/submitting_jobs/submitting_an_interactive_job.md b/content/guides/submitting_jobs/submitting_an_interactive_job.md index 38bfbbd71395cffe950a791c2f916f1ae798daa8..ba5eed2c9127813c6c11c519fc69413c1da22a5e 100644 --- a/content/guides/submitting_jobs/submitting_an_interactive_job.md +++ b/content/guides/submitting_jobs/submitting_an_interactive_job.md @@ -1,31 +1,24 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Submitting an Interactive Job </span> -======================================================================= - -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Adam Caprez</span> on Sep 14, 2018 - -<span -class="aui-icon aui-icon-small aui-iconfont-error confluence-information-macro-icon"></span> ++++ +title = "Submitting an Interactive Job" +description = "How to run an interactive job on HCC resources." ++++ +{{% notice info %}} The `/home` directories are read-only on the worker nodes. You will need to compile or run your processing in `/work`. +{{% /notice %}} Submitting an interactive job is done with the command `srun`. -``` syntaxhighlighter-pre +{{< highlight bash >}} $ srun --pty $SHELL -``` +{{< /highlight >}} or to allocate 4 cores per node: -``` syntaxhighlighter-pre +{{< highlight bash >}} $ srun --nodes=1 --ntasks-per-node=4 --mem-per-cpu=1024 --pty $SHELL -``` +{{< /highlight >}} Submitting an interactive job is useful if you require extra resources to run some processing by hand. It is also very useful to debug your @@ -35,34 +28,25 @@ And interactive job is scheduled onto a worker node just like a regular job. You can provide options to the interactive job just as you would a regular SLURM job. -Priority for short jobs ------------------------ +### Priority for short jobs To run short jobs for testing and development work, a job can specify a different quality of service (QoS). The *short* QoS increases a jobs priority so it will run as soon as possible. -| <span style="color: rgb(0,51,102);">SLURM Specification</span> | -|----------------------------------------------------------------| -| `--qos=short` | - -Limits per user for 'short' QoS - -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> +| SLURM Specification | +|---------------------| +| `--qos=short` | -- 6 hour job run time -- 2 jobs of 16 CPUs or fewer -- No more than 256 CPUs in use for *short* jobs from all users -**Using the short QoS** +{{% panel theme="warning" header="Limits per user for 'short' QoS" %}} +- 6 hour job run time +- 2 jobs of 16 CPUs or fewer +- No more than 256 CPUs in use for *short* jobs from all users +{{% /panel %}} -``` syntaxhighlighter-pre +{{% panel theme="info" header="Using the short QoS" %}} +{{< highlight bash >}} srun --qos=short --nodes=1 --ntasks-per-node=1 --mem-per-cpu=1024 --pty $SHELL -``` - - - - - - +{{< /highlight >}} +{{% /panel %}} diff --git a/content/guides/submitting_jobs/submitting_an_mpi_job.md b/content/guides/submitting_jobs/submitting_an_mpi_job.md index 9ee1f536bbbcbab7fc8e697d14c5b3593417692f..89315e421b87fc772867f7beb02e3f085a60f0e8 100644 --- a/content/guides/submitting_jobs/submitting_an_mpi_job.md +++ b/content/guides/submitting_jobs/submitting_an_mpi_job.md @@ -1,1344 +1,44 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Submitting an MPI Job </span> -=============================================================== - -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Natasha Pavlovikj</span> on Sep 04, 2014 ++++ +title = "Submitting an MPI Job" +description = "How to submit an MPI job on HCC resources." ++++ This script requests 16 cores on nodes with InfiniBand: -**mpi.submit** - -\#!/bin/sh -\#SBATCH --ntasks=16 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --time=03:15:00 -\#SBATCH --error=/work/\[groupname\]/\[username\]/job.%J.err -\#SBATCH --output=/work/\[groupname\]/\[username\]/job.%J.out - - +{{% panel theme="info" header="mpi.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --ntasks=16 +#SBATCH --mem-per-cpu=1024 +#SBATCH --time=03:15:00 +#SBATCH --error=/work/[groupname]/[username]/job.%J.err +#SBATCH --output=/work/[groupname]/[username]/job.%J.out -| | -|------------------------------------------| -| module load compiler/gcc/8.2 openmpi/2.1 | - -mpirun /home/\[groupname\]/\[username\]/mpiprogram +module load compiler/gcc/8.2 openmpi/2.1 +mpirun /home/[groupname]/[username]/mpiprogram +{{< /highlight >}} +{{% /panel %}} The above job will allocate 16 cores on the default partition. The 16 cores could be on any of the nodes in the partition, even split between multiple nodes. -Advanced Submission -------------------- +### Advanced Submission Some users may prefer to specify more details. This will allocate 32 tasks, 16 on each of two nodes: -**mpi.submit** - -\#!/bin/sh -\#SBATCH --nodes=2 -\#SBATCH --ntasks-per-node=16 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --time=03:15:00 -\#SBATCH --error=/work/\[groupname\]/\[username\]/job.%J.err -\#SBATCH --output=/work/\[groupname\]/\[username\]/job.%J.out - - - -| | -|------------------------------------------| -| module load compiler/gcc/8.2 openmpi/2.1 | - -mpirun /home/\[groupname\]/\[username\]/mpiprogram - -Attachments: ------------- - -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8127339.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8127409.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8127664.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[cb\_mpi.xsl](attachments/332242/8127418.xsl) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8127740.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945694.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945728.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945787.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945822.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945856.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945890.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8945942.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8946012.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863198.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863232.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863284.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863359.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863415.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863452.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863551.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863653.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863710.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863794.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863846.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/9863907.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/10387508.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/10944557.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/10944597.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11632795.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11632883.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11632933.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633008.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633085.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633155.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633234.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633285.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633375.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633438.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633478.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633552.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633637.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633678.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633725.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633765.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633868.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633932.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11633972.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634020.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634062.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634102.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634151.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634199.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634293.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634427.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634530.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634593.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634671.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634721.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11634919.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635000.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635123.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635228.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635412.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635678.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635772.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635820.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635862.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635904.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11635946.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636033.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636136.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636188.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636299.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636355.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636442.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636548.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636594.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636662.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636805.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636886.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11636930.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11637104.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11637173.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11637627.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11637671.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11637911.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11638002.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11638077.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11638147.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/11638190.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/12550233.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/12550281.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/12550331.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13041742.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13041830.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13041902.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13041964.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042120.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042244.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042293.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042336.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042439.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042509.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042606.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13042652.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13598776.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13598998.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13599353.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13599409.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13599452.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/13599512.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14057614.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14057657.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14057735.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14057837.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14057924.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14417972.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/14418218.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15171766.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15171839.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15171998.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15172399.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15172477.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15172598.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925318.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925365.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925410.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925459.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925504.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925555.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925607.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925656.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925702.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925747.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925795.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925842.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925888.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/15925941.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16515104.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16515520.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16515742.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16515868.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16515914.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516022.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516148.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516251.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516330.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516431.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516504.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516552.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516603.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516746.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16516983.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517086.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517198.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517363.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517441.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517547.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517594.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517640.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517686.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517776.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16517913.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16518085.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16518197.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16518646.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16518692.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16518780.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16518890.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519035.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519083.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519257.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519334.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519411.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519459.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519536.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519617.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519707.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519805.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519922.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16519968.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520014.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520162.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520240.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520300.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520415.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520461.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520621.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520710.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520790.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16520868.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16521026.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16521141.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16521249.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/16521333.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039378.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039426.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039474.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039522.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039572.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039632.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039834.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039882.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17039930.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040012.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040100.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040182.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040326.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040406.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040454.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040536.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17040679.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041058.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041124.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041195.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041526.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041578.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041769.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041897.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17041984.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042074.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042159.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042279.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042364.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042457.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042582.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042669.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042778.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042864.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17042920.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043039.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043134.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043185.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043270.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043360.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043447.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043579.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043665.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043786.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043871.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17043965.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044055.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044174.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044261.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044348.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044451.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044555.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044651.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044748.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17044848.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17045001.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/17047070.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18546740.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18546883.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547048.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547121.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547190.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547377.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547524.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547625.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547725.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547835.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547896.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18547996.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548160.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548279.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548380.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548525.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548612.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548725.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548825.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18548885.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/18549026.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/20709497.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/20709599.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21069914.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070032.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070170.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070282.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070520.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070623.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070742.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21070843.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071037.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071148.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071329.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071430.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071592.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071728.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071831.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21071934.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072034.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072175.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072275.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072377.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072497.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072646.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072748.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072850.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21072979.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21073079.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21073273.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21073421.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21073480.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/21073610.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150110.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150169.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150314.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150457.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150557.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150657.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150795.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24150901.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151056.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151159.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151262.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151405.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151519.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151687.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151752.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24151911.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24152024.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24152268.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24152457.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24152559.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24152765.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24152838.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24153009.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24153109.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24153209.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24153397.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24153498.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/24153557.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27721810.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27721932.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722038.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722166.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722267.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722369.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722648.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722763.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722867.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/27722969.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/29065522.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/29065598.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/29065832.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/29065932.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30441591.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30441692.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30441836.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30441895.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30441995.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442210.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442336.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442395.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442621.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442721.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442783.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30442847.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443029.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443139.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443251.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443378.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443588.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443691.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443807.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30443912.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444017.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444128.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444228.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444352.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444465.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444617.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444768.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444870.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30444972.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445036.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445142.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445244.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445471.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445574.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445686.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445789.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30445892.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446001.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446070.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446216.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446322.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446429.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446621.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446680.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446857.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30446959.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447121.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447183.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447292.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447516.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447620.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447693.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30447793.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448028.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448128.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448334.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448478.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448705.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448808.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30448915.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/30449023.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33685584.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33685695.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33685807.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33685912.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33686049.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33686153.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33686292.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33687828.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33688435.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33689761.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/33689981.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35324021.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35324131.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35324601.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35324891.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35325623.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35325736.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/35326723.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332242/8127220.xml) -(application/octet-stream) - - +{{% panel theme="info" header="mpi.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --nodes=2 +#SBATCH --ntasks-per-node=16 +#SBATCH --mem-per-cpu=1024 +#SBATCH --time=03:15:00 +#SBATCH --error=/work/[groupname]/[username]/job.%J.err +#SBATCH --output=/work/[groupname]/[username]/job.%J.out + +module load compiler/gcc/8.2 openmpi/2.1 +mpirun /home/[groupname]/[username]/mpiprogram +{{< /highlight >}} +{{% /panel %}} diff --git a/content/guides/submitting_jobs/submitting_an_openmp_job.md b/content/guides/submitting_jobs/submitting_an_openmp_job.md index db0b1d46000aeb4fe2c536e1e356d7d87cce7030..7c04168c8d6bc215cdb771a76ab3acc2e23f5eba 100644 --- a/content/guides/submitting_jobs/submitting_an_openmp_job.md +++ b/content/guides/submitting_jobs/submitting_an_openmp_job.md @@ -1,21 +1,14 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) ++++ +title = "Submitting an OpenMP Job" +description = "How to submit an OpenMP job on HCC resources." ++++ -<span id="title-text"> HCC-DOCS : Submitting an OpenMP Job </span> -================================================================== +Submitting an OpenMP job is different from +[Submitting an MPI Job]({{< relref "submitting_an_mpi_job" >}}) +since you must request multiple cores from a single node. -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Adam Caprez</span> on Jun 10, 2013 - -Submitting an OpenMP job is different from [Submitting an MPI -Job](Submitting-an-MPI-Job_332242.html) since you must request multiple -cores from a single node. - -**OpenMP example submission** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="OpenMP example submission" %}} +{{< highlight batch >}} #!/bin/sh #SBATCH --ntasks-per-node=16 # 16 cores #SBATCH --nodes=1 # 1 node @@ -24,26 +17,26 @@ cores from a single node. #SBATCH --error=/work/[groupname]/[username]/job.%J.err #SBATCH --output=/work/[groupname]/[username]/job.%J.out -export OMP_NUM_THREADS=16 +export OMP_NUM_THREADS=${SLURM_NTASKS_PER_NODE} ./openmp-app.exe -``` +{{< /highlight >}} +{{% /panel %}} Notice that we used `ntasks-per-node` to specify the number of cores we want on a single node. Additionally, we specify that we only want 1 `node`. `OMP_NUM_THREADS` is required to limit the number of cores that OpenMP -will use on the node. It should match the `ntasks-per-node` value. +will use on the node. It is set to ${SLURM_NTASKS_PER_NODE} to +automatically match the `ntasks-per-node` value (in this example 16). ### Compiling -Directions to compile OpenMP can be found on [Compiling an OpenMP -Application](Compiling-an-OpenMP-Application_332316.html). +Directions to compile OpenMP can be found on +[Compiling an OpenMP Application] +({{< relref "/guides/running_applications/compiling_source_code/compiling_an_openmp_application" >}}). ### Further Documentation Further OpenMP documentation can be found on LLNL's -<a href="https://computing.llnl.gov/tutorials/openMP/" class="external-link">OpenMP</a> -website. - - +[OpenMP](https://computing.llnl.gov/tutorials/openMP) website. diff --git a/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md b/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md index 9f7ed283f684958361127f5ef3cdb3d78e725a9d..bae4b76360a0af1262ea5cab04019a4856c23345 100644 --- a/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md +++ b/content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md @@ -1,1426 +1,94 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) ++++ +title = "Submitting CUDA or OpenACC Jobs" +description = "How to submit GPU (CUDA/OpenACC) jobs on HCC resources." ++++ -<span id="title-text"> HCC-DOCS : Submitting CUDA or OpenACC Jobs </span> -========================================================================= - -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Zhongtao Zhang</span> on Mar 08, 2018 - -GPUs available --------------- +### Available GPUs Crane has four types of GPUs available in the **gpu** partition. The type of GPU is configured as a SLURM feature, so you can specify a type of GPU in your job resource requirements if necessary. -<table> -<thead> -<tr class="header"> -<th>Description</th> -<th>SLURM feature</th> -<th>Available Hardware</th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td>Tesla K20, non-IB</td> -<td>gpu_k20</td> -<td>3 nodes - 2 GPUs per node</td> -</tr> -<tr class="even"> -<td>Tesla K20, with IB</td> -<td>gpu_k20</td> -<td>3 nodes - 3 GPUs per node</td> -</tr> -<tr class="odd"> -<td>Tesla K40, with IB</td> -<td>gpu_k40</td> -<td><p>5 nodes - 4 K40M GPUs per node</p> -<p>1 node - 2 K40C GPUs per node</p></td> -</tr> -<tr class="even"> -<td>Tesla P100, with OPA</td> -<td>gpu_p100</td> -<td>2 nodes - 2 GPUs per node</td> -</tr> -</tbody> -</table> +| Description | SLURM Feature | Available Hardware | +| -------------------- | ------------- | ---------------------------- | +| Tesla K20, non-IB | gpu_k20 | 3 nodes - 2 GPUs per node | +| Teska K20, with IB | gpu_k20 | 3 nodes - 3 GPUs per node | +| Tesla K40, with IB | gpu_k40 | 5 nodes - 4 K40M GPUs per node<br> 1 node - 2 K40C GPUs | +| Tesla P100, with OPA | gpu_p100 | 2 nodes - 2 GPUs per node | + To run your job on the next available GPU regardless of type, add the following options to your srun or sbatch command: -``` syntaxhighlighter-pre +{{< highlight batch >}} --partition=gpu --gres=gpu -``` +{{< /highlight >}} To run on a specific type of GPU, you can constrain your job to require a feature. To run on K40 GPUs for example: -``` syntaxhighlighter-pre +{{< highlight batch >}} --partition=gpu --gres=gpu --constraint=gpu_k40 -``` - -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> +{{< /highlight >}} +{{% notice info %}} You may request multiple GPUs by changing the` --gres` value to -`-gres=gpu:2`. Note that this value is **per node**. For example, `--nodes=2 --gres=gpu:2 `will request 2 nodes with 2 GPUs each, for a total of 4 GPUs. +{{% /notice %}} -Compiling ---------- +### Compiling Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes. -Therefore, you must run an [interactive -job](Submitting-an-Interactive-Job_332265.html) to compile. An example -command to compile in the **gpu** partition could be: +Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}}) +to compile. An example command to compile in the **gpu** partition could be: -``` syntaxhighlighter-pre +{{< highlight batch >}} $ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL -``` +{{< /highlight >}} The above command will start a shell on a GPU node with 6 cores and 6GB of ram in order to compile a GPU job. The above command could also be useful if you want to run a test GPU job interactively. -Submitting Jobs ---------------- +### Submitting Jobs CUDA and OpenACC submissions require running on GPU nodes. -**cuda.submit** - -\#!/bin/sh -\#SBATCH --time=03:15:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=cuda -\#SBATCH --partition=gpu -\#SBATCH --gres=gpu -\#SBATCH --error=/work/\[groupname\]/\[username\]/job.%J.err -\#SBATCH --output=/work/\[groupname\]/\[username\]/job.%J.out - - - -| | -|----------------------| -| module load cuda/6.5 | - +{{% panel theme="info" header="cuda.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=03:15:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=cuda +#SBATCH --partition=gpu +#SBATCH --gres=gpu +#SBATCH --error=/work/[groupname]/[username]/job.%J.err +#SBATCH --output=/work/[groupname]/[username]/job.%J.out + +module load cuda/8.0 ./cuda-app.exe - - +{{< /highlight >}} +{{% /panel %}} OpenACC submissions require loading the PGI compiler (which is currently required to compile as well). -**openacc.submit** -\#!/bin/sh -\#SBATCH --time=03:15:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=cuda-acc -\#SBATCH --partition=gpu -\#SBATCH --gres=gpu -\#SBATCH --error=/work/\[groupname\]/\[username\]/job.%J.err -\#SBATCH --output=/work/\[groupname\]/\[username\]/job.%J.out +{{% panel theme="info" header="openacc.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=03:15:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=cuda-acc +#SBATCH --partition=gpu +#SBATCH --gres=gpu +#SBATCH --error=/work/[groupname]/[username]/job.%J.err +#SBATCH --output=/work/[groupname]/[username]/job.%J.out - - -| | -|--------------------------------------| -| module load cuda/6.5 compiler/pgi/16 | +module load cuda/8.0 compiler/pgi/16 ./acc-app.exe - -Attachments: ------------- - -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8127340.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8127410.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8127665.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[cb\_cuda\_sj.xsl](attachments/332311/8127419.xsl) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[cb\_openacc\_sj.xsl](attachments/332311/8127420.xsl) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8127741.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945695.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945729.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945788.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945823.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945857.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945891.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8945943.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8946013.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863199.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863233.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863285.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863360.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863416.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863453.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863552.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863654.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863711.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863795.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863847.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/9863908.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/10387509.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/10944558.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/10944598.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11632796.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11632884.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11632934.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633009.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633086.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633156.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633235.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633286.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633376.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633439.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633479.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633553.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633638.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633679.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633726.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633766.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633869.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633933.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11633973.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634021.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634063.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634103.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634152.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634200.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634294.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634428.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634531.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634594.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634672.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634722.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11634920.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635001.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635124.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635229.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635413.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635679.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635773.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635821.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635863.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635905.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11635947.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636034.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636137.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636189.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636300.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636356.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636443.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636549.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636595.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636663.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636806.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636887.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11636931.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11637105.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11637174.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11637628.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11637672.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11637912.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11638003.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11638078.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11638148.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/11638191.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/12550234.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/12550282.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/12550332.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13041743.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13041831.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13041903.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13041965.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042121.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042245.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042294.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042337.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042440.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042510.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042607.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13042653.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13598777.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13598999.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13599354.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13599410.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13599453.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/13599513.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14057615.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14057658.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14057736.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14057838.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14057925.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14417973.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/14418219.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15171767.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15171840.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15171999.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15172400.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15172478.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15172599.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925319.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925366.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925411.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925460.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925505.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925556.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925608.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925657.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925703.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925748.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925796.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925843.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925889.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/15925942.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16515107.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16515523.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16515745.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16515871.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16515917.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516025.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516151.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516254.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516333.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516434.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516507.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516555.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516606.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516749.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16516986.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517089.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517201.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517366.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517444.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517550.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517597.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517643.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517689.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517779.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16517916.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16518088.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16518200.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16518649.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16518695.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16518783.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16518893.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519038.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519086.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519260.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519337.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519414.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519462.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519539.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519620.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519710.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519808.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519925.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16519971.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520017.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520165.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520243.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520303.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520418.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520464.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520624.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520713.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520793.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16520871.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16521029.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16521144.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16521252.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/16521336.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039381.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039429.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039477.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039525.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039575.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039635.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039837.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039885.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17039933.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040015.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040103.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040185.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040329.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040409.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040457.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040539.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17040682.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041061.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041127.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041198.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041529.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041581.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041770.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041898.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17041985.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042075.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042160.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042280.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042365.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042458.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042583.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042670.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042779.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042865.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17042921.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043040.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043135.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043186.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043271.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043361.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043448.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043580.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043666.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043787.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043872.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17043966.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044056.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044175.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044262.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044349.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044452.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044556.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044652.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044749.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17044849.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17045002.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/17047071.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18546741.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18546884.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547049.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547122.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547191.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547378.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547525.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547626.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547726.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547836.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547897.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18547997.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548161.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548280.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548381.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548526.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548613.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548726.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548826.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18548886.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/18549027.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/20709498.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/20709600.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21069915.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070033.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070171.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070283.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070521.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070624.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070743.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21070844.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071038.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071149.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071330.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071431.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071593.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071729.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071832.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21071935.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072035.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072176.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072276.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072378.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072498.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072647.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072749.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072851.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21072980.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21073080.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21073274.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21073422.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21073481.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/21073611.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150111.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150170.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150315.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150458.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150558.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150658.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150796.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24150902.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151057.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151160.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151263.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151406.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151520.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151688.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151753.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24151912.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24152025.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24152269.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24152458.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24152560.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24152766.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24152839.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24153010.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24153110.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24153210.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24153398.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24153499.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/24153558.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27721811.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27721933.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722039.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722167.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722268.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722370.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722649.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722764.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722868.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/27722970.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/29065523.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/29065599.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/29065833.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/29065933.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30441592.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30441693.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30441837.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30441896.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30441996.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442211.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442337.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442396.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442622.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442722.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442784.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30442848.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443030.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443140.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443252.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443379.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443589.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443692.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443808.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30443913.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444018.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444129.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444229.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444353.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444466.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444618.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444769.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444871.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30444973.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445037.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445143.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445245.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445472.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445575.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445687.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445790.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30445893.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446002.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446071.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446217.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446323.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446430.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446622.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446681.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446858.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30446960.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447122.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447184.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447293.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447517.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447621.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447694.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30447794.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448029.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448129.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448335.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448479.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448706.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448809.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30448916.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/30449024.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33685585.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33685696.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33685808.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33685913.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33686050.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33686154.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33686293.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33687829.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33688436.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33689762.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/33689982.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35324022.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35324132.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35324602.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35324892.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35325624.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35325737.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/35326724.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/332311/8127227.xml) -(application/octet-stream) - - +{{< /highlight >}} +{{% /panel %}} diff --git a/content/guides/submitting_jobs/submitting_htcondor_jobs.md b/content/guides/submitting_jobs/submitting_htcondor_jobs.md index e1e689a27eb050c7713815b0774bb474af239884..a917caf294139139c2b6ec8759e8595c3c21f0c9 100644 --- a/content/guides/submitting_jobs/submitting_htcondor_jobs.md +++ b/content/guides/submitting_jobs/submitting_htcondor_jobs.md @@ -1,28 +1,16 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Submitting HTCondor jobs </span> -================================================================== - -Created by <span class="author"> Derek Weitzel</span>, last modified on -Sep 13, 2013 - -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> - -HTCondor is only available on Sandhills. ++++ +title = "Submitting HTCondor Jobs" +description = "How to submit HTCondor Jobs on HCC resources." ++++ If you require features of HTCondor, such as DAGMan or Pegasus, -<a href="http://research.cs.wisc.edu/htcondor/" class="external-link">HTCondor</a> can -submit jobs onto Sandhills using HTCondor's PBS integration. This can +[HTCondor](http://research.cs.wisc.edu/htcondor/) can +submit jobs using HTCondor's PBS integration. This can be done by adding `grid_resource = pbs` to the submit file. An example submission script is below: -**submit.condor** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="submit.condor" %}} +{{< highlight batch >}} universe = grid grid_resource = pbs executable = test.sh @@ -31,25 +19,21 @@ error = stuff.err log = stuff.log batch_queue = guest queue -``` +{{< /highlight >}} +{{% /panel %}} The above script will translate the condor submit file into a SLURM submit file, and execute the `test.sh` executable on a worker node. -<span -class="aui-icon aui-icon-small aui-iconfont-error confluence-information-macro-icon"></span> - -The `/home` directories are read only on the Sandhills worker nodes. You +{{% notice warning %}} +The `/home` directories are read only on the worker nodes. You have to submit your jobs from the `/work` directory just as you would in SLURM. +{{% /notice %}} -Using Pegasus -------------- +### Using Pegasus -If you are using -<a href="http://pegasus.isi.edu/" class="external-link">Pegasus</a>, +If you are using [Pegasus](http://pegasus.isi.edu), instructions on using the *glite* interface (as shown above) are available on the -<a href="http://pegasus.isi.edu/wms/docs/latest/execution_environments.php#glite" class="external-link">User Guide</a>. - - +[User Guide](http://pegasus.isi.edu/wms/docs/latest/execution_environments.php#glite). diff --git a/content/guides/submitting_jobs/submitting_jobs.md b/content/guides/submitting_jobs/submitting_jobs.md deleted file mode 100644 index 1f2d3071503619875b87829df5f909e16b2721c1..0000000000000000000000000000000000000000 --- a/content/guides/submitting_jobs/submitting_jobs.md +++ /dev/null @@ -1,227 +0,0 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) - -<span id="title-text"> HCC-DOCS : Submitting Jobs </span> -========================================================= - -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Natasha Pavlovikj</span> on Aug 10, 2017 - -Crane, Sandhills and Tusker are managed by -the <a href="https://slurm.schedmd.com" class="external-link">SLURM</a> resource -manager. In order to run processing on Crane, Sandhills or Tusker, you -must create a SLURM script that will run your processing. After -submitting the job, SLURM will schedule your processing on an available -worker node. - -Before writing a submit file, you may need to [compile your -application](Compiling-Source-Code_332258.html). - -- [Ensure proper working directory for job - output](#SubmittingJobs-Ensureproperworkingdirectoryforjoboutput) -- [Creating a SLURM Submit - File](#SubmittingJobs-CreatingaSLURMSubmitFile) -- [Submitting the job](#SubmittingJobs-Submittingthejob) -- [Checking Job Status](#SubmittingJobs-CheckingJobStatus) - - [Checking Job Start](#SubmittingJobs-CheckingJobStart) -- [Next Steps](#SubmittingJobs-NextSteps) - -Ensure proper working directory for job output ----------------------------------------------- - -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> - -All SLURM job output should be directed to your /work path. - -**manual specification of /work path** - -``` syntaxhighlighter-pre -$ cd /work/[groupname]/[username] -``` - -The environment variable $WORK can also be used. - -**using environment for /work path** - -``` syntaxhighlighter-pre -$ cd $WORK -$ pwd -/work/[groupname]/[username] -``` - -Review how /work differs from /home [here.](Handling-Data_332256.html) - -Creating a SLURM Submit File ----------------------------- - -<span -class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span> - -The below example is for a serial job. For submitting MPI jobs, please -look at the [MPI Submission Guide.](Submitting-an-MPI-Job_332242.html) - -A SLURM submit file is broken into 2 sections, the job description and -the processing. SLURM job description are prepended with `#SBATCH` in -the submit file. - -**SLURM Submit File** - -``` syntaxhighlighter-pre -#!/bin/sh -#SBATCH --time=03:15:00 # Run time in hh:mm:ss -#SBATCH --mem-per-cpu=1024 # Maximum memory required per CPU (in megabytes) -#SBATCH --job-name=hello-world -#SBATCH --error=/work/[groupname]/[username]/job.%J.err -#SBATCH --output=/work/[groupname]/[username]/job.%J.out - -module load example/test - -hostname -sleep 60 -``` - -- **time** - Maximum walltime the job can run. After this time has expired, the - job will be stopped. -- **mem-per-cpu** - Memory that is allocated per core for the job. If you exceed this - memory limit, your job will be stopped. -- **mem** - Specify the real memory required per node in MegaBytes. If you - exceed this limit, your job will be stopped. Note that for you - should ask for less memory than each node actually has. For - instance, Tusker has 1TB, 512GB and 256GB of RAM per node. You may - only request 1000GB of RAM for the 1TB node, 500GB of RAM for the - 512GB nodes, and 250GB of RAM for the 256GB nodes. For Crane, the - max is 500GB. -- **job-name - **The name of the job. Will be reported in the job listing. -- **partition** - The partition the job should run in. Partitions determine the job's - priority and on what nodes the partition can run on. See [Available - Partitions on - Sandhills](Available-Partitions-on-Sandhills_332211.html), and - [Available Partitions on Crane and - Tusker](https://hcc-docs.unl.edu/display/HCCDOC/Available+Partitions+on+Crane+and+Tusker) for - a list of possible partitions. -- **error** - Location of the stderr will be written for the job. `[groupname]` - and `[username]` should be replaced your group name and username. - Your username can be retrieved with the command `id -un` and your - group with `id -ng`. -- **output** - Location of the stdout will be written for the job. - -More advanced submit commands can be found on the -<a href="https://slurm.schedmd.com/sbatch.html" class="external-link">SLURM Docs</a>. - You can also find an example of a MPI submission on [Submitting an MPI -Job](Submitting-an-MPI-Job_332242.html). - -Submitting the job ------------------- - -Submitting the SLURM job is done by command `sbatch`. SLURM will read -the submit file, and schedule the job according to the description in -the submit file. - -Submitting the job described above is: - -**SLURM Submission** - -``` syntaxhighlighter-pre -$ sbatch example.slurm -Submitted batch job 24603 -``` - -The job was successfully submitted. - -Checking Job Status -------------------- - -Job status is found with the command `squeue`. It will provide -information such as: - -- The State of the job: - - **R** - Running - - **PD** - Pending - <span style="color: rgb(0,0,0);">Job is - awaiting resource allocation.</span> - - <span style="color: rgb(0,0,0);">Additional codes are available - on the - <a href="http://slurm.schedmd.com/squeue.html" class="external-link">squeue</a> - page.</span><span style="color: rgb(0,0,0);"> - </span> -- <span style="color: rgb(0,0,0);">Job Name</span> -- <span style="color: rgb(0,0,0);">Run Time</span> -- <span style="color: rgb(0,0,0);">Nodes running the job</span> - -Checking the status of the job is easiest by filtering by your username, -using the `-u` option to squeue. - -``` syntaxhighlighter-pre -$ squeue -u <username> - JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) - 24605 batch hello-wo <username> R 0:56 1 b01 -``` - -Additionally, if you want to see the status of a specific partition, for -example if you are part of a -[partition](Available-Partitions-on-Sandhills_332211.html), you can use -the `-p` option to `squeue`: - -``` syntaxhighlighter-pre -$ squeue -p esquared - JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) - 73435 esquared MyRandom tingting R 10:35:20 1 ri19n10 - 73436 esquared MyRandom tingting R 10:35:20 1 ri19n12 - 73735 esquared SW2_driv hroehr R 10:14:11 1 ri20n07 - 73736 esquared SW2_driv hroehr R 10:14:11 1 ri20n07 -``` - -### Checking Job Start - -You may view the start time of your job with the -command `squeue --start`. The output of the command will show the -expected start time of the jobs. - -``` syntaxhighlighter-pre -$ squeue --start --user lypeng - JOBID PARTITION NAME USER ST START_TIME NODES NODELIST(REASON) - 5822 batch Starace lypeng PD 2013-06-08T00:05:09 3 (Priority) - 5823 batch Starace lypeng PD 2013-06-08T00:07:39 3 (Priority) - 5824 batch Starace lypeng PD 2013-06-08T00:09:09 3 (Priority) - 5825 batch Starace lypeng PD 2013-06-08T00:12:09 3 (Priority) - 5826 batch Starace lypeng PD 2013-06-08T00:12:39 3 (Priority) - 5827 batch Starace lypeng PD 2013-06-08T00:12:39 3 (Priority) - 5828 batch Starace lypeng PD 2013-06-08T00:12:39 3 (Priority) - 5829 batch Starace lypeng PD 2013-06-08T00:13:09 3 (Priority) - 5830 batch Starace lypeng PD 2013-06-08T00:13:09 3 (Priority) - 5831 batch Starace lypeng PD 2013-06-08T00:14:09 3 (Priority) - 5832 batch Starace lypeng PD N/A 3 (Priority) -``` - -The output shows the expected start time of the jobs, as well as the -reason that the jobs are currently idle (in this case, low priority of -the user due to running numerous jobs already). - - - -<span -style="color: rgb(0,0,0);font-size: 20.0px;line-height: 1.5;">Removing -the Job</span> - -Removing the job is done with the `scancel` command. The only argument -to the `scancel` command is the job id. For the job above, the command -is: - -``` syntaxhighlighter-pre -$ scancel 24605 -``` - -Next Steps ----------- - - - - diff --git a/content/guides/submitting_jobs/submitting_matlab_jobs.md b/content/guides/submitting_jobs/submitting_matlab_jobs.md index a13940ae2e009e9538210bae86730519d708ffd8..c137f6505b4a0282f7f04baf024e66371db2292a 100644 --- a/content/guides/submitting_jobs/submitting_matlab_jobs.md +++ b/content/guides/submitting_jobs/submitting_matlab_jobs.md @@ -1,1358 +1,61 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) ++++ +title = "Submitting MATLAB Jobs" +description = "How to submit MATLAB jobs on HCC resources." ++++ -<span id="title-text"> HCC-DOCS : Submitting Matlab Jobs </span> -================================================================ +Submitting Matlab jobs is very similar to +[submitting MPI jobs]({{< relref "submitting_an_mpi_job" >}}) or +[serial jobs]({{< relref "/guides/submitting_jobs/_index.md" >}}) +(depending if you are using parallela matlab). -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Jingchao Zhang</span> on Mar 30, 2018 - -Submitting Matlab jobs is very similar to [submitting MPI -jobs](Submitting-an-MPI-Job_332242.html) or [serial -jobs](Submitting-Jobs_332222.html) (depending if you are using parallel -matlab). - -Submit File ------------ +### Submit File The submit file will need to be modified to allow Matlab to work. - Specifically, these two lines should be added before calling matlab: - -**serial\_matlab.submit** +Specifically, these two lines should be added before calling matlab: -\#!/bin/sh -\#SBATCH --time=03:15:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=\[job\_name\] -\#SBATCH --error=/work/\[groupname\]/\[username\]/job.%J.err -\#SBATCH --output=/work/\[groupname\]/\[username\]/job.%J.out +{{% panel theme="info" header="serial_matlab.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=03:15:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=[job_name] +#SBATCH --error=/work/[groupname]/[username]/job.%J.err +#SBATCH --output=/work/[groupname]/[username]/job.%J.out module load matlab/r2014b +matlab -nodisplay -r "[matlab script name], quit" +{{< /highlight >}} +{{% /panel %}} -matlab -nodisplay -r "\[matlab script name\], quit" - - - -<span style="color: rgb(0,0,0);">Parallel Matlab .m file</span> ---------------------------------------------------------------- +### Parallel Matlab .m file The submit file: -**parallel\_matlab.submit** - -\#!/bin/sh -\#SBATCH --nodes=1 -\#SBATCH --ntasks-per-node=5 -\#SBATCH --time=03:15:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=\[job\_name\] -\#SBATCH --error=/work/\[groupname\]/\[username\]/job.%J.err -\#SBATCH --output=/work/\[groupname\]/\[username\]/job.%J.out +{{% panel theme="info" header="parallel_matlab.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=5 +#SBATCH --time=03:15:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=[job_name] +#SBATCH --error=/work/[groupname]/[username]/job.%J.err +#SBATCH --output=/work/[groupname]/[username]/job.%J.out module load matlab/r2014b +matlab -nodisplay -r "[matlab script name], quit" +{{< /highlight >}} +{{% /panel %}} -matlab -nodisplay -r "\[matlab script name\], quit" - - +#### Matlab File Additions In addition to the changes in the submit file, if you are running -parallel Matlab, you will also need to add to the .m file an additional -line. - -**Matlab File Additions** +parallel Matlab, you will also need to add to the .m file the additional +lines: -``` syntaxhighlighter-pre +{{< highlight batch >}} ... i=str2num(getenv('SLURM_TASKS_PER_NODE')); parpool(i); ... -``` - -Attachments: ------------- - -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8127341.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8127411.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8127666.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[cb\_matlab.xsl](attachments/2097230/8127421.xsl) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8127742.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945696.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945730.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945789.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945824.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945858.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945892.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8945944.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8946014.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863200.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863234.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863286.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863361.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863417.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863454.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863553.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863655.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863712.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863796.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863848.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/9863909.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/10387510.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/10944559.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/10944599.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11632797.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11632885.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11632935.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633010.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633087.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633157.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633236.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633287.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633377.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633440.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633480.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633554.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633639.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633680.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633727.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633767.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633870.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633934.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11633974.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634022.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634064.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634104.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634153.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634201.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634295.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634429.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634532.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634595.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634673.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634723.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11634921.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635002.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635125.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635230.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635414.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635680.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635774.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635822.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635864.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635906.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11635948.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636035.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636138.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636190.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636301.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636357.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636444.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636550.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636596.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636664.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636807.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636888.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11636932.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11637106.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11637175.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11637629.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11637673.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11637913.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11638004.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11638079.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11638149.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/11638192.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/12550235.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/12550283.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/12550333.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13041744.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13041832.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13041904.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13041966.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042122.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042246.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042295.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042338.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042441.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042511.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042608.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13042654.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13598778.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13599000.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13599355.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13599411.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13599454.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/13599514.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14057616.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14057659.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14057737.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14057839.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14057926.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14417974.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/14418220.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15171768.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15171841.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15172000.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15172401.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15172479.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15172600.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925320.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925367.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925412.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925461.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925506.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925557.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925609.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925658.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925704.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925749.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925797.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925844.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925890.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/15925943.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16515109.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16515525.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16515747.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16515873.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16515919.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516027.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516153.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516256.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516335.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516436.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516509.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516557.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516608.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516751.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16516988.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517091.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517203.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517368.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517446.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517552.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517599.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517645.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517691.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517781.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16517918.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16518090.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16518202.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16518651.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16518697.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16518785.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16518895.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519040.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519088.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519262.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519339.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519416.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519464.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519541.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519622.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519712.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519810.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519927.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16519973.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520019.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520167.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520245.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520305.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520420.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520466.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520626.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520715.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520795.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16520873.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16521031.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16521146.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16521254.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/16521338.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039383.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039431.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039479.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039527.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039577.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039637.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039839.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039887.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17039935.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040017.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040105.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040187.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040331.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040411.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040459.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040541.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17040684.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041063.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041129.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041200.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041531.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041583.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041771.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041899.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17041986.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042076.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042161.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042281.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042366.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042459.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042584.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042671.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042780.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042866.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17042922.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043041.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043136.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043187.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043272.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043362.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043449.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043581.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043667.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043788.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043873.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17043967.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044057.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044176.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044263.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044350.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044453.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044557.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044653.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044750.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17044850.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17045003.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/17047072.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18546742.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18546885.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547050.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547123.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547192.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547379.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547526.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547627.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547727.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547837.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547898.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18547998.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548162.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548281.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548382.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548527.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548614.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548727.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548827.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18548887.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/18549028.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/20709499.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/20709601.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21069916.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070034.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070172.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070284.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070522.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070625.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070744.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21070845.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071039.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071150.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071331.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071432.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071594.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071730.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071833.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21071936.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072036.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072177.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072277.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072379.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072499.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072648.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072750.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072852.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21072981.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21073081.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21073275.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21073423.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21073482.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/21073612.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150112.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150171.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150316.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150459.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150559.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150659.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150797.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24150903.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151058.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151161.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151264.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151407.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151521.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151689.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151754.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24151913.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24152026.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24152270.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24152459.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24152561.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24152767.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24152840.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24153011.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24153111.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24153211.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24153399.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24153500.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/24153559.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27721812.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27721934.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722040.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722168.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722269.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722371.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722650.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722765.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722869.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/27722971.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/29065524.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/29065600.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/29065834.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/29065934.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30441593.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30441694.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30441838.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30441897.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30441997.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442212.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442338.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442397.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442623.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442723.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442786.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30442849.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443031.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443141.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443253.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443380.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443590.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443693.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443809.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30443914.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444019.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444130.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444230.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444354.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444467.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444619.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444770.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444872.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30444974.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445038.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445144.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445246.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445473.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445576.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445688.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445791.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30445894.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446003.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446072.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446218.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446324.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446431.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446623.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446682.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446859.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30446961.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447123.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447185.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447294.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447519.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447622.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447695.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30447795.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448030.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448130.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448336.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448480.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448707.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448810.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30448917.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/30449025.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33685586.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33685697.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33685809.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33685914.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33686051.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33686155.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33686294.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33687830.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33688437.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33689763.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/33689983.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35324023.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35324133.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35324603.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35324893.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35325625.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35325738.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/35326725.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2097230/8127234.xml) -(application/octet-stream) - - +{{< /highlight >}} diff --git a/content/guides/submitting_jobs/submitting_r_jobs.md b/content/guides/submitting_jobs/submitting_r_jobs.md index 72ffc19bb392e234a14b398edae1afd82e104e66..ffbe293bd00bf205a56ba9237777acf825b2f380 100644 --- a/content/guides/submitting_jobs/submitting_r_jobs.md +++ b/content/guides/submitting_jobs/submitting_r_jobs.md @@ -1,255 +1,209 @@ -1. [HCC-DOCS](index.html) -2. [HCC-DOCS Home](HCC-DOCS-Home_327685.html) -3. [HCC Documentation](HCC-Documentation_332651.html) -4. [Submitting Jobs](Submitting-Jobs_332222.html) - -<span id="title-text"> HCC-DOCS : Submitting R jobs </span> -=========================================================== - -Created by <span class="author"> Derek Weitzel</span>, last modified by -<span class="editor"> Natasha Pavlovikj</span> on Apr 04, 2018 - -<span -style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;">Submitting -an R job is very similar to submitting a serial job shown -on </span>[Submitting Jobs](Submitting-Jobs_332222.html) - -<span -style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;"></span> - -- [Running R scripts in - batch](#SubmittingRjobs-RunningRscriptsinbatch) - - [Running R scripts using \`R CMD - BATCH\`](#SubmittingRjobs-RunningRscriptsusing%60RCMDBATCH%60) - - [Running R scripts using - \`Rscript\`](#SubmittingRjobs-RunningRscriptsusing%60Rscript%60) -- [Multicore (parallel) R - submission](#SubmittingRjobs-Multicore(parallel)Rsubmission) -- [Multinode R submission with - Rmpi](#SubmittingRjobs-MultinodeRsubmissionwithRmpi) -- [Adding packages](#SubmittingRjobs-Addingpackages) - - [Installing packages - interactively](#SubmittingRjobs-Installingpackagesinteractively) - - [Installing packages using R CMD - INSTALL](#SubmittingRjobs-InstallingpackagesusingRCMDINSTALL) ++++ +title = "Submitting R Jobs" +description = "How to submit R jobs on HCC resources." ++++ + +Submitting an R job is very similar to submitting a serial job shown +on [Submitting Jobs]({{< relref "/guides/submitting_jobs/_index.md" >}}). + +- [Running R scripts in batch](#running-r-scripts-in-batch) + - [Running R scripts using `R CMD BATCH`](#running-r-scripts-using-r-cmd-batch) + - [Running R scripts using `Rscript`](#running-r-scripts-using-rscript) +- [Multicore (parallel) R submission](#multicore-parallel-r-submission) +- [Multinode R submission with Rmpi](#multinode-r-submission-with-rmpi) +- [Adding packages](#adding-packages) + - [Installing packages interactively](#installing-packages-interactively) + - [Installing packages using R CMD INSTALL](#installing-packages-using-r-cmd-install) +### Running R scripts in batch -Running R scripts in batch --------------------------- +There are two primary commands to use when submitting R scripts: `Rscript` +and `R CMD BATCH`. Both commands will execute the passed script but +differ in the way they process output. -<span -style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;">There -are two primary commands to use when submitting R scripts: \`Rscript\` -and \`R CMD BATCH\`. Both commands will execute the passed script but -differ in the way they process output.</span> +#### Running R scripts using `R CMD BATCH` -#### <span style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;">Running R scripts using \`R CMD BATCH\` </span> - -<span -style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;">When -utilizing \`R CMD BATCH\` all output will be directed to an \`.Rout\` +When utilizing `R CMD BATCH` all output will be directed to an `.Rout` file named after your script unless otherwise specified. For -example:</span> - -**serial\_R.submit** - -\#!/bin/sh -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob +example: - - -| | -|-------------------| -| module load R/3.3 | +{{% panel theme="info" header="serial_R.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob +module load R/3.5 R CMD BATCH Rcode.R +{{< /highlight >}} +{{% /panel %}} -<span -style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;">In -the above example, output for the job will be found in the file -\`Rcode.Rout\`. Notice that we did not specify output and error files in +In the above example, output for the job will be found in the file +`Rcode.Rout`. Notice that we did not specify output and error files in our SLURM directives, these are not needed as all R output will go into -the \`.Rout\` file. To direct output to a specific location, follow your -\`R CMD BATCH\` command with the name of the file where you want output -directed to, as follows:</span> +the `.Rout` file. To direct output to a specific location, follow your +`R CMD BATCH` command with the name of the file where you want output +directed to, as follows: - -**serial\_R.submit** - -\#!/bin/sh -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob - - - -| | -|-------------------| -| module load R/3.3 | +{{% panel theme="info" header="serial_R.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob +module load R/3.5 R CMD BATCH Rcode.R Rcodeoutput.txt +{{< /highlight >}} +{{% /panel %}} -<span -style="color: rgb(51,51,51);font-size: 14.0px;line-height: 1.4285715;">In -this example, output from running the script \`Rcode.R\` will be placed -in the file \`Rcodeoutput.txt\`. -</span> - -To pass arguments to the script, they need to be specified after \`R CMD -BATCH\` but before the script to be executed, and preferably preceded -with \`--args\` as follows: - -**serial\_R.submit** +In this example, output from running the script `Rcode.R` will be placed +in the file `Rcodeoutput.txt`. -\#!/bin/sh -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob +To pass arguments to the script, they need to be specified after `R CMD +BATCH` but before the script to be executed, and preferably preceded +with `--args` as follows: - +{{% panel theme="info" header="serial_R.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob -| | -|-------------------| -| module load R/3.3 | +module load R/3.5 +R CMD BATCH "--args argument1 argument2 argument3" Rcode.R Rcodeoutput.txt +{{< /highlight >}} +{{% /panel %}} -R CMD BATCH "–args argument1 argument2 argument3" Rcode.R -Rcodeoutput.txt -#### Running R scripts using \`Rscript\` +#### Running R scripts using `Rscript` -Using \`Rscript\` to execute R scripts differs from R CMD BATCH in that +Using `Rscript` to execute R scripts differs from R CMD BATCH in that all output and errors from the script are directed to STDOUT and STDERR in a manner similar to other programs. This gives the user larger control over where to direct the output. For example, to run our script -using \`Rscript\` the submit script could look like the following: - -**serial\_R.submit** - -\#!/bin/sh -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob -\#SBATCH --error=TestJob.%J.stderr -\#SBATCH --output=TestJob.%J.stdout - - - -| | -|-------------------| -| module load R/3.3 | - +using `Rscript` the submit script could look like the following: + +{{% panel theme="info" header="serial_R.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob +#SBATCH --error=TestJob.%J.stderr +#SBATCH --output=TestJob.%J.stdout + +module load R/3.5 Rscript Rcode.R +{{< /highlight >}} +{{% /panel %}} In the above example, STDOUT will be directed to the output file -\`TestJob.%J.stdout\` and STDERR directed to \`TestJob.%J.stderr\`. You -will notice that the example is very similar to to the [serial -example](Submitting-Jobs_332222.html). The important line is the -`module load` command. That tells Tusker to load the R framework into -the environment so jobs may use it. +`TestJob.%J.stdout` and STDERR directed to `TestJob.%J.stderr`. You +will notice that the example is very similar to to the +[serial example]({{< relref "/guides/submitting_jobs/_index.md" >}}). +The important line is the `module load` command. +That tells the cluster to load the R framework into the environment so jobs may use it. -To pass arguments to the script when using \`Rscript\`, the arguments +To pass arguments to the script when using `Rscript`, the arguments will follow the script name as in the example below: -**serial\_R.submit** - -\#!/bin/sh -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob -\#SBATCH --error=TestJob.%J.stderr -\#SBATCH --output=TestJob.%J.stdout - - - -| | -|-------------------| -| module load R/3.3 | +{{% panel theme="info" header="serial_R.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob +#SBATCH --error=TestJob.%J.stderr +#SBATCH --output=TestJob.%J.stdout +module load R/3.5 Rscript Rcode.R argument1 argument2 argument3 +{{< /highlight >}} +{{% /panel %}} -Multicore (parallel) R submission ---------------------------------- - -Submitting a multicore R job to SLURM is very similar to [Submitting an -OpenMP Job](Submitting-an-OpenMP-Job_332306.html), since both are -running multicore jobs on a single node. Below is an example: +--- -**parallel\_R.submit** +### Multicore (parallel) R submission -\#!/bin/sh -\#SBATCH --ntasks-per-node=16 -\#SBATCH --nodes=1 -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob -\#SBATCH --error=TestJob.%J.stdout -\#SBATCH --output=TestJob.%J.stderr +Submitting a multicore R job to SLURM is very similar to +[Submitting an OpenMP Job]({{< relref "submitting_an_openmp_job" >}}), +since both are running multicore jobs on a single node. Below is an example: - - -| | -|-------------------| -| module load R/3.3 | +{{% panel theme="info" header="parallel_R.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --ntasks-per-node=16 +#SBATCH --nodes=1 +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob +#SBATCH --error=TestJob.%J.stdout +#SBATCH --output=TestJob.%J.stderr +module load R/3.5 R CMD BATCH Rcode.R +{{< /highlight >}} +{{% /panel %}} The above example will submit a single job which can use up to 16 cores. - Be sure to use limits in your R code so you only use 16 cores, or your performance will suffer. For example, when using the -<a href="http://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf" class="external-link">parallel</a> +[parallel](http://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf) package function mclapply: -**parallel.R** - -``` syntaxhighlighter-pre +{{% panel theme="info" header="parallel.R" %}} +{{< highlight R >}} library("parallel") ... mclapply(rep(4, 5), rnorm, mc.cores=16) -``` - -Multinode R submission with Rmpi --------------------------------- - -Submitting a multinode MPI R job to SLURM is very similar to [Submitting -an MPI Job](Submitting-an-MPI-Job_332242.html), since both are running -multicore jobs on a multiple nodes. Below is an example of running Rmpi -on Crane on 2 nodes and 32 cores: - -**Rmpi.submit** - -\#!/bin/sh -\#SBATCH --nodes=2 -\#SBATCH --ntasks-per-node=16 -\#SBATCH --time=00:30:00 -\#SBATCH --mem-per-cpu=1024 -\#SBATCH --job-name=TestJob -\#SBATCH --error=TestJob.%J.stdout -\#SBATCH --output=TestJob.%J.stderr - -module load compiler/gcc/4.9 openmpi/1.10 R/3.3 -export OMPI\_MCA\_mtl=^psm +{{< /highlight >}} +{{% /panel %}} + +--- + +### Multinode R submission with Rmpi + +Submitting a multinode MPI R job to SLURM is very similar to +[Submitting an MPI Job]({{< relref "submitting_an_mpi_job" >}}), +since both are running multicore jobs on a multiple nodes. +Below is an example of running Rmpi on Crane on 2 nodes and 32 cores: + +{{% panel theme="info" header="Rmpi.submit" %}} +{{< highlight batch >}} +#!/bin/sh +#SBATCH --nodes=2 +#SBATCH --ntasks-per-node=16 +#SBATCH --time=00:30:00 +#SBATCH --mem-per-cpu=1024 +#SBATCH --job-name=TestJob +#SBATCH --error=TestJob.%J.stdout +#SBATCH --output=TestJob.%J.stderr + +module load compiler/gcc/4.9 openmpi/1.10 R/3.5 +export OMPI_MCA_mtl=^psm mpirun -n 1 R CMD BATCH Rmpi.R +{{< /highlight >}} +{{% /panel %}} -When you run Rmpi job on Crane, please use the line **export -OMPI\_MCA\_mtl=^psm** in your submit script. On the other hand, if you +When you run Rmpi job on Crane, please use the line `export +OMPI_MCA_mtl=^psm` in your submit script. On the other hand, if you run Rmpi job on Tusker, you **do not need** to add this line. This is because of the different Infiniband cards Tusker and Crane use. Regardless of how may cores your job uses, the Rmpi package should -always be run with **mpirun -n 1** because it spawns additional +always be run with `mpirun -n 1` because it spawns additional processes dynamically. Please find below an example of Rmpi R script provided by -<a href="https://rcc.uchicago.edu/docs/software/environments/R/index.html#rmpi" class="external-link">The University of Chicago Research Computing Center</a>: - -**Rmpi.R** +[The University of Chicago Research Computing Center](https://rcc.uchicago.edu/docs/software/environments/R/index.html#rmpi): -``` syntaxhighlighter-pre +{{% panel theme="info" header="Rmpi.R" %}} +{{< highlight R >}} library(Rmpi) # initialize an Rmpi environment @@ -267,34 +221,35 @@ mpi.remote.exec(paste("I am", id, "of", ns, "running on", host)) # close down the Rmpi environment mpi.close.Rslaves(dellog = FALSE) mpi.exit() -``` +{{< /highlight >}} +{{% /panel %}} -Adding packages ---------------- +--- + +### Adding packages There are two options to install packages. The first is to run R on the login node and run R interactively to install packages. The second is to -use the \`R CMD INSTALL\` command. - -<span -class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span> +use the `R CMD INSTALL` command. +{{% notice info %}} All R packages must be installed from the login node. R libraries are stored in user's home directories which are not writable from the worker nodes. +{{% /notice %}} #### Installing packages interactively -1. Load the R module with the command \`module load R\` - 1. Note that each version of R uses its own user libraries. To - install packages under a specific version of R, specify which - version by using the module load command followed by the version - number. For example, to load R version 3.3, you would use the - command \`module load R/3.3\` -2. Run R interactively using the command \`R\` -3. From within R, use the \`install.packages()\` command to install - desired packages. For example, to install the package \`ggplot2\` - use the command \`install.packages("ggplot2") +1. Load the R module with the command `module load R` + - Note that each version of R uses its own user libraries. To + install packages under a specific version of R, specify which + version by using the module load command followed by the version + number. For example, to load R version 3.5, you would use the + command `module load R/3.5` +2. Run R interactively using the command `R` +3. From within R, use the `install.packages()` command to install + desired packages. For example, to install the package `ggplot2` + use the command `install.packages("ggplot2")` Some R packages, require external compilers or additional libraries. If you see an error when installing your package you might need to load @@ -303,1307 +258,20 @@ more information about this, refer to the package documentation. #### Installing packages using R CMD INSTALL -To install packages using \`R CMD INSTALL\` the zipped package must +To install packages using `R CMD INSTALL` the zipped package must already be downloaded to the cluster. You can download package source -using \`wget\`. Then the \`R CMD INSTALL\` command can be used when +using `wget`. Then the `R CMD INSTALL` command can be used when pointed to the full path of the source tar file. For example, to install ggplot2 the following commands are used: -``` syntaxhighlighter-pre +{{< highlight bash >}} # Download the package source: wget https://cran.r-project.org/src/contrib/ggplot2_2.2.1.tar.gz # Install the package: R CMD INSTALL ./ggplot2_2.2.1.tar.gz -``` +{{< /highlight >}} -Additional information on using the \`R CMD INSTALL\` command can be -found in the R documentation which can be seen by typing \`?INSTALL\` +Additional information on using the `R CMD INSTALL` command can be +found in the R documentation which can be seen by typing `?INSTALL` within the R console. - -Attachments: ------------- - -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8127342.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8127412.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8127667.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[cb\_R\_sj.xsl](attachments/2098144/8127422.xsl) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8127743.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945697.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945731.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945790.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945825.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945859.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945893.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8945945.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8946015.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863201.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863235.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863287.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863362.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863418.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863455.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863554.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863656.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863713.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863797.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863849.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/9863910.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/10387511.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/10944560.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/10944600.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11632798.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11632886.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11632936.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633011.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633088.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633158.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633237.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633288.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633378.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633441.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633481.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633555.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633640.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633681.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633728.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633768.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633871.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633935.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11633975.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634023.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634065.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634105.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634154.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634202.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634296.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634430.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634533.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634596.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634674.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634724.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11634922.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635003.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635126.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635231.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635415.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635681.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635775.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635823.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635865.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635907.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11635949.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636036.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636139.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636191.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636302.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636358.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636445.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636551.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636597.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636665.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636808.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636889.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11636933.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11637107.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11637176.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11637630.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11637674.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11637914.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11638005.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11638080.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11638150.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/11638193.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/12550236.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/12550284.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/12550334.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13041745.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13041833.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13041905.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13041967.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042123.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042247.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042296.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042339.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042442.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042512.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042609.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13042655.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13598779.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13599001.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13599356.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13599412.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13599455.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/13599515.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14057617.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14057660.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14057738.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14057840.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14057927.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14417975.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/14418221.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15171769.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15171842.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15172001.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15172402.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15172480.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15172601.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925321.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925368.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925413.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925462.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925507.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925558.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925610.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925659.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925705.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925750.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925798.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925845.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925891.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/15925944.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16515117.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16515533.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16515755.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16515881.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16515927.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516035.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516161.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516264.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516343.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516444.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516517.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516565.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516616.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516759.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16516996.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517099.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517211.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517376.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517454.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517560.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517607.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517653.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517699.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517789.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16517926.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16518098.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16518210.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16518659.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16518705.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16518793.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16518903.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519048.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519096.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519270.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519347.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519424.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519472.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519549.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519630.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519720.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519818.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519935.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16519981.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520027.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520175.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520253.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520313.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520428.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520474.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520634.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520723.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520803.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16520881.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16521039.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16521154.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16521262.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/16521346.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039391.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039439.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039487.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039535.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039585.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039645.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039847.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039895.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17039943.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040025.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040113.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040195.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040339.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040419.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040467.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040549.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17040692.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041071.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041137.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041208.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041539.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041591.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041774.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041902.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17041989.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042079.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042164.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042284.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042369.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042462.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042587.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042674.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042783.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042869.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17042925.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043044.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043139.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043190.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043275.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043365.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043452.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043584.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043670.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043791.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043876.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17043970.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044060.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044179.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044266.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044353.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044456.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044560.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044656.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044753.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17044853.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17045006.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/17047075.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18546745.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18546888.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547053.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547126.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547195.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547382.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547529.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547630.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547730.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547840.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18547901.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548001.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548165.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548284.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548385.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548530.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548617.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548730.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548830.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18548890.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/18549031.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/20709443.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/20709545.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21069919.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070037.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070175.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070287.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070525.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070628.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070747.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21070848.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071042.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071153.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071334.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071435.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071597.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071733.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071836.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21071939.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072039.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072180.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072280.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072382.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072502.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072651.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072753.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072855.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21072984.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21073084.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21073278.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21073426.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21073485.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/21073615.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150115.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150174.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150319.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150462.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150562.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150662.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150800.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24150906.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151061.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151164.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151267.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151410.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151524.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151692.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151757.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24151916.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24152029.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24152273.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24152462.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24152564.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24152770.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24152843.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24153014.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24153114.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24153214.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24153402.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24153503.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/24153562.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27721815.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27721937.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722043.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722171.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722272.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722374.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722653.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722768.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722872.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/27722974.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/29065527.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/29065603.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/29065837.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/29065937.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30441537.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30441638.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30441782.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30441841.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30441941.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442156.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442282.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442341.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442567.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442667.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442729.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442793.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30442975.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443085.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443197.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443324.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443534.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443637.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443753.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443858.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30443963.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444074.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444174.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444298.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444411.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444563.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444714.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444816.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444918.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30444982.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445088.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445190.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445417.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445520.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445632.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445735.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445838.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30445947.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446016.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446162.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446268.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446375.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446567.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446626.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446803.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30446905.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447067.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447129.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447238.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447462.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447566.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447639.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447739.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30447974.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448074.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448280.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448423.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448650.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448753.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448860.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/30448968.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33685589.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33685700.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33685812.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33685917.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33686054.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33686158.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33686297.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33687833.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33688440.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33689766.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/33689986.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35324026.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35324136.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35324606.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35324896.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35325628.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35325741.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/35326728.xml) -(application/octet-stream) -<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" /> -[tusker\_modules.xml](attachments/2098144/8127243.xml) -(application/octet-stream) - - diff --git a/layouts/shortcodes/readfile.html b/layouts/shortcodes/readfile.html new file mode 100644 index 0000000000000000000000000000000000000000..117c7e8706c730334394135141c4b0e0104ca248 --- /dev/null +++ b/layouts/shortcodes/readfile.html @@ -0,0 +1,8 @@ +{{$file := .Get "file"}} +{{- if eq (.Get "markdown") "true" -}} +{{- $file | readFile | markdownify -}} +{{- else if (.Get "highlight") -}} +{{- highlight ($file | readFile) (.Get "highlight") "" -}} +{{- else -}} +{{ $file | readFile | safeHTML }} +{{- end -}} diff --git a/static/html/crane.html b/static/html/crane.html new file mode 100644 index 0000000000000000000000000000000000000000..938e512729f9766797d32621f1410bc80b2f2813 --- /dev/null +++ b/static/html/crane.html @@ -0,0 +1,36 @@ +<p><em>last generated Wed Nov 7 15:47:00 2018</em></p> + <table style="font-size: small"> + <colgroup> + <col style="width: 8.83281%;"/> + <col style="width: 11.9085%;"/> + <col style="width: 18.4543%;"/> + <col style="width: 18.3754%;"/> + <col style="width: 19.2429%;"/> + <col style="width: 7.57098%;"/> + <col style="width: 7.80757%;"/> + <col style="width: 7.80757%;"/> + </colgroup> + <tbody> + <tr style="font-size: x-small"> + <th>Partition</th> + <th>Owner</th> + <th>Node total (NODExCPU/MEM/FEATURE)</th> + <th>Description</th> + <th>SLURM Specification</span> </th> + <th>Max Job Run Time</th> + <th>Max CPUs Per User</th> + <th>Max Jobs Per User</th> + </tr> + + <tr> + <td colspan="1">batch</td> + <td colspan="1">shared</td> + <td colspan="1">399 total (377x16/64GB/ib; 6x36/256GB/opa; 16x36/512GB/opa)</td> + <td colspan="1">(default, no specification)</td> + <td colspan="1">#SBATCH --partition=batch</td> + <td colspan="1" style="text-align: center;" >7-00:00:00</td> + <td colspan="1" style="text-align: center;" >ib:2000 opa:576</td> + <td colspan="1" style="text-align: center;" >1000</td> + </tr> + </tbody> + </table> diff --git a/static/html/tusker.html b/static/html/tusker.html new file mode 100644 index 0000000000000000000000000000000000000000..fb1b42481185e26245793f8419b843a30a31c19b --- /dev/null +++ b/static/html/tusker.html @@ -0,0 +1,36 @@ +<p><em>last generated Wed Nov 7 15:14:27 2018</em></p> + <table style="font-size: small"> + <colgroup> + <col style="width: 8.83281%;"/> + <col style="width: 11.9085%;"/> + <col style="width: 18.4543%;"/> + <col style="width: 18.3754%;"/> + <col style="width: 19.2429%;"/> + <col style="width: 7.57098%;"/> + <col style="width: 7.80757%;"/> + <col style="width: 7.80757%;"/> + </colgroup> + <tbody> + <tr style="font-size: x-small"> + <th>Partition</th> + <th>Owner</th> + <th>Node total (NODExCPU/MEM/FEATURE)</th> + <th>Description</th> + <th>SLURM Specification</span> </th> + <th>Max Job Run Time</th> + <th>Max CPUs Per User</th> + <th>Max Jobs Per User</th> + </tr> + + <tr> + <td colspan="1" >batch</td> + <td colspan="1" >shared</td> + <td colspan="1" >38 total (38x64/256GB/none)</td> + <td colspan="1" >(default, no specification)</td> + <td colspan="1" >#SBATCH --partition=batch</td> + <td colspan="1" style="text-align: center;" >7-00:00:00</td> + <td colspan="1" style="text-align: center;" >2000</td> + <td colspan="1" style="text-align: center;" >1000</td> + </tr> + </tbody> + </table> diff --git a/static/images/21070053.png b/static/images/21070053.png new file mode 100644 index 0000000000000000000000000000000000000000..eb2f8aac1719410a14a038f951c6f143e1b99ca8 Binary files /dev/null and b/static/images/21070053.png differ diff --git a/static/images/21070054.png b/static/images/21070054.png new file mode 100644 index 0000000000000000000000000000000000000000..b26d60f7259f54eaa5779fb03a671db16a834581 Binary files /dev/null and b/static/images/21070054.png differ diff --git a/static/images/21070055.png b/static/images/21070055.png new file mode 100644 index 0000000000000000000000000000000000000000..b72cb9e8df9882962508b6a5ffece7120ff1f818 Binary files /dev/null and b/static/images/21070055.png differ diff --git a/static/images/21070056.png b/static/images/21070056.png new file mode 100644 index 0000000000000000000000000000000000000000..a17a4f506b8e86b7378aa0ba93de42fa158cd5f7 Binary files /dev/null and b/static/images/21070056.png differ diff --git a/static/images/21070057.png b/static/images/21070057.png new file mode 100644 index 0000000000000000000000000000000000000000..72096a04589579968e0e56354a5f99818eba60c5 Binary files /dev/null and b/static/images/21070057.png differ diff --git a/static/images/4980738.png b/static/images/4980738.png new file mode 100644 index 0000000000000000000000000000000000000000..f3415ccdf513b0299438f5b8a94769e689b7fed9 Binary files /dev/null and b/static/images/4980738.png differ