Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • FAQ
  • RDPv10
  • UNL_OneDrive
  • atticguidelines
  • data_share
  • globus-auto-backups
  • good-hcc-practice-rep-workflow
  • hchen2016-faq-home-is-full
  • ipynb-doc
  • master
  • rclone-fix
  • sislam2-master-patch-51693
  • sislam2-master-patch-86974
  • site_url
  • test
15 results

Target

Select target project
  • dweitzel2/hcc-docs
  • OMCCLUNG2/hcc-docs
  • salmandjing/hcc-docs
  • hcc/hcc-docs
4 results
Select Git revision
  • 26-add-screenshots-for-newer-rdp-v10-client
  • 28-overview-page-for-connecting-2
  • AddExamples
  • OMCCLUNG2-master-patch-74599
  • RDPv10
  • globus-auto-backups
  • gpu_update
  • master
  • mtanash2-master-patch-75717
  • mtanash2-master-patch-83333
  • mtanash2-master-patch-87890
  • mtanash2-master-patch-96320
  • patch-1
  • patch-2
  • patch-3
  • runTime
  • submitting-jobs-overview
  • tharvill1-master-patch-26973
18 results
Show changes
Show whitespace changes
Inline Side-by-side
Showing
with 0 additions and 2055 deletions
content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/velvet/running_velvet_with_single_end_and_paired_end_data.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Running Velvet with Single-End and Paired-End Data"
description = "How to run velvet with single-end and paired-end data on HCC resources"
weight = "10"
+++
## Running Velvet with Single-End and Paired-End short fasta data with k-mer=51, 8 CPUs and 100GB of RAM
The first step of running Velvet is to run **velveth**:
{{% panel header="`velveth.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
#SBATCH --job-name=Velvet_Velveth
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=168:00:00
#SBATCH --mem=10gb
#SBATCH --output=Velveth.%J.out
#SBATCH --error=Velveth.%J.err
module load velvet/1.2
export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE
velveth output_directory/ 51 -fasta -short input_reads.fasta -fasta -shortPaired2 -separate input_reads_pair_1.fasta input_reads_pair_2.fasta
{{< /highlight >}}
{{% /panel %}}
After running **velveth**, the next step is to run **velvetg** on the `output_directory/` and files generated from **velveth**:
{{% panel header="`velvetg.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
#SBATCH --job-name=Velvet_Velvetg
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=168:00:00
#SBATCH --mem=100gb
#SBATCH --output=Velvetg.%J.out
#SBATCH --error=Velvetg.%J.err
module load velvet/1.2
export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE
velvetg output_directory/ -min_contig_lgth 200
{{< /highlight >}}
{{% /panel %}}
Both **velveth** and **velvetg** are multi-threaded.
### Velvet Output
{{% panel header="`Output directory after velveth`"%}}
{{< highlight bash >}}
$ ls output_directory/
Log Roadmaps Sequences
{{< /highlight >}}
{{% /panel %}}
{{% panel header="`Output directory after velvetg`"%}}
{{< highlight bash >}}
$ ls output_directory/
contigs.fa Graph LastGraph Log PreGraph Roadmaps Sequences stats.txt
{{< /highlight >}}
{{% /panel %}}
The output fasta file `contigs.fa` is the final Velvet output that contains the assembled contigs. More information about the output files is provided in the Velvet manual.
content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/velvet/running_velvet_with_single_end_data.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Running Velvet with Single-End Data"
description = "How to run velvet with single-end data on HCC resources"
weight = "10"
+++
## Running Velvet with Single-End short fasta data with k-mer=31, 8 CPUs and 100GB of RAM
The first step of running Velvet is to run **velveth**:
{{% panel header="`velveth.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
#SBATCH --job-name=Velvet_Velveth
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=168:00:00
#SBATCH --mem=10gb
#SBATCH --output=Velveth.%J.out
#SBATCH --error=Velveth.%J.err
module load velvet/1.2
export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE
velveth output_directory/ 31 -fasta -short input_reads.fasta
{{< /highlight >}}
{{% /panel %}}
After running **velveth**, the next step is to run **velvetg** on the `output_directory/` and files generated from **velveth**:
{{% panel header="`velvetg.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
#SBATCH --job-name=Velvet_Velvetg
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=168:00:00
#SBATCH --mem=100gb
#SBATCH --output=Velvetg.%J.out
#SBATCH --error=Velvetg.%J.err
module load velvet/1.2
export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE
velvetg output_directory/ -min_contig_lgth 200
{{< /highlight >}}
{{% /panel %}}
Both **velveth** and **velvetg** are multi-threaded.
### Velvet Output
{{% panel header="`Output directory after velveth`"%}}
{{< highlight bash >}}
$ ls output_directory/
Log Roadmaps Sequences
{{< /highlight >}}
{{% /panel %}}
{{% panel header="`Output directory after velvetg`"%}}
{{< highlight bash >}}
$ ls output_directory/
contigs.fa Graph LastGraph Log PreGraph Roadmaps Sequences stats.txt
{{< /highlight >}}
{{% /panel %}}
The output fasta file `contigs.fa` is the final Velvet output that contains the assembled contigs. More information about the output files is provided in the Velvet manual.
content/guides/running_applications/bioinformatics_tools/downloading_sra_data_from_ncbi.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Downloading SRA data from NCBI"
description = "How to download data from NCBI"
weight = "52"
+++
One way to download high-volume data from NCBI is to use command line
utilities, such as **wget**, **ftp** or Aspera Connect **ascp**
plugin. The Aspera Connect plugin is commonly used high-performance transfer
plugin that provides the best transfer speed.
This plugin is available on our clusters as a module. In order to use it, load the appropriate module first:
{{< highlight bash >}}
$ module load aspera-cli
{{< /highlight >}}
The basic usage of the Aspera plugin is
{{< highlight bash >}}
$ ascp -i $ASPERA_PUBLIC_KEY -k 1 -T -l <max_download_rate_in_Mbps>m anonftp@ftp.ncbi.nlm.nih.gov:/<files_to_transfer> <local_work_output_directory>
{{< /highlight >}}
where **-k 1** enables resume of partial transfers, **-T** disables encryption for maximum throughput, and **-l** sets the transfer rate.
**\<files_to_transfer\>** mentioned in the basic usage of Aspera
plugin has a specifically defined pattern that needs to be followed:
{{< highlight bash >}}
<files_to_transfer> = /sra/sra-instant/reads/ByRun/sra/SRR|ERR|DRR/<first_6_characters_of_accession>/<accession>/<accession>.sra
{{< /highlight >}}
where **SRR\|ERR\|DRR** should be either **SRR**, **ERR **or **DRR** and should match the prefix of the target **.sra** file.
More **ascp** options can be seen by using:
{{< highlight bash >}}
$ ascp --help
{{< /highlight >}}
For example, if you want to download the **SRR304976** file from NCBI in your $WORK **data/** directory with downloading speed of **1000 Mbps**, you should use the following command:
{{< highlight bash >}}
$ ascp -i $ASPERA_PUBLIC_KEY -k 1 -T -l 1000m anonftp@ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR304/SRR304976/SRR304976.sra /work/[groupname]/[username]/data/
{{< /highlight >}}
content/guides/running_applications/bioinformatics_tools/pre_processing_tools/_index.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Pre-processing Tools"
description = "How to use pre-processing tools on HCC machines"
weight = "52"
+++
{{% children %}}
\ No newline at end of file
content/guides/running_applications/bioinformatics_tools/reference_based_assembly_tools/_index.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Reference-Based Assembly Tools"
description = "How to use reference based assembly tools on HCC machines"
weight = "52"
+++
{{% children %}}
\ No newline at end of file
content/guides/running_applications/bioinformatics_tools/removing_detecting_redundant_sequences/_index.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Tools for Removing/Detecting Redundant Sequences"
description = "How to use tools for removing/detecting redundant sequences on HCC machines"
weight = "52"
+++
{{% children %}}
content/guides/running_applications/module_commands.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Module Commands"
description = "How to use the module utility on HCC resources."
+++
`module` commands provide an HPC system user the capability to compile
into their source code using any type of library that is
available on the server. The `module` command gives each user the
ability to modify their environmental `PATH` and `LD_LIBRARY_PATH`
variables.
{{% notice info %}}
Please note that if you compile your application using a particular
module, you must include the appropriate module load statement in your
submit script.
{{% /notice %}}
### List Modules Loaded
{{% panel theme="info" header="Example Usage: module list" %}}
{{< highlight bash >}}
module list
No Modulefiles Currently Loaded.
echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
{{< /highlight >}}
{{% /panel %}}
### List Modules Available
{{% panel theme="info" header="Example Usage: Listing Available Modules" %}}
{{< highlight bash >}}
module avail
---------------------------------------------- /util/opt/Modules/modulefiles ----------------------------------------------
NCL/6.0 bowtie/2.0.0-beta6 compiler/pgi/12 hdfeos5/1.14 mplus/7.0 szip/2.1
NCL/6.0dist compiler/gcc/4.6 cufflinks/2.0.2 hugeseq/1.0 netcdf/4.1 tophat/2.0.5
NCO/4.1 compiler/gcc/4.7 deprecated intel-mkl/11 netcdf/4.2 udunits/2.1
R/2.15 compiler/intel/11 hdf4/4.2 intel-mkl/12 openmpi/1.5 zlib/1.2
WRF/WRF compiler/intel/12 hdf5/1.8 lsdyna/5.1.1 openmpi/1.6
acml/5.1 compiler/open64/4.5 hdf5/1.8.6 lsdyna/6.0.0 samtools/0.1
bowtie/0.12.8 compiler/pgi/11 hdfeos2/2.18 mplus/6.12 sas/9.3
{{< /highlight >}}
{{% /panel %}}
#### module load \<module-name\>
Places the binaries and libraries for \<module-name\> into your `PATH` and `LD_LIBRARY_PATH`.
{{% panel theme="info" header="Example Usage: Loading Desired Module" %}}
{{< highlight bash >}}
module load compiler/pgi/11
module list
Currently Loaded Modulefiles:
1) compiler/pgi/11
echo $PATH
/util/comp/pgi/linux86-64/11/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
{{< /highlight >}}
{{% /panel %}}
#### module unload \<module-name\>
Removes the binaries and libraries associated with \<module-name\> from your PATH and `LD_LIBRARY_PATH`.
{{% panel theme="info" header="Example Usage: module unload" %}}
{{< highlight bash >}}
module unload compiler/pgi/11
module list
No Modulefiles Currently Loaded.
echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
{{< /highlight >}}
{{% /panel %}}
#### module purge
**Purges** all previously **loaded** module libraries and binaries from
your `PATH` and `LD_LIBRARY_PATH`.
{{% panel theme="info" header="Example Usage: module purge" %}}
{{< highlight bash >}}
module load compiler/open64
module load zlib/1.2
module list
Currently Loaded Modulefiles:
1) zlib/1.2 2) compiler/open64/4.5
module purge
module list
No Modulefiles Currently Loaded.
{{< /highlight >}}
{{% /panel %}}
#### module help
To see a complete list of module commands/options.
**Example Usage: module help**
{{% panel theme="info" header="Example Usage: module help" %}}
{{< highlight bash >}}
Usage: module [options] sub-command [args ...]
Options:
-h -? -H --help This help message
-s availStyle --style=availStyle Site controlled avail style: system en_grouped (default: en_grouped)
--regression_testing Lmod regression testing
-D Program tracing written to stderr
--debug=dbglvl Program tracing written to stderr
--pin_versions=pinVersions When doing a restore use specified version, do not follow defaults
-d --default List default modules only when used with avail
-q --quiet Do not print out warnings
--expert Expert mode
-t --terse Write out in machine readable format for commands: list, avail, spider, savelist
--initial_load loading Lmod for first time in a user shell
--latest Load latest (ignore default)
--ignore_cache Treat the cache file(s) as out-of-date
--novice Turn off expert and quiet flag
--raw Print modulefile in raw output when used with show
-w twidth --width=twidth Use this as max term width
-v --version Print version info and quit
-r --regexp use regular expression match
--gitversion Dump git version in a machine readable way and quit
--dumpversion Dump version in a machine readable way and quit
--check_syntax --checkSyntax Checking module command syntax: do not load
--config Report Lmod Configuration
--config_json Report Lmod Configuration in json format
--mt Report Module Table State
--timer report run times
--force force removal of a sticky module or save an empty collection
--redirect Send the output of list, avail, spider to stdout (not stderr)
--no_redirect Force output of list, avail and spider to stderr
--show_hidden Avail and spider will report hidden modules
--spider_timeout=timeout a timeout for spider
-T --trace
module [options] sub-command [args ...]
Help sub-commands:
------------------
help prints this message
help module [...] print help message from module(s)
Loading/Unloading sub-commands:
-------------------------------
load | add module [...] load module(s)
try-load | try-add module [...] Add module(s), do not complain if not found
del | unload module [...] Remove module(s), do not complain if not found
swap | sw | switch m1 m2 unload m1 and load m2
purge unload all modules
refresh reload aliases from current list of modules.
update reload all currently loaded modules.
Listing / Searching sub-commands:
---------------------------------
list List loaded modules
list s1 s2 ... List loaded modules that match the pattern
avail | av List available modules
avail | av string List available modules that contain "string".
spider List all possible modules
spider module List all possible version of that module file
spider string List all module that contain the "string".
spider name/version Detailed information about that version of the module.
whatis module Print whatis information about module
keyword | key string Search all name and whatis that contain "string".
Searching with Lmod:
--------------------
All searching (spider, list, avail, keyword) support regular expressions:
spider -r '^p' Finds all the modules that start with `p' or `P'
spider -r mpi Finds all modules that have "mpi" in their name.
spider -r 'mpi$ Finds all modules that end with "mpi" in their name.
Handling a collection of modules:
--------------------------------
save | s Save the current list of modules to a user defined "default" collection.
save | s name Save the current list of modules to "name" collection.
reset The same as "restore system"
restore | r Restore modules from the user's "default" or system default.
restore | r name Restore modules from "name" collection.
restore system Restore module state to system defaults.
savelist List of saved collections.
describe | mcc name Describe the contents of a module collection.
Deprecated commands:
--------------------
getdefault [name] load name collection of modules or user's "default" if no name given.
===> Use "restore" instead <====
setdefault [name] Save current list of modules to name if given, otherwise save as the default list for you the user.
===> Use "save" instead. <====
Miscellaneous sub-commands:
---------------------------
show modulefile show the commands in the module file.
use [-a] path Prepend or Append path to MODULEPATH.
unuse path remove path from MODULEPATH.
tablelist output list of active modules as a lua table.
Important Environment Variables:
--------------------------------
LMOD_COLORIZE If defined to be "YES" then Lmod prints properties and warning in color.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Lmod Web Sites
Documentation: http://lmod.readthedocs.org
Github: https://github.com/TACC/Lmod
Sourceforge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod
To report a bug please read http://lmod.readthedocs.io/en/latest/075_bug_reporting.html
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Modules based on Lua: Version 7.4.16 2017-05-23 11:10 -05:00
by Robert McLay mclay@tacc.utexas.edu
{{< /highlight >}}
{{% /panel %}}
content/guides/running_applications/mpi_jobs_on_hcc.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "MPI Jobs on HCC"
description = "How to compile and run MPI programs on HCC machines"
weight = "52"
+++
This quick start demonstrates how to implement a parallel (MPI)
Fortran/C program on HCC supercomputers. The sample codes and submit
scripts can be downloaded from [mpi_dir.zip](/attachments/mpi_dir.zip).
#### Login to a HCC Cluster
Log in to a HCC cluster through PuTTY ([For Windows Users]({{< relref "/quickstarts/connecting/for_windows_users">}})) or Terminal ([For Mac/Linux
Users]({{< relref "/quickstarts/connecting/for_maclinux_users">}})) and make a subdirectory called `mpi_dir` under the `$WORK` directory.
{{< highlight bash >}}
$ cd $WORK
$ mkdir mpi_dir
{{< /highlight >}}
In the subdirectory `mpi_dir`, save all the relevant codes. Here we
include two demo programs, `demo_f_mpi.f90` and `demo_c_mpi.c`, that
compute the sum from 1 to 20 through parallel processes. A
straightforward parallelization scheme is used for demonstration
purpose. First, the master core (i.e. `myid=0`) distributes equal
computation workload to a certain number of cores (as specified by
`--ntasks `in the submit script). Then, each worker core computes a
partial summation as output. Finally, the master core collects the
outputs from all worker cores and perform an overall summation. For easy
comparison with the serial code ([Fortran/C on HCC]({{< relref "fortran_c_on_hcc">}})), the
added lines in the parallel code (MPI) are marked with "!=" or "//=".
{{%expand "demo_f_mpi.f90" %}}
{{< highlight fortran >}}
Program demo_f_mpi
!====== MPI =====
use mpi
!================
implicit none
integer, parameter :: N = 20
real*8 w
integer i
common/sol/ x
real*8 x
real*8, dimension(N) :: y
!============================== MPI =================================
integer ind
real*8, dimension(:), allocatable :: y_local
integer numnodes,myid,rc,ierr,start_local,end_local,N_local
real*8 allsum
!====================================================================
!============================== MPI =================================
call mpi_init( ierr )
call mpi_comm_rank ( mpi_comm_world, myid, ierr )
call mpi_comm_size ( mpi_comm_world, numnodes, ierr )
!
N_local = N/numnodes
allocate ( y_local(N_local) )
start_local = N_local*myid + 1
end_local = N_local*myid + N_local
!====================================================================
do i = start_local, end_local
w = i*1d0
call proc(w)
ind = i - N_local*myid
y_local(ind) = x
! y(i) = x
! write(6,*) 'i, y(i)', i, y(i)
enddo
! write(6,*) 'sum(y) =',sum(y)
!============================================== MPI =====================================================
call mpi_reduce( sum(y_local), allsum, 1, mpi_real8, mpi_sum, 0, mpi_comm_world, ierr )
call mpi_gather ( y_local, N_local, mpi_real8, y, N_local, mpi_real8, 0, mpi_comm_world, ierr )
if (myid == 0) then
write(6,*) '-----------------------------------------'
write(6,*) '*Final output from... myid=', myid
write(6,*) 'numnodes =', numnodes
write(6,*) 'mpi_sum =', allsum
write(6,*) 'y=...'
do i = 1, N
write(6,*) y(i)
enddo
write(6,*) 'sum(y)=', sum(y)
endif
deallocate( y_local )
call mpi_finalize(rc)
!========================================================================================================
Stop
End Program
Subroutine proc(w)
real*8, intent(in) :: w
common/sol/ x
real*8 x
x = w
Return
End Subroutine
{{< /highlight >}}
{{% /expand %}}
{{%expand "demo_c_mpi.c" %}}
{{< highlight c >}}
//demo_c_mpi
#include <stdio.h>
//======= MPI ========
#include "mpi.h"
#include <stdlib.h>
//====================
double proc(double w){
double x;
x = w;
return x;
}
int main(int argc, char* argv[]){
int N=20;
double w;
int i;
double x;
double y[N];
double sum;
//=============================== MPI ============================
int ind;
double *y_local;
int numnodes,myid,rc,ierr,start_local,end_local,N_local;
double allsum;
//================================================================
//=============================== MPI ============================
MPI_Init(&argc, &argv);
MPI_Comm_rank( MPI_COMM_WORLD, &myid );
MPI_Comm_size ( MPI_COMM_WORLD, &numnodes );
N_local = N/numnodes;
y_local=(double *) malloc(N_local*sizeof(double));
start_local = N_local*myid + 1;
end_local = N_local*myid + N_local;
//================================================================
for (i = start_local; i <= end_local; i++){
w = i*1e0;
x = proc(w);
ind = i - N_local*myid;
y_local[ind-1] = x;
// y[i-1] = x;
// printf("i,x= %d %lf\n", i, y[i-1]) ;
}
sum = 0e0;
for (i = 1; i<= N_local; i++){
sum = sum + y_local[i-1];
}
// printf("sum(y)= %lf\n", sum);
//====================================== MPI ===========================================
MPI_Reduce( &sum, &allsum, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD );
MPI_Gather( &y_local[0], N_local, MPI_DOUBLE, &y[0], N_local, MPI_DOUBLE, 0, MPI_COMM_WORLD );
if (myid == 0){
printf("-----------------------------------\n");
printf("*Final output from... myid= %d\n", myid);
printf("numnodes = %d\n", numnodes);
printf("mpi_sum = %lf\n", allsum);
printf("y=...\n");
for (i = 1; i <= N; i++){
printf("%lf\n", y[i-1]);
}
sum = 0e0;
for (i = 1; i<= N; i++){
sum = sum + y[i-1];
}
printf("sum(y) = %lf\n", sum);
}
free( y_local );
MPI_Finalize ();
//======================================================================================
return 0;
}
{{< /highlight >}}
{{% /expand %}}
---
#### Compiling the Code
The compiling of a MPI code requires first loading a compiler "engine"
such as `gcc`, `intel`, or `pgi` and then loading a MPI wrapper
`openmpi`. Here we will use the GNU Complier Collection, `gcc`, for
demonstration.
{{< highlight bash >}}
$ module load compiler/gcc/6.1 openmpi/2.1
$ mpif90 demo_f_mpi.f90 -o demo_f_mpi.x
$ mpicc demo_c_mpi.c -o demo_c_mpi.x
{{< /highlight >}}
The above commends load the `gcc` complier with the `openmpi` wrapper.
The compiling commands `mpif90` or `mpicc` are used to compile the codes
to`.x` files (executables).
### Creating a Submit Script
Create a submit script to request 5 cores (with `--ntasks`). A parallel
execution command `mpirun ./` needs to enter to last line before the
main program name.
{{% panel header="`submit_f.mpi`"%}}
{{< highlight bash >}}
#!/bin/sh
#SBATCH --ntasks=5
#SBATCH --mem-per-cpu=1024
#SBATCH --time=00:01:00
#SBATCH --job-name=Fortran
#SBATCH --error=Fortran.%J.err
#SBATCH --output=Fortran.%J.out
mpirun ./demo_f_mpi.x
{{< /highlight >}}
{{% /panel %}}
{{% panel header="`submit_c.mpi`"%}}
{{< highlight bash >}}
#!/bin/sh
#SBATCH --ntasks=5
#SBATCH --mem-per-cpu=1024
#SBATCH --time=00:01:00
#SBATCH --job-name=C
#SBATCH --error=C.%J.err
#SBATCH --output=C.%J.out
mpirun ./demo_c_mpi.x
{{< /highlight >}}
{{% /panel %}}
#### Submit the Job
The job can be submitted through the command `sbatch`. The job status
can be monitored by entering `squeue` with the `-u` option.
{{< highlight bash >}}
$ sbatch submit_f.mpi
$ sbatch submit_c.mpi
$ squeue -u <username>
{{< /highlight >}}
Replace `<username>` with your HCC username.
Sample Output
-------------
The sum from 1 to 20 is computed and printed to the `.out` file (see
below). The outputs from the 5 cores are collected and processed by the
master core (i.e. `myid=0`).
{{%expand "Fortran.out" %}}
{{< highlight batchfile>}}
-----------------------------------------
*Final output from... myid= 0
numnodes = 5
mpi_sum = 210.00000000000000
y=...
1.0000000000000000
2.0000000000000000
3.0000000000000000
4.0000000000000000
5.0000000000000000
6.0000000000000000
7.0000000000000000
8.0000000000000000
9.0000000000000000
10.000000000000000
11.000000000000000
12.000000000000000
13.000000000000000
14.000000000000000
15.000000000000000
16.000000000000000
17.000000000000000
18.000000000000000
19.000000000000000
20.000000000000000
sum(y)= 210.00000000000000
{{< /highlight >}}
{{% /expand %}}
{{%expand "C.out" %}}
{{< highlight batchfile>}}
-----------------------------------
*Final output from... myid= 0
numnodes = 5
mpi_sum = 210.000000
y=...
1.000000
2.000000
3.000000
4.000000
5.000000
6.000000
7.000000
8.000000
9.000000
10.000000
11.000000
12.000000
13.000000
14.000000
15.000000
16.000000
17.000000
18.000000
19.000000
20.000000
sum(y) = 210.000000
{{< /highlight >}}
{{% /expand %}}
content/guides/running_applications/running_ocean_land_atmosphere_model_olam.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Running OLAM at HCC"
description = "How to run the OLAM (Ocean Land Atmosphere Model) on HCC resources."
+++
### OLAM compilation on Tusker
##### pgi/11 compilation with mpi and openmp enabled
1. Load modules:
{{< highlight bash >}}
module load compiler/pgi/11 openmpi/1.6 szip/2.1 zlib/1.2 NCL/6.1dist
{{< /highlight >}}
2. Edit the `include.mk` file.
{{% panel theme="info" header="include.mk" %}}
{{< highlight batch >}}
#----------------- LINUX Intel Fortran ifort/gcc ---------------
F_COMP=mpif90
# If the compiler supports (and the user wants to use)
# the module IEEE_ARITHMETIC, uncomment below
IEEE_ARITHMETIC=yes
# If using MPI libraries:
OLAM_MPI=yes
# If parallel hdf5 is supported, uncomment the next line
OLAM_PARALLEL_HDF5=yes
# If you use the ED2 model, uncomment the next line
#USE_ED2=yes
MPI_PATH=/util/opt/openmpi/1.6/pgi/11
PAR_INCS=-I$(MPI_PATH)/include:$(MPI_PATH)/lib
PAR_LIBS=-L$(MPI_PATH)/lib -lmpi
# OPTIMIZED:
F_OPTS=-O3 -traceback -mp
#F_OPTS=-xHost -O3 -fno-alias -ip -openmp -traceback
#F_OPTS=-g -O3 -xHost -traceback
# DEBUG:
#F_OPTS=-g -fp-model precise -check bounds -traceback \
# -debug extended -check uninit -ftrapuv
# FORTRAN FLAGS FOR BIG FILES WHICH WOULD HAVE EXCESSIVE COMPILATION TIME
#SLOW_FFLAGS=-O1 -g -no-ip -traceback
C_COMP=mpicc
#C_COMP=mpicc
C_OPTS=-DUNDERSCORE -DLITTLE
NCARG_DIR=/util/src/ncl_ncarg/ncl_ncarg-6.1.2/lib
LIBNCARG=-L$(NCARG_DIR) -lncarg -lncarg_gks -lncarg_c \
-L/usr/lib64 -lX11 -ldl -lpthread -lgfortran -lcairo
HDF5_LIBS=-L/util/opt/hdf5/1.8.13/openmpi/1.6/pgi/11/lib -lhdf5_fortran -lhdf5 -lz -lm
HDF5_INCS=-I/util/opt/hdf5/1.8.13/openmpi/1.6/pgi/11/include
NETCDF_LIBS=-L/util/opt/netcdf/4.2/pgi/11/lib -lnetcdf
NETCDF_INCS=-I/util/opt/netcdf/4.2/pgi/11/include
LOADER=$(F_COMP)
LOADER_OPTS=-mp
#LOADER_OPTS=-static-intel $(F_OPTS)
# For Apple OSX: the stack size needs to be increased at link time
# LOADER_OPTS=-static-intel $(F_OPTS) -Wl,-stack_size -Wl,0x10000000
# to allow ifort compiler to link with pg-compiled ncar graphics:
# LIBS=-z muldefs -L/opt/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc
## IMPORTANT: Need to specify this flag in ED2
#USE_HDF5=1
{{< /highlight >}}
{{% /panel %}}
3. Command: `make clean`
4. Command: `make -j 8`
##### intel/12 compilation with mpi and openmp enabled
1. Load modules:
{{< highlight bash >}}
module load compiler/intel/12 openmpi/1.6 szip/2.1 zlib/1.2
{{< /highlight >}}
2. Edit the `include.mk` file.
{{% panel theme="info" header="include.mk" %}}
{{< highlight batch >}}
#----------------- LINUX Intel Fortran ifort/gcc ---------------
F_COMP=mpif90
# If the compiler supports (and the user wants to use)
# the module IEEE_ARITHMETIC, uncomment below
IEEE_ARITHMETIC=yes
# If using MPI libraries:
OLAM_MPI=yes
# If parallel hdf5 is supported, uncomment the next line
OLAM_PARALLEL_HDF5=yes
# If you use the ED2 model, uncomment the next line
#USE_ED2=yes
MPI_PATH=/util/opt/openmpi/1.6/intel/12
PAR_INCS=-I$(MPI_PATH)/include:$(MPI_PATH)/lib
PAR_LIBS=-L$(MPI_PATH)/lib -lmpi
# OPTIMIZED:
F_OPTS=-O3 -traceback -openmp
#F_OPTS=-xHost -O3 -fno-alias -ip -openmp -traceback
#F_OPTS=-g -O3 -xHost -traceback
# DEBUG:
#F_OPTS=-g -fp-model precise -check bounds -traceback \
# -debug extended -check uninit -ftrapuv
# FORTRAN FLAGS FOR BIG FILES WHICH WOULD HAVE EXCESSIVE COMPILATION TIME
#SLOW_FFLAGS=-O1 -g -no-ip -traceback
C_COMP=mpicc
#C_COMP=mpicc
C_OPTS=-DUNDERSCORE -DLITTLE
NCARG_DIR=/util/src/ncl_ncarg/ncl_ncarg-6.1.2/lib
LIBNCARG=-L$(NCARG_DIR) -lncarg -lncarg_gks -lncarg_c \
-L/usr/lib64 -lX11 -ldl -lpthread -lgfortran -lcairo
HDF5_LIBS=-L/util/opt/hdf5/1.8.13/openmpi/1.6/intel/12/lib -lhdf5_fortran -lhdf5 -lz -lm
HDF5_INCS=-I/util/opt/hdf5/1.8.13/openmpi/1.6/intel/12/include
NETCDF_LIBS=-L/util/opt/netcdf/4.2/intel/12/lib -lnetcdf
NETCDF_INCS=-I/util/opt/netcdf/4.2/intel/12/include
LOADER=$(F_COMP)
LOADER_OPTS=-openmp
#LOADER_OPTS=-static-intel $(F_OPTS)
# For Apple OSX: the stack size needs to be increased at link time
# LOADER_OPTS=-static-intel $(F_OPTS) -Wl,-stack_size -Wl,0x10000000
# to allow ifort compiler to link with pg-compiled ncar graphics:
# LIBS=-z muldefs -L/opt/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc
## IMPORTANT: Need to specify this flag in ED2
#USE_HDF5=1
{{< /highlight >}}
{{% /panel %}}
3. Command: `make clean`
4. Command: `make -j 8`
### OLAM compilation on Crane
##### Intel/15 compiler with OpenMPI/1.10
1. Load modules:
{{< highlight bash >}}
module load compiler/intel/15 openmpi/1.10 NCL/6.1 netcdf/4.4 phdf5/1.8 szip/2.1 zlib/1.2
{{< /highlight >}}
2. Edit the `include.mk` file:
{{% panel theme="info" header="include.mk" %}}
{{< highlight batch >}}
#----------------- LINUX Intel Fortran ifort/gcc ---------------
F_COMP=/util/opt/hdf5/1.8/openmpi/1.10/intel/15/bin/h5pfc
# If the compiler supports (and the user wants to use)
# the module IEEE_ARITHMETIC, uncomment below
IEEE_ARITHMETIC=yes
# If using MPI libraries:
OLAM_MPI=yes
# If parallel hdf5 is supported, uncomment the next line
OLAM_PARALLEL_HDF5=yes
# If you use the ED2 model, uncomment the next line
#USE_ED2=yes
#MPI_PATH=/usr/local/mpich
PAR_INCS=-I/util/opt/openmpi/1.10/intel/15/include
PAR_LIBS=-L/util/opt/openmpi/1.10/intel/15/lib
# OPTIMIZED:
F_OPTS=-xHost -O3 -fno-alias -ip -openmp -traceback
#F_OPTS=-g -O3 -xHost -traceback
# DEBUG:
#F_OPTS=-g -fp-model precise -check bounds -traceback \
# -debug extended -check uninit -ftrapuv
# EXTRA OPTIONS FOR FIXED-SOURCE CODE
FIXED_SRC_FLAGS=-fixed -132
# FORTRAN FLAGS FOR BIG FILES WHICH WOULD HAVE EXCESSIVE COMPILATION TIME
SLOW_FFLAGS=-O1 -g -no-ip -traceback
#C_COMP=icc
C_COMP=mpicc
C_OPTS=-O3 -DUNDERSCORE -DLITTLE
NCARG_DIR=/util/opt/NCL/6.1/lib
LIBNCARG=-L$(NCARG_DIR) -lncarg -lncarg_gks -lncarg_c \
-L/usr/lib64 -lX11 -ldl -lpng -lpthread -lgfortran -lcairo
HDF5_LIBS=-L/util/opt/hdf5/1.8/openmpi/1.10/intel/15/lib
HDF5_INCS=-I/util/opt/hdf5/1.8/openmpi/1.10/intel/15/include
NETCDF_LIBS=-L/util/opt/netcdf/4.4/intel/15/lib -lnetcdf
NETCDF_INCS=-I/util/opt/netcdf/4.4/intel/15/include
LOADER=$(F_COMP)
LOADER_OPTS=-static-intel $(F_OPTS)
# For Apple OSX: the stack size needs to be increased at link time
# LOADER_OPTS=-static-intel $(F_OPTS) -Wl,-stack_size -Wl,0x10000000
# to allow ifort compiler to link with pg-compiled ncar graphics:
# LIBS=-z muldefs -L/opt/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc
## IMPORTANT: Need to specify this flag in ED2
USE_HDF5=1
{{< /highlight >}}
{{% /panel %}}
3. Command: `make clean`
4. Command: `make -j 8`
### Sample SLURM submit scripts
##### PGI compiler:
{{% panel theme="info" header="Sample submit script for PGI compiler" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --ntasks=8 # 8 cores
#SBATCH --mem-per-cpu=1024 # Minimum memory required per CPU (in megabytes)
#SBATCH --time=03:15:00 # Run time in hh:mm:ss
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load compiler/pgi/11 openmpi/1.6 szip/2.1 zlib/1.2
mpirun /path/to/olam-4.2c-mpi
{{< /highlight >}}
{{% /panel %}}
##### Intel compiler:
{{% panel theme="info" header="Sample submit script for Intel compiler" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --ntasks=8 # 8 cores
#SBATCH --mem-per-cpu=1024 # Minimum memory required per CPU (in megabytes)
#SBATCH --time=03:15:00 # Run time in hh:mm:ss
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load compiler/intel/12 openmpi/1.6 szip/2.1 zlib/1.2
mpirun /path/to/olam-4.2c-mpi
{{< /highlight >}}
{{% /panel %}}
content/guides/running_applications/running_theano.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Running Theano"
description = "How to run the Theano on HCC resources."
+++
Theano is available on HCC resources via the modules system. CPU
versions are available on Tusker; both CPU and GPU
versions are available on Crane. Additionally, installs for both Python
2.7 and 3.6 are provided.
### Initial Setup
Theano attempts to write to a `~/.theano` directory in some
circumstances, which can cause errors as the `/home` filesystem is
read-only on HCC machines. As a workaround, create the directory on
`/work` and make a symlink from `/home`:
{{% panel theme="info" header="Create & symlink .theano directory" %}}
{{< highlight bash >}}
mkdir -p $WORK/.theano
ln -s $WORK/.theano $HOME/.theano
{{< /highlight >}}
{{% /panel %}}
This only needs to be done once on each HCC machine.
### Running the CPU version
To use the CPU version, simply load the module and run your Python code.
You can choose between the Python 2.7, 3.5 or 3.6 environments:
{{% panel theme="info" header="Python 2.7 version" %}}
{{< highlight bash >}}
module load theano/py27/1.0
python my_python2_script.py
{{< /highlight >}}
{{% /panel %}}
or
{{% panel theme="info" header="Python 3.5 version" %}}
{{< highlight bash >}}
module load theano/py35/1.0
python my_python3_script.py
{{< /highlight >}}
{{% /panel %}}
or
{{% panel theme="info" header="Python 3.6 version" %}}
{{< highlight bash >}}
module load theano/py36/1.0
python my_python3_script.py
{{< /highlight >}}
{{% /panel %}}
### Running the GPU version
To use the GPU version, first create a `~/.theanorc` file with the
following contents (or append to an existing file as needed):
{{% panel theme="info" header="~/.theanorc" %}}
{{< highlight batch >}}
[global]
device = cuda
{{< /highlight >}}
{{% /panel %}}
Next, load the theano module:
{{% panel theme="info" header="Load the theano module" %}}
{{< highlight bash >}}
module load theano/py27/0.9
{{< /highlight >}}
{{% /panel %}}
To test the GPU support, start an interactive job on a GPU node and
import the theano module within the Python interpreter. You should see
output similar to the following:
{{% panel theme="info" header="GPU support test" %}}
{{< highlight python >}}
Python 2.7.15 | packaged by conda-forge | (default, May 8 2018, 14:46:53)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
Using cuDNN version 7005 on context None
Mapped name None to device cuda: Tesla K20m (0000:03:00.0)
{{< /highlight >}}
{{% /panel %}}
content/guides/running_applications/using_anaconda_package_manager.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Using Anaconda Package Manager"
description = "How to use the Anaconda Package Manager on HCC resources."
+++
[Anaconda](https://www.anaconda.com/what-is-anaconda),
from [Anaconda, Inc](https://www.anaconda.com)
is a completely free enterprise-ready distribution for large-scale data
processing, predictive analytics, and scientific computing. It includes
over 195 of the most popular Python packages for science, math,
engineering, and data analysis. **It also offers the ability to easily
create custom _environments_ by mixing and matching different versions
of Python and/or R and other packages into isolated environments that
individual users are free to create.** Anaconda includes the `conda`
package and environment manager to make managing these environments
straightforward.
- [Using Anaconda](#using-anaconda)
- [Creating custom Anaconda Environment](#creating-custom-anaconda-environment)
- [Creating custom GPU Anaconda Environment](#creating-custom-gpu-anaconda-environment)
- [Adding Packages to an Existing Environment](#adding-packages-to-an-existing-environment)
- [Using an Anaconda Environment in a Jupyter Notebook on Crane](#using-an-anaconda-environment-in-a-jupyter-notebook-on-crane)
### Using Anaconda
While the standard methods of installing packages via `pip`
and `easy_install` work with Anaconda, the preferred method is using
the `conda` command.
{{% notice info %}}
Full documentation on using Conda is available
at http://conda.pydata.org/docs/
A [cheatsheet](/attachments/11635089.pdf) is also provided.
{{% /notice %}}
A few examples of the basic commands are provided here. For a full
explanation of all of Anaconda/Conda's capabilities, see the
documentation linked above.
Anaconda is provided through the `anaconda` module on HCC machines. To
begin using it, load the Anaconda module.
{{% panel theme="info" header="Load the Anaconda module to start using Conda" %}}
{{< highlight bash >}}
module load anaconda
{{< /highlight >}}
{{% /panel %}}
To display general information about Conda/Anaconda, use the `info` subcommand.
{{% panel theme="info" header="Display general information about Conda/Anaconda" %}}
{{< highlight bash >}}
conda info
{{< /highlight >}}
{{% /panel %}}
Conda allows the easy creation of isolated, custom environments with
packages and versions of your choosing. To show all currently available
environments, and which is active, use the `info `subcommand with the
`-e` option.
{{% panel theme="info" header="List available environments" %}}
{{< highlight bash >}}
conda info -e
{{< /highlight >}}
{{% /panel %}}
The active environment will be marked with an asterisk (\*) character.
The `list` command will show all packages installed
in the currently active environment.
{{% panel theme="info" header="List installed packages in current environment" %}}
{{< highlight bash >}}
conda list
{{< /highlight >}}
{{% /panel %}}
To find the names of packages, use the `search` subcommand.
{{% panel theme="info" header="Search for packages" %}}
{{< highlight bash >}}
conda search numpy
{{< /highlight >}}
{{% /panel %}}
If the package is available, this will also display available package
versions and compatible Python versions the package may be installed
under.
### Creating Custom Anaconda Environment
The `create` command is used to create a new environment. It requires
at a minimum a name for the environment, and at least one package to
install. For example, suppose we wish to create a new environment, and
need version 1.8 of NumPy.
{{% panel theme="info" header="Create a new environment by providing a name and package specification" %}}
{{< highlight bash >}}
conda create -n mynumpy numpy=1.8
{{< /highlight >}}
{{% /panel %}}
This will create a new environment called 'mynumpy' and installed NumPy
version 1.8, along with any required dependencies.
To use the environment, we must first *activate* it.
{{% panel theme="info" header="Activate environment" %}}
{{< highlight bash >}}
source activate mynumpy
{{< /highlight >}}
{{% /panel %}}
Our new environment is now active, and we can use it. The shell prompt
will change to indicate this as well (this can be disable if desired).
### Creating Custom GPU Anaconda Environment
We provide GPU versions of various frameworks such as `tensorflow`, `keras`, `theano`, via [modules](../module_commands). However, sometimes you may need additional libraries or packages that are not available as part of these modules. In this case, you will need to create your own GPU Anaconda environment.
To do this, you need to first clone one of our GPU modules to a new Anaconda environment, and then install the desired packages in this new environment.
The reason for this is that the GPU modules we support are built using the specific CUDA drivers our GPU nodes have. If you just create custom GPU environment without cloning the module, your code will not utilize the GPUs.
For example, if you want to use `tensorflow` with additional packages, first do:
{{% panel theme="info" header="Cloning GPU module to a new Anaconda environment" %}}
{{< highlight bash >}}
module load tensorflow-gpu/py36/1.12
conda create -n tensorflow-gpu-1.12-custom --clone $CONDA_DEFAULT_ENV
module purge
{{< /highlight >}}
{{% /panel %}}
This will create a new `tensorflow-gpu-1.12-custom` environment in your home directory that is a copy of the `tensorflow-gpu` module. Then, you can install the additional packages you need in this environment.
{{% panel theme="info" header="Install new packages in the currently active environment" %}}
{{< highlight bash >}}
module load anaconda
source activate tensorflow-gpu-1.12-custom
conda install <packages>
{{< /highlight >}}
{{% /panel %}}
Next, whenever you want to use this custom GPU Anaconda environment, you need to add these two lines in your submit script:
{{< highlight bash >}}
module load anaconda
source activate tensorflow-gpu-1.12-custom
{{< /highlight >}}
{{% notice info %}}
If you have custom GPU Anaconda environment please only use the two lines from above and **DO NOT** load the module you have cloned earlier. Using `module load tensorflow-gpu/py36/1.12` and `source activate tensorflow-gpu-1.12-custom` in the same script is **wrong** and may give you various errors and incorrect results.
{{% /notice %}}
### Adding Packages to an Existing Environment
To install additional packages in an environment, use the `install`
subcommand. Suppose we want to install iPython in our 'mynumpy'
environment. While the environment is active, use `install `with no
additional arguments.
{{% panel theme="info" header="Install a new package in the currently active environment" %}}
{{< highlight bash >}}
conda install ipython
{{< /highlight >}}
{{% /panel %}}
If you aren't currently in the environment you wish to install the
package in, add the `-n `option to specify the name.
{{% panel theme="info" header="Install new packages in a specified environment" %}}
{{< highlight bash >}}
conda install -n mynumpy ipython
{{< /highlight >}}
{{% /panel %}}
The `remove` subcommand to uninstall a package functions similarly.
{{% panel theme="info" header="Remove package from currently active environment" %}}
{{< highlight bash >}}
conda remove ipython
{{< /highlight >}}
{{% /panel %}}
{{% panel theme="info" header="Remove package from environment specified by name" %}}
{{< highlight bash >}}
conda remove -n mynumpy ipython
{{< /highlight >}}
{{% /panel %}}
To exit an environment, we *deactivate* it.
{{% panel theme="info" header="Exit current environment" %}}
{{< highlight bash >}}
source deactivate
{{< /highlight >}}
{{% /panel %}}
Finally, to completely remove an environment, add the `--all `option
to `remove`.
{{% panel theme="info" header="Completely remove an environment" %}}
{{< highlight bash >}}
conda remove -n mynumpy --all
{{< /highlight >}}
{{% /panel %}}
### Using an Anaconda Environment in a Jupyter Notebook on Crane
It is not difficult to make an Anaconda environment available to a
Jupyter Notebook. To do so, follow the steps below, replacing
`myenv` with the name of the Python or R environment you wish to use:
1. Stop any running Jupyter Notebooks and ensure you are logged out of
the JupyterHub instance at https://crane.unl.edu
1. If you are not logged out, please click the Control Panel button
located in the top right corner.
2. Click the "Stop My Server" Button to terminate the Jupyter
server.
3. Click the logout button in the top right corner.
2. Using the command-line environment, load the target conda
environment:
{{< highlight bash >}}source activate myenv{{< /highlight >}}
3. Install the Jupyter kernel and add the environment:
1. For a **Python** conda environment, install the IPykernel
package, and then the kernel specification:
{{< highlight bash >}}
# Install ipykernel
conda install ipykernel
# Install the kernel specification
python -m ipykernel install --user --name "$CONDA_DEFAULT_ENV" --display-name "Python ($CONDA_DEFAULT_ENV)"
{{< /highlight >}}
2. For an **R** conda environment, install the jupyter\_client and
IRkernel packages, and then the kernel specification:
{{< highlight bash >}}
# Install PNG support for R, the R kernel for Jupyter, and the Jupyter client
conda install r-png
conda install r-irkernel jupyter_client
# Install jupyter_client 5.2.3 from anaconda channel for bug workaround
conda install -c anaconda jupyter_client
# Install the kernel specification
R -e "IRkernel::installspec(name = '$CONDA_DEFAULT_ENV', displayname = 'R ($CONDA_DEFAULT_ENV)', user = TRUE)"
{{< /highlight >}}
4. Once you have the environment set up, deactivate it:
{{< highlight bash >}}source deactivate{{< /highlight >}}
5. To make your conda environments accessible from the worker nodes,
enter the following commands:
{{< highlight bash >}}
mkdir -p $WORK/.jupyter
mv ~/.local/share/jupyter/kernels $WORK/.jupyter
ln -s $WORK/.jupyter/kernels ~/.local/share/jupyter/kernels
{{< /highlight >}}
{{% notice note %}}
**Note**: Step 5 only needs to be done once. Any future created
environments will automatically be accessible from SLURM notebooks
once this is done.
{{% /notice %}}
6. Login to JupyterHub at https://crane.unl.edu
and create a new notebook using the environment by selecting the
correct entry in the `New` dropdown menu in the top right
corner.
{{< figure src="/images/24151931.png" height="400" class="img-border">}}
content/guides/sandstone/_index.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Sandstone"
description = "How to use HCC's sandstone environment"
weight = "45"
+++
### Overview
The HCC Sandstone environment is a GUI interface to the Crane cluster featuring a file browser, text editor, web terminal and SLURM script helper,
To login to the Sandstone environment, go to [crane.unl.edu](https://crane.unl.edu) in your web browser and sign in using your HCC Login Info and DUO authentication.
Upon login, you will land at the File Browser.
### File Browser
The file broswer allows you to view, access, and transfer files on Crane. On the left side you will have your available spaces, both your home and work directories. In the upper right, you have buttons to upload files, create a file, and create a directory.
{{< figure src="/images/SandstonefileBrowserOver.png" height="300" >}}
Clicking on either will change your current directory to either your home or work directory and display your user/group usage and quotas. You can navigate directories by clicking through them.
{{< figure src="/images/SandstonefileOptions.png" height="300" >}}
Clicking on a file or directory will bring up some options such as the permissions and actions to do such as editing the file, duplicating or moving it, deleting it, and downloading it.
### Editor
The editor is a basic text editor that allows you to have multiple files loaded and manipulate the files. A small file explorer is available on the left side to access more files. There are similar actions available for files above the mini file browser.
{{< figure src="/images/Sandstoneeditor.png" height="300" >}}
Like most text editors, basic functions exist to undo and redo changes, find and replace, and most importantly, to save the file.
{{< figure src="/images/SandstoneedtiorDropDown.png" height="300" >}}
### Terminal
The terminal gives you access to the linux command line on crane, similar to what you would have if you SSH'd directly into Crane. Once the login and quote screen, you can enter commands and interact as you would with a standard terminal.
{{< figure src="/images/SandstoneTerminal.png" height="300" >}}
### Slurm Assist
Slurm assist is a tool to help create and run slurm submit scripts. The first step is to select a base profile from the profile dropdown menu. Options will appear and the directives will automatically appear. The options are editable to better fit to your specific job with more details found in our submitting jobs documentation. After the directives are filled out, you can then add the commands to start your job in the script section. To save the job, select 'save script for later' and save the script in a known location for later.
{{< figure src="/images/SandstoneSASettings.png" height="300" >}}
From here, you can also schedule the script recently create, by selecting "Schedule Job". A confirmation will appear with the Job ID and then an instruction on how to view the status of your job.
{{< figure src="/images/SandstoneJobConf.png" height="150" >}}
{{< figure src="/images/SandstoneSAStatus.png" height="300" >}}
You can view the progress of other jobs from slurm assist by going to the status page. Here you will see the State of the job, its ID, name, group name, runtime, and the start and end times.
{{< figure src="/images/SandstoneSAStatusPage.png" height="150" >}}
{{< figure src="/images/SandstoneSAStatuses.png" height="300" >}}
\ No newline at end of file
content/guides/submitting_jobs/condor_jobs_on_hcc.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Condor Jobs on HCC"
description = "How to run jobs using Condor on HCC machines"
weight = "54"
+++
This quick start demonstrates how to run multiple copies of Fortran/C program
using Condor on HCC supercomputers. The sample codes and submit scripts
can be downloaded from [condor_dir.zip](/attachments/3178558.zip).
#### Login to a HCC Cluster
Log in to a HCC cluster through PuTTY ([For Windows Users]({{< relref "/quickstarts/connecting/for_windows_users">}})) or Terminal ([For Mac/Linux Users]({{< relref "/quickstarts/connecting/for_maclinux_users">}})) and make a subdirectory called `condor_dir` under the `$WORK` directory. In the subdirectory `condor_dir`, create job subdirectories that host the input data files. Here we create two job subdirectories, `job_0` and `job_1`, and put a data file (`data.dat`) in each subdirectory. The data file in `job_0` has a column of data listing the integers from 1 to 5. The data file in `job_1` has a integer list from 6 to 10.
{{< highlight bash >}}
$ cd $WORK
$ mkdir condor_dir
$ cd condor_dir
$ mkdir job_0
$ mkdir job_1
{{< /highlight >}}
In the subdirectory condor`_dir`, save all the relevant codes. Here we
include two demo programs, `demo_f_condor.f90` and `demo_c_condor.c`,
that compute the sum of the data stored in each job subdirectory
(`job_0` and `job_1`). The parallelization scheme here is as the
following. First, the master computer node send out many copies of the
executable from the `condor_dir` subdirectory and a copy of the data
file in each job subdirectories. The number of executable copies is
specified in the submit script (`queue`), and it usually matches with
the number of job subdirectories. Next, the workload is distributed
among a pool of worker computer nodes. At any given time, the number of
available worker nodes may vary. Each worker node executes the jobs
independent of other worker nodes. The output files are separately
stored in the job subdirectory. No additional coding are needed to make
the serial code turned "parallel". Parallelization here is achieved
through the submit script.
{{%expand "demo_condor.f90" %}}
{{< highlight fortran >}}
Program demo_f_condor
implicit none
integer, parameter :: N = 5
real*8 w
integer i
common/sol/ x
real*8 x
real*8, dimension(N) :: y_local
real*8, dimension(N) :: input_data
open(10, file='data.dat')
do i = 1,N
read(10,*) input_data(i)
enddo
do i = 1,N
w = input_data(i)*1d0
call proc(w)
y_local(i) = x
write(6,*) 'i,x = ', i, y_local(i)
enddo
write(6,*) 'sum(y) =',sum(y_local)
Stop
End Program
Subroutine proc(w)
real*8, intent(in) :: w
common/sol/ x
real*8 x
x = w
Return
End Subroutine
{{< /highlight >}}
{{% /expand %}}
{{%expand "demo_c_condor.c" %}}
{{< highlight c >}}
//demo_c_condor
#include <stdio.h>
double proc(double w){
double x;
x = w;
return x;
}
int main(int argc, char* argv[]){
int N=5;
double w;
int i;
double x;
double y_local[N];
double sum;
double input_data[N];
FILE *fp;
fp = fopen("data.dat","r");
for (i = 1; i<= N; i++){
fscanf(fp, "%lf", &input_data[i-1]);
}
for (i = 1; i <= N; i++){
w = input_data[i-1]*1e0;
x = proc(w);
y_local[i-1] = x;
printf("i,x= %d %lf\n", i, y_local[i-1]) ;
}
sum = 0e0;
for (i = 1; i<= N; i++){
sum = sum + y_local[i-1];
}
printf("sum(y)= %lf\n", sum);
return 0;
}
{{< /highlight >}}
{{% /expand %}}
---
#### Compiling the Code
The compiled executable needs to match the "standard" environment of the
worker node. The easies way is to directly use the compilers installed
on the HCC supercomputer without loading extra modules. The standard
compiler of the HCC supercomputer is GNU Compier Collection. The version
can be looked up by the command lines `gcc -v` or `gfortran -v`.
{{< highlight bash >}}
$ gfortran demo_f_condor.f90 -o demo_f_condor.x
$ gcc demo_c_condor.c -o demo_c_condor.x
{{< /highlight >}}
#### Creating a Submit Script
Create a submit script to request 2 jobs (queue). The name of the job
subdirectories is specified in the line `initialdir`. The
`$(process)` macro assigns integer numbers to the job subdirectory
name `job_`. The numbers run form `0` to `queue-1`. The name of the input
data file is specified in the line `transfer_input_files`.
{{% panel header="`submit_f.condor`"%}}
{{< highlight bash >}}
universe = grid
grid_resource = pbs
batch_queue = guest
should_transfer_files = yes
when_to_transfer_output = on_exit
executable = demo_f_condor.x
output = Fortran_$(process).out
error = Fortran_$(process).err
initialdir = job_$(process)
transfer_input_files = data.dat
queue 2
{{< /highlight >}}
{{% /panel %}}
{{% panel header="`submit_c.condor`"%}}
{{< highlight bash >}}
universe = grid
grid_resource = pbs
batch_queue = guest
should_transfer_files = yes
when_to_transfer_output = on_exit
executable = demo_c_condor.x
output = C_$(process).out
error = C_$(process).err
initialdir = job_$(process)
transfer_input_files = data.dat
queue 2
{{< /highlight >}}
{{% /panel %}}
#### Submit the Job
The job can be submitted through the command `condor_submit`. The job
status can be monitored by entering `condor_q` followed by the
username.
{{< highlight bash >}}
$ condor_submit submit_f.condor
$ condor_submit submit_c.condor
$ condor_q <username>
{{< /highlight >}}
Replace `<username>` with your HCC username.
Sample Output
-------------
In the job subdirectory `job_0`, the sum from 1 to 5 is computed and
printed to the `.out` file. In the job subdirectory `job_1`, the sum
from 6 to 10 is computed and printed to the `.out` file.
{{%expand "Fortran_0.out" %}}
{{< highlight batchfile>}}
i,x = 1 1.0000000000000000
i,x = 2 2.0000000000000000
i,x = 3 3.0000000000000000
i,x = 4 4.0000000000000000
i,x = 5 5.0000000000000000
sum(y) = 15.000000000000000
{{< /highlight >}}
{{% /expand %}}
{{%expand "Fortran_1.out" %}}
{{< highlight batchfile>}}
i,x = 1 6.0000000000000000
i,x = 2 7.0000000000000000
i,x = 3 8.0000000000000000
i,x = 4 9.0000000000000000
i,x = 5 10.000000000000000
sum(y) = 40.000000000000000
{{< /highlight >}}
{{% /expand %}}
content/guides/submitting_jobs/monitoring_jobs.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Monitoring Jobs"
description = "How to find out information about running and completed jobs."
+++
Careful examination of running times, memory usage and output files will
allow you to ensure the job completed correctly and give you a good idea
of what memory and time limits to request in the future.
### Monitoring Completed Jobs:
To see the runtime and memory usage of a job that has completed, use the
sacct command:
{{< highlight bash >}}
sacct
{{< /highlight >}}
Lists all jobs by the current user and displays information such as
JobID, JobName, State, and ExitCode.
{{< figure src="/images/21070053.png" height="150" >}}
Coupling this command with the --format flag will allow you to see more
than the default information about a job. Fields to display should be
listed as a comma separated list after the --format flag (without
spaces). For example, to see the Elapsed time and Maximum used memory by
a job, this command can be used:
{{< highlight bash >}}
sacct --format JobID,JobName,Elapsed,MaxRSS
{{< /highlight >}}
{{< figure src="/images/21070054.png" height="150" >}}
Additional arguments and format field information can be found in
[the SLURM documentation](https://slurm.schedmd.com/sacct.html).
### Monitoring Running Jobs:
There are two ways to monitor running jobs, the top command and
monitoring the cgroup files. Top is helpful when monitoring
multi-process jobs, whereas the cgroup files provide information on
memory usage. Both of these tools require the use of an interactive job
on the same node as the job to be monitored.
{{% notice warning %}}
If the job to be monitored is using all available resources for a node,
the user will not be able to obtain a simultaneous interactive job.
{{% /notice %}}
After the job to be monitored is submitted and has begun to run, request
an interactive job on the same node using the srun command:
{{< highlight bash >}}
srun --jobid=<JOB_ID> --pty bash
{{< /highlight >}}
Where `<JOB_ID>` is replaced by the job id for the monitored job as
assigned by SLURM.
Alternately, you can request the interactive job by nodename as follows:
{{< highlight bash >}}
srun --nodelist=<NODE_ID> --pty bash
{{< /highlight >}}
Where `<NODE_ID>` is replaced by the node name that the monitored
job is running. This information can be found out by looking at the
squeue output under the `NODELIST` column.
{{< figure src="/images/21070055.png" width="700" >}}
Once the interactive job begins, you can run top to view the processes
on the node you are on:
{{< figure src="/images/21070056.png" height="400" >}}
Output for top displays each running process on the node. From the above
image, we can see the various MATLAB processes being run by user
cathrine98. To filter the list of processes, you can type `u` followed
by the username of the user who owns the processes. To exit this screen,
press `q`.
During a running job, the cgroup folder is created which contains much
of the information used by sacct. These files can provide a live
overview of resources used for a running job. To access the cgroup
files, you will need to be in an interactive job on the same node as the
monitored job. To view specific files, and information, use one of the
following commands:
##### To view current memory usage:
{{< highlight bash >}}
less /cgroup/memory/slurm/uid_<UID>/job_<SLURM_JOB_ID>/memory.usage_in_bytes
{{< /highlight >}}
Where `<UID>` is replaced by your UID and `<SLURM_JOB_ID>` is
replaced by the monitored job's Job ID as assigned by Slurm.
{{% notice note %}}
To find your uid, use the command `id -u`. Your UID never changes and is
the same on all HCC clusters (*not* on Anvil, however!).
{{% /notice %}}
##### To view maximum memory usage from start of job to current point:
{{< highlight bash >}}
cat /cgroup/memory/slurm/uid_${UID}/job_${SLURM_JOBID}/memory.max_usage_in_bytes
{{< /highlight >}}
content/guides/submitting_jobs/partitions/_index.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Partitions"
description = "Listing of partitions on Tusker and Crane."
scripts = ["https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.min.js", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-pager.min.js","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-filter.min.js","/js/sort-table.js"]
css = ["http://mottie.github.io/tablesorter/css/theme.default.css","https://mottie.github.io/tablesorter/css/theme.dropbox.css", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/jquery.tablesorter.pager.min.css","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/filter.formatter.min.css"]
+++
Partitions are used in Crane and Tusker to distinguish different
resources. You can view the partitions with the command `sinfo`.
### Crane:
[Full list for Crane]({{< relref "crane_available_partitions" >}})
### Tusker:
[Full list for Tusker]({{< relref "tusker_available_partitions" >}})
#### Priority for short jobs
To run short jobs for testing and development work, a job can specify a
different quality of service (QoS). The *short* QoS increases a jobs
priority so it will run as soon as possible.
| SLURM Specification |
|----------------------- |
| `#SBATCH --qos=short` |
{{% panel theme="warning" header="Limits per user for 'short' QoS" %}}
- 6 hour job run time
- 2 jobs of 16 CPUs or fewer
- No more than 256 CPUs in use for *short* jobs from all users
{{% /panel %}}
### Limitations of Jobs
Overall limitations of maximum job wall time. CPUs, etc. are set for
all jobs with the default setting (when thea "–qos=" section is omitted)
and "short" jobs (described as above) on Tusker and Crane.
The limitations are shown in the following form.
| | SLURM Specification | Max Job Run Time | Max CPUs per User | Max Jobs per User |
| ------- | -------------------- | ---------------- | ----------------- | ----------------- |
| Default | Leave blank | 7 days | 2000 | 1000 |
| Short | #SBATCH --qos=short | 6 hours | 16 | 2 |
Please also note that the memory and
local hard drive limits are subject to the physical limitations of the
nodes, described in the resources capabilities section of the
[HCC Documentation]({{< relref "/#resource-capabilities" >}})
and the partition sections above.
### Owned Partitions
Partitions marked as owned by a group means only specific groups are
allowed to submit jobs to that partition. Groups are manually added to
the list allowed to submit jobs to the partition. If you are unable to
submit jobs to a partition, and you feel that you should be, please
contact {{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu).
### Guest Partition
The `guest` partition can be used by users and groups that do not own
dedicated resources on Crane. Jobs running in the `guest` partition
will run on the owned resources with Intel OPA interconnect. The jobs
are preempted when the resources are needed by the resource owners and
are restarted on another node.
### tmp_anvil Partition
We have put Anvil nodes which are not running Openstack in this
partition. They have Intel Xeon E5-2650 v3 2.30GHz 2 CPU/20 cores and
256GB memory per node. However, they don't have Infiniband or OPA
interconnect. They are suitable for serial or single node parallel jobs.
The nodes in this partition are subjected to be drained and move to our
Openstack cloud when more cloud resources are needed without notice in
advance.
### Use of Infiniband or OPA
Crane nodes use either Infiniband or Intel Omni-Path interconnects in
the batch partition. Most users don't need to worry about which one to
choose. Jobs will automatically be scheduled for either of them by the
scheduler. However, if the user wants to use one of the interconnects
exclusively, the SLURM constraint keyword is available. Here are the
examples:
{{% panel theme="info" header="SLURM Specification: Omni-Path" %}}
{{< highlight bash >}}
#SBATCH --constraint=opa
{{< /highlight >}}
{{% /panel %}}
{{% panel theme="info" header="SLURM Specification: Infiniband" %}}
{{< highlight bash >}}
#SBATCH --constraint=ib
{{< /highlight >}}
{{% /panel %}}
content/guides/submitting_jobs/partitions/crane_available_partitions.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Available Partitions for Crane"
description = "List of available partitions for crane.unl.edu."
scripts = ["https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.min.js", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-pager.min.js","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-filter.min.js","/js/sort-table.js"]
css = ["http://mottie.github.io/tablesorter/css/theme.default.css","https://mottie.github.io/tablesorter/css/theme.dropbox.css", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/jquery.tablesorter.pager.min.css","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/filter.formatter.min.css"]
+++
### Crane:
{{< table url="http://crane-head.unl.edu:8192/slurm/partitions/json" >}}
content/guides/submitting_jobs/partitions/tusker_available_partitions.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Available Partitions for Tusker"
description = "List of available partitions for tusker.unl.edu."
scripts = ["https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.min.js", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-pager.min.js","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-filter.min.js","/js/sort-table.js"]
css = ["http://mottie.github.io/tablesorter/css/theme.default.css","https://mottie.github.io/tablesorter/css/theme.dropbox.css", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/jquery.tablesorter.pager.min.css","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/filter.formatter.min.css"]
+++
### Tusker:
{{< table url="http://tusker-head.unl.edu:8192/slurm/partitions/json" >}}
Two nodes have 512GB of memory instead of 256GB (Max Request = 500GB), and two have 1024GB of memory (Max Request = 1000GB).
content/guides/submitting_jobs/submitting_an_interactive_job.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Submitting an Interactive Job"
description = "How to run an interactive job on HCC resources."
+++
{{% notice info %}}
The `/home` directories are read-only on the worker nodes. You will need
to compile or run your processing in `/work`.
{{% /notice %}}
Submitting an interactive job is done with the command `srun`.
{{< highlight bash >}}
$ srun --pty $SHELL
{{< /highlight >}}
or to allocate 4 cores per node:
{{< highlight bash >}}
$ srun --nodes=1 --ntasks-per-node=4 --mem-per-cpu=1024 --pty $SHELL
{{< /highlight >}}
Submitting an interactive job is useful if you require extra resources
to run some processing by hand. It is also very useful to debug your
processing.
And interactive job is scheduled onto a worker node just like a regular
job. You can provide options to the interactive job just as you would a
regular SLURM job.
### Priority for short jobs
To run short jobs for testing and development work, a job can specify a
different quality of service (QoS). The *short* QoS increases a jobs
priority so it will run as soon as possible.
| SLURM Specification |
|---------------------|
| `--qos=short` |
{{% panel theme="warning" header="Limits per user for 'short' QoS" %}}
- 6 hour job run time
- 2 jobs of 16 CPUs or fewer
- No more than 256 CPUs in use for *short* jobs from all users
{{% /panel %}}
{{% panel theme="info" header="Using the short QoS" %}}
{{< highlight bash >}}
srun --qos=short --nodes=1 --ntasks-per-node=1 --mem-per-cpu=1024 --pty $SHELL
{{< /highlight >}}
{{% /panel %}}
content/guides/submitting_jobs/submitting_an_mpi_job.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Submitting an MPI Job"
description = "How to submit an MPI job on HCC resources."
+++
This script requests 16 cores on nodes with InfiniBand:
{{% panel theme="info" header="mpi.submit" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1024
#SBATCH --time=03:15:00
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load compiler/gcc/8.2 openmpi/2.1
mpirun /home/[groupname]/[username]/mpiprogram
{{< /highlight >}}
{{% /panel %}}
The above job will allocate 16 cores on the default partition. The 16
cores could be on any of the nodes in the partition, even split between
multiple nodes.
### Advanced Submission
Some users may prefer to specify more details. This will allocate 32
tasks, 16 on each of two nodes:
{{% panel theme="info" header="mpi.submit" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --mem-per-cpu=1024
#SBATCH --time=03:15:00
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load compiler/gcc/8.2 openmpi/2.1
mpirun /home/[groupname]/[username]/mpiprogram
{{< /highlight >}}
{{% /panel %}}
content/guides/submitting_jobs/submitting_cuda_or_openacc_jobs.md deleted 100644 → 0
View file @ 2383ec36
+++
title = "Submitting CUDA or OpenACC Jobs"
description = "How to submit GPU (CUDA/OpenACC) jobs on HCC resources."
+++
### Available GPUs
Crane has four types of GPUs available in the **gpu** partition. The
type of GPU is configured as a SLURM feature, so you can specify a type
of GPU in your job resource requirements if necessary.
| Description | SLURM Feature | Available Hardware |
| -------------------- | ------------- | ---------------------------- |
| Tesla K20, non-IB | gpu_k20 | 3 nodes - 2 GPUs per node |
| Teska K20, with IB | gpu_k20 | 3 nodes - 3 GPUs per node |
| Tesla K40, with IB | gpu_k40 | 5 nodes - 4 K40M GPUs per node<br> 1 node - 2 K40C GPUs |
| Tesla P100, with OPA | gpu_p100 | 2 nodes - 2 GPUs per node |
To run your job on the next available GPU regardless of type, add the
following options to your srun or sbatch command:
{{< highlight batch >}}
--partition=gpu --gres=gpu
{{< /highlight >}}
To run on a specific type of GPU, you can constrain your job to require
a feature. To run on K40 GPUs for example:
{{< highlight batch >}}
--partition=gpu --gres=gpu --constraint=gpu_k40
{{< /highlight >}}
{{% notice info %}}
You may request multiple GPUs by changing the` --gres` value to
-`-gres=gpu:2`. Note that this value is **per node**. For example,
`--nodes=2 --gres=gpu:2 `will request 2 nodes with 2 GPUs each, for a
total of 4 GPUs.
{{% /notice %}}
### Compiling
Compilation of CUDA or OpenACC jobs must be performed on the GPU nodes.
Therefore, you must run an [interactive job]({{< relref "submitting_an_interactive_job" >}})
to compile. An example command to compile in the **gpu** partition could be:
{{< highlight batch >}}
$ srun --partition=gpu --gres=gpu --mem-per-cpu=1024 --ntasks-per-node=6 --nodes=1 --pty $SHELL
{{< /highlight >}}
The above command will start a shell on a GPU node with 6 cores and 6GB
of ram in order to compile a GPU job. The above command could also be
useful if you want to run a test GPU job interactively.
### Submitting Jobs
CUDA and OpenACC submissions require running on GPU nodes.
{{% panel theme="info" header="cuda.submit" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --time=03:15:00
#SBATCH --mem-per-cpu=1024
#SBATCH --job-name=cuda
#SBATCH --partition=gpu
#SBATCH --gres=gpu
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load cuda/8.0
./cuda-app.exe
{{< /highlight >}}
{{% /panel %}}
OpenACC submissions require loading the PGI compiler (which is currently
required to compile as well).
{{% panel theme="info" header="openacc.submit" %}}
{{< highlight batch >}}
#!/bin/sh
#SBATCH --time=03:15:00
#SBATCH --mem-per-cpu=1024
#SBATCH --job-name=cuda-acc
#SBATCH --partition=gpu
#SBATCH --gres=gpu
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
module load cuda/8.0 compiler/pgi/16
./acc-app.exe
{{< /highlight >}}
{{% /panel %}}