patch-1 to master · Holland Computing Center / HCC docs

Some changes are not shown.

For a faster browsing experience, only 345 of 901 files are shown. Download one of the files below to see all changes.

.gitignore

+4
−0

Original line number
Original line
Diff line number
Diff line

public/

public/

.cache/

__pycache__/

site/

**/.DS_Store

.gitlab-ci.yml

+27
−9

Original line number
Original line
Diff line number
Diff line

variables:

variables:

  BASE_URL: "//hcc.unl.edu/docs"

  BASE_URL: "//hcc.unl.edu/docs"

  DEPLOY_ROOT: "/var/www/html/hcc-docs"

  DEPLOY_ROOT: "/var/www/html/hcc-docs"

  HUGO_TARBALL:  "https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_Linux-64bit.tar.gz"

  MKDOCS_PREFIX: /tmp/gitlab-${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}/mkdocs

  GIT_SUBMODULE_STRATEGY: recursive

  MICROMAMBA_ROOT_PREFIX: /tmp/gitlab-${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}/micromamba

.fetch_external_files: &fetch_external_files

  - curl -L https://raw.githubusercontent.com/unlhcc/singularity-dockerfiles/master/IMAGELIST.md > docs/static/markdown/singularity-images.md

  - curl -L http://swan-head.unl.edu:8192/bio/data/json > docs/static/json/biodata.json

  - curl -L http://swan-head.unl.edu:8192/lmod/spider/json > docs/static/json/lmod.json

  - curl -L http://swan-head.unl.edu:8192/slurm/partitions/json > docs/static/json/swan_partitions.json

stages:

stages:

  - test

  - test

@@ -10,15 +16,19 @@ stages:

test:

test:

  stage: test

  stage: test

  image: unlhcc/docker-glibc

  image: python:3.11

  except:

  except:

    - master

    - master

  tags:

  tags:

    - docker

    - docker

  before_script:

  before_script:

    - curl -L -o - ${HUGO_TARBALL} | tar -zxv -C /usr/local/bin

    - apt-get -q update

    - apt-get -q -y install zip pandoc

    - pip install -q -r requirements.txt

    - *fetch_external_files

  script:

  script:

    - hugo --ignoreCache -v

    - mkdocs build

    - ./pandoc-build.sh

deploy:

deploy:

  stage: deploy

  stage: deploy

@@ -29,8 +39,16 @@ deploy:

    - master

    - master

  tags:

  tags:

    - hcc-docs-prod

    - hcc-docs-prod

  before_script:

    - mkdir -p ${MICROMAMBA_ROOT_PREFIX} ${MKDOCS_PREFIX}

    - *fetch_external_files

    - micromamba create -y -q -p ${MKDOCS_PREFIX}/mkdocs python=3.11 pip cairo pandoc

    - micromamba run -p ${MKDOCS_PREFIX}/mkdocs pip install -q -r requirements.txt

  script:

  script:

    - curl -L https://raw.githubusercontent.com/unlhcc/singularity-dockerfiles/master/IMAGELIST.md > static/markdown/singularity-images.md

    - micromamba run -p ${MKDOCS_PREFIX}/mkdocs mkdocs build

    - hugo --ignoreCache -b ${BASE_URL} -v

    - find site/ -type f -name "*.html" -exec sed -i -e 's/src="\//src="\/docs\//g' -e 's/href="\//href="\/docs\//g' {} +

    - pandoc -s content/facilities.md -o public/facilities.docx

    - micromamba run -p ${MKDOCS_PREFIX}/mkdocs ./pandoc-build.sh

    - rsync -avz --delete public/ $DEPLOY_ROOT

    - rsync -avz --delete site/ ${DEPLOY_ROOT}

  after_script:

    - rm -rf /tmp/gitlab-${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}

CONTRIBUTING.md

+3
−3

Original line number
Original line
Diff line number
Diff line

@@ -42,10 +42,10 @@ or a factual error.

1.  Checking for outstanding [issues](https://git.unl.edu/hcc/hcc-docs/issues) is a quick way to identify contribution areas. Additionally,

1.  Checking for outstanding [issues](https://git.unl.edu/hcc/hcc-docs/issues) is a quick way to identify contribution areas. Additionally,

    review an comments on existing merge requests are welcome.

    review an comments on existing merge requests are welcome.

2.  All documentation content can be found within the `content` folder. Subdirectories indicate menu items,

2.  All documentation content can be found within the `docs` folder. Subdirectories indicate menu items,

    with the top level document called `_index.md` within each subdirectory.

    with the top level document called `index.md` within each subdirectory.

Note: please do not submit modifications to theme or CSS items, `_header` or `_footer` files unless they 

Note: please do not submit modifications to theme or CSS items unless they 

adhere to [UNL style guidelines](https://wdn.unl.edu/unl-web-framework-5-standards-guide).

adhere to [UNL style guidelines](https://wdn.unl.edu/unl-web-framework-5-standards-guide).

## What to Contribute

## What to Contribute

README.md

+148
−72

Original line number
Original line
Diff line number
Diff line

@@ -3,8 +3,8 @@ HCC-DOCS

This is the repository for HCC's user documentation.  The documentation is a

This is the repository for HCC's user documentation.  The documentation is a

combination of [Markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

combination of [Markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

for the source and the [Hugo](https://gohugo.io) static site generator.  Currently, 

for the source and the [MkDocs](https://www.mkdocs.org/) static site generator.  Currently, 

the theme used is [docDock](https://themes.gohugo.io/docdock/).

the theme used is [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/).

The website is viewable at https://hcc.unl.edu/docs.

The website is viewable at https://hcc.unl.edu/docs.

@@ -12,121 +12,183 @@ How to contribute

-----------------

-----------------

Since the documentation is a git repo, the standard git workflow of creating a branch for your

Since the documentation is a git repo, the standard git workflow of creating a branch for your

changes and opening a merge request applies.  The steps below detail how to install Hugo, 

changes and opening a merge request applies.  The steps below detail how to install MkDocs, 

edit/create content, and preview your changes locally before creating the MR/PR.

edit/create content, and preview your changes locally before creating the MR/PR.

### Setup your environment

### Setup your environment

#### Clone the repo (including submodules) and create a new branch

#### Clone the repo (including submodules) and create a new branch

Clone the repo somewhere convenient.  The DocDock theme is provided via a git submodule,

Clone the repo somewhere convenient. 

so after cloning, run `git submodule update --init --recursive` in the root of the repo.

Run `git checkout -b <mybranch>` to create a new branch for your changes.

#### Download and install Hugo

#### Download and install MkDocs

Download the appropriate Hugo tarball from the Hugo [releases page](https://github.com/gohugoio/hugo/releases).

MkDocs, the Material for MkDocs theme, and all plugins are installable with pip. 

(Direct links for [Linux](https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_Linux-64bit.tar.gz), [Mac](https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_macOS-64bit.tar.gz), and [Windows](https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_Windows-64bit.zip))

```bash

pip install mkdocs mkdocs-material mkdocs-nav-weight \

The `hugo` program is a single binary with no dependencies and does not require root priviledges, 

    mkdocs-material[imaging] mkdocs-awesome-nav mkdocs-macros-plugin \

so just place it somewhere convenient in your `PATH` (i.e. `~/bin`).

    mkdocs-table-reader-plugin mkdocs-include-markdown-plugin

Verify it works by running the command `hugo version`:

```

bash-4.4# hugo version

Hugo Static Site Generator v0.51 linux/amd64 BuildDate: 2018-11-07T10:10:13Z

```

```

### Previewing the site and adding content

### Previewing the site and adding content

#### Use hugo server to preview the site

#### Use mkdocs serve to preview the site

The `hugo server` command will build the site and run a small webserver to allow you to

The `mkdocs serve` command will build the site and run a small webserver to allow you to

preview it.  Depending whether you're working on your local machine or a VM on Anvil,

preview it.  Depending whether you're working on your local machine or a VM on Anvil,

the option you pass needs to be slightly different.  The `hugo server` command should be

the option you pass needs to be slightly different.  The `mkdocs serve` command should be

run from the root of the repository (i.e. the same directory the `config.toml` file

run from the root of the repository (i.e. the same directory the `mkdocs.yml` file

resides in).

resides in).

**If you're working on your local laptop/deskop:**

This will allow you to view the webiste by going to `http://localhost:8080` in your web browser. 

Run the command:

Alternatively you can build the HTML files to view in a web browser using the `mkdocs build` command 

```

from the root of the repository. 

hugo server -b http://127.0.0.1

```

You should now be able to preview the website by going to `http://localhost:1313` in your web

browser. 

**If you're working in a VM on Anvil:**

**If you're working in a VM on Anvil:**

You'll first need to allow port 1313.  If your VM is under the `swanson` project, there is

You'll first need to allow port 8080.  If your VM is under the `swanson` project, there is

already an existing "Hugo" security group.  Just add that group to your VM.  Otherwise, first

already an existing "Port 8080" security group.  Just add that group to your VM.  Otherwise, first

create it and allow TCP port 1313 though.  You may also need to allow port 1313 through your

create it and allow TCP port 8080 though.  You may also need to allow port 8080 through your

instance's firewall.

instance's firewall.

Once that is done, run the command:

Once that is done, run the command:

```

```

hugo server -b http://<instance IP> --bind <instance IP>

mkdocs serve

```

```

where *\<instance IP\>* is the 10.71.x.x address of your instance.

You should now be able to preview the website by going to `http://<instance IP>:1313` in your web

You should now be able to preview the website by going to `http://<instance IP>:8080` in your web

browser.

browser, where *\<instance IP\>* is the 10.71.x.x address of your instance.

*Note:  The hugo server command will watch the filesystem for changes and force an update in your browswer,

*Note:  The mkdocs serve command will watch the filesystem for changes and force an update in your browswer,

so you can simply leave the server running as you work.*

so you can simply leave the server running as you work.*

**Note that the `mkdocs serve` command will not serve image files. If you need to preview images, you'll need to

run nginx via Docker as noted below.**

#### Running nginx using Docker

The built-in mkdocs server doesn't serve images. To fully preview the site either locally or on your VM

you'll need to run an nginx container. Install Docker and ensure port 8080 is allowed through to your VM as above if needed.

First build the site by running

```

mkdocs build

```

in the repo root. Still in the top-level folder of the repo, then run

```

docker run -d --name nginx-hcc-docs -v "${PWD}/site:/usr/share/nginx/html:ro" -p 8080:80 nginx:latest

```

This will start an nginx container in the background. You can then access the site at port 8080 either on localhost

or your VM's IP.  Note that the autorebuild function does not work here.

So you'll need to manually run `mkdocs rebuild` to pick up changes.

Once you're done, remove the container by running

```

docker rm -f nginx-docs

```

#### Adding content

#### Adding content

The page content for the site is under the `content/` directory.  The directory structure will determine

The page content for the site is under the `docs/` directory.  The directory structure will determine

how the site itself is structured, so place content based on where in the site you would like it to be.

how the site itself is structured, so place content based on where in the site you would like it to be.

Each level in the tree will need to have an `_index.md` file to designate it as a new section.  The content

Each level in the tree will need to have an `index.md` file to designate it as a new section.  The content

itself is almost all standard markdown.  However, there are a few things that are specific to Hugo that

itself is almost all standard markdown.  However, there are a few things that are specific to MkDocs that

are used:

are used:

1.  *Front matter*  

1.  *YAML Style Metadata*  

    [Front matter](https://gohugo.io/content-management/front-matter) is Hugo-specific metadata added

    [YAML Style Metadata](https://www.mkdocs.org/user-guide/writing-your-docs/#yaml-style-meta-data) is mkdocs-specific metadata added

    to the page.  The format can be TOML, YAML, or JSON.  The example here uses TOML, which is identified

    to the page.  The format needs to be YAML.  The example here is for our contact-us page, and is identified

    by opening and closing `+++`.  At minimum, each page should have a title and description:  

    by opening and closing `---`  At minimum, each page should have a title and description:  

    ```

    ```

    +++

    ---

    title = "Contact Us"

    title: "Contact Us"

    description = "Contact Information for HCC"

    description: "Contact Information for HCC"

    +++

    ---

    ```  

    ```  

    Another useful parameter is the `weight` option.  This will determine the order of display in the navigation tree;

    Another useful parameter is the `weight` option.  This will determine the order of display in the navigation tree;

    lower numbers are displayed before higher numbers.

    lower numbers are displayed before higher numbers.

2.  *Shortcodes*  

2.  *Macros and variables*  

    [Shortcodes](https://gohugo.io/content-management/shortcodes) are "helper functions" to add content

    [Macros](https://mkdocs-macros-plugin.readthedocs.io/en/latest/) are "helper functions" to add content

    that would otherwise require using raw HTML.  Hugo provides some [built-in](https://gohugo.io/content-management/shortcodes/#use-hugo-s-built-in-shortcodes)

    that would otherwise require using raw HTML.  

    shortcodes. The DocDock theme also provides its own [shortcodes](http://docdock.netlify.com/shortcodes).

    Shortcodes are added via a `{{% shortcodename parameters %}}` declaration.  For example, to embed a YouTube

    Currently, a few custom ones are implemented. 

    video with ID `-Vh7SyC-3mA`:  

    - `{{ youtube('Emded_URL_OR_Share_Code') }}` - Embeds a youtube video into the docs

    ```

    - `{{ md_table('path_or_url') }}` - Embeds a markdown table as a sortable table. 

    {{< youtube -Vh7SyC-3mA >}}

    - `{{ json_table('path_or_url') }}` - Embeds a JSON table as a sortable table. 

    ```  

    Other useful shortcodes include language-specific syntax highlighting, figures, and automatic listing

    Variables allow for the use of defining shorthand codes for using the same value in multiple places in the documentation.

    of page children for navgiation/summary pages.

    These are all defined in the `mkdocs.yml` file in the `extra:` section. 

3.  *Using `relref` for link generation*  

    Variables are called using `{{ variable.path.to.value }}` where the path starts after the `extra:` block. 

    The `relref` shortcode should generally be used for linking to other pages within the site.  This will

    generate relative links that should work no matter what the base URL is.  For example, assume you're

    ```yaml

    editing "my\_page.md" and want to link to the page "my\_other\_page.md" in the subdirectory "mystuff".

    extra:

    To do so, add the line  

      hcc:  

      # Swan Cluster Specific Information

        swan:

          xfer: "swan-xfer.unl.edu"

          ood:

            name: "Swan Open OnDemand"

            url: "https://swan-ood.unl.edu"

          work:

            block: "100 TiB"

    ```

    ```

    [My other page]({{< relref "mystuff/my_other_page" >}})

    Using a section of the YAML above we can get the following exampls:

    - `{{ hcc.swan.xfer }}` would yield the URL for Swan's high speed transfer node

    - `{{ hcc.swan.ood.url }}` would yield the block quota for Swan's $WORK filesystem

    - `{{ hcc.swan.work.block }}` would yield the Open OnDemand URL for Swan

    - ` {{ children('path_after_docs') }} ` - List child pages and Descriptions - do not add trailing `/`

    A good file for examples of the use of these variables is `docs/handling_data/index.md` where many variables 

    are used to discuss the filesystems.

3.  Code Blocks

    Code blocks are able to use language specific formatting and use the standard markdown format for codeblocks. 

    ```` markdown title="Example Code Block"

    ``` python

    print('Hello World!')

    ```

    ```

    ````

4.  Links 

    MkDocs uses standard markdown links, `[Text to Display](URL or path/to/file)`

    Note that you can omit the `.md` suffix from the filename.

    Note that you can omit the `.md` suffix from the filename.

4.  *Static content*  

5.  *Static content*  

    Hugo has special handling for static content (images, linked zip files, etc.).  In the root of the repo,

    there is a `static/` directory; all static content should go there.  Subdirectories can be made to keep

    1.  Images

    things organized; there are existing directories for `images` and `attachments`.  Feel free to create

        Images are stored in `docs/images`. 

    additional directories if it makes sense.  To link to static content, you can use an absolute path from

    any page.  For example, assuming there is an image in the repo at `static/images/my_image.png`,

        Images can then be embeded in one of two methods. The path starts from the `images` directory. 

    you can link to it using the `figure` shortcode as `{{< figure src="/images/my_image.png" >}}`.

        Using markdown: `![Optional Image Title](/image/picture.png)`

        Using HTML: `<img src="/images/picture.png">`

        - HTML will also allow the use of standard `img` embedding and customization. 

    2.  Markdown and HTML Files

        Sometimes there are cases where information is stored statically elsewhere. These files are organized in `docs/static`

        ``` markdown

        {% 

            include "relative/path/to/file.html"

        %}

        ```

    -  **Special note for .png files**:  In order to keep the repo size as small as possible, all images

    -  **Special note for .png files**:  In order to keep the repo size as small as possible, all images

       should be png's if possible, and further processed using the [pngquant](https://pngquant.org) compression

       should be png's if possible, and further processed using the [pngquant](https://pngquant.org) compression

       tool.  This will reduce the size by ~3-4x while keeping good quality.  For example, to process and

       tool.  This will reduce the size by ~3-4x while keeping good quality.  For example, to process and

@@ -134,6 +196,20 @@ are used:

       prior to committing the files to the repo, otherwise git will be forced to keep multiple copies as the files

       prior to committing the files to the repo, otherwise git will be forced to keep multiple copies as the files

       are binary.*

       are binary.*

6. Callouts

    These are the callouts used to highlight information and have a few customization options. 

    Callout Template:

    ``` markdown

    !!! type "Optional Title"

        Content goes here. It is important to make sure to have space in the markdown file above and below the callout. 

        Content must also be indented 

    ```

    A full list of callout types and additional usage information is available in the [Material for MkDocs documentation page](https://squidfunk.github.io/mkdocs-material/reference/admonitions/#supported-types).

#### Adding your changes

#### Adding your changes

Once you've got your set of changes, add and commit them to your branch.  Push the branch and

Once you've got your set of changes, add and commit them to your branch.  Push the branch and

archetypes/default.mddeleted100644 → 0

+0
−6

Original line number
Original line
Diff line number
Diff line

---

title: "{{ replace .Name "-" " " | title }}"

date: {{ .Date }}

draft: true

---

config.tomldeleted100644 → 0

+0
−41

Original line number
Original line
Diff line number
Diff line

#baseURL = "http://hcc-docs.unl.edu/"

languageCode = "en-us"

title = "HCC-DOCS"

theme = "docdock"

contentdir    = "content"

layoutdir     = "layouts"

publishdir    = "public"

baseURL = "/"

canonifyURLs = "true"

relativeURLs = "true"

DefaultContentLanguage = "en"

pygmentsCodeFences = true

pygmentsStyle = "monokailight"

defaultContentLanguage = "en"

defaultContentLanguageInSubdir= false

enableMissingTranslationPlaceholders = false

[params]

editURL = "https://git.unl.edu/hcc/hcc-docs/edit/master/content/"

showVisitedLinks = true # default is false

themeStyle = "original" # "original" or "flex" # default "flex"

themeVariant = "blue" # choose theme variant "green", "gold" , "gray", "blue" (default)

ordersectionsby = "weight" # ordersectionsby = "title"

disableHomeIcon = false # default is false

disableSearch = false # default is false

disableNavChevron = false # set true to hide next/prev chevron, default is false

highlightClientSide = false # set true to use highlight.pack.js instead of the default hugo chroma highlighter

menushortcutsnewtab = true # set true to open shortcuts links to a new tab/window

enableGitInfo = true

[outputs]

home = [ "HTML", "RSS", "JSON"]

[[menu.shortcuts]]

name = "hcc.unl.edu"

identifier = "ds"

url = "https://hcc.unl.edu"

weight = 10

content/_footer.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

  title = "Footer"

+++

{{< icon name="copyright-mark" >}} [Holland Computing Center] (https://hcc.unl.edu) | 118 Schorr Center, Lincoln NE 68588 | {{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu) | {{< icon name="phone-alt" >}}402-472-5041

See something wrong?  Help us fix it by [contributing](https://git.unl.edu/hcc/hcc-docs/blob/master/CONTRIBUTING.md)!

content/_header.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Header"

+++

{{< figure src="/images/UNMasterwhite.gif" link="https://nebraska.edu" target="_blank" >}}

### [Holland Computing Center](https://hcc.unl.edu)

#### [HCC-DOCS]({{< relref "/" >}})

content/_index.mddeleted100644 → 0

+0
−106

Original line number
Original line
Diff line number
Diff line

+++

title = "HCC Documentation"

description = "HCC Documentation Home"

weight = "1"

+++

HCC Documentation

============================

The Holland Computing Center supports a diverse collection of research

computing hardware.  Anyone in the University of Nebraska system is

welcome to apply for an account on HCC machines.

Access to these resources is by default shared with the rest of the user

community via various job schedulers. These policies may be found on the

pages for the various resources. Alternatively, a user may buy into an

existing resource, acquiring 'priority access'. Finally, several

machines are available via Condor for opportunistic use. This will allow

users almost immediate access, but the job is subject to preemption.

#### [New Users Sign Up](http://hcc.unl.edu/new-user-request)

#### [Quick Start Guides](/quickstarts)

Which Cluster to Use?

---------------------

**Crane**: Crane is the newest and most powerful HCC resource . If you

are new to using HCC resources, Crane is the recommended cluster to use

initially.  Limitations: Crane has only 2 CPU/16 cores and 64GB RAM per

node. CraneOPA has 2 CPU/36 cores with a maximum of 512GB RAM per node.

**Rhino**: Rhino is intended for large memory (RAM) computing needs.

Rhino has 4 AMD Interlagos CPUs (64 cores) per node, with either 192GB or 256GB RAM per

node in the default partition. For extremely large RAM needs, there is also

a 'highmem' partition with 2 x 512GB and 2 x 1TB nodes.

User Login

----------

For Windows users, please refer to this link [For Windows Users]({{< relref "for_windows_users" >}}).

For Mac or Linux users, please refer to this link [For Mac/Linux Users]({{< relref "for_maclinux_users">}}).

**Logging into Crane or Rhino**

{{< highlight bash >}}

ssh <username>@crane.unl.edu

{{< /highlight >}}

or

{{< highlight bash >}}

ssh <username>@rhino.unl.edu

{{< /highlight >}}

Duo Security

------------

Duo two-factor authentication is **required** for access to HCC

resources. Registration and usage of Duo security can be found in this

section: [Setting up and using Duo]({{< relref "setting_up_and_using_duo">}})

**Important Notes**

-   The Crane and Rhino clusters are separate. But, they are

    similar enough that submission scripts on whichever one will work on

    another, and vice versa (excluding GPU resources and some combinations of

    RAM/core requests).

-   The worker nodes cannot write to the `/home` directories. You must

    use your `/work` directory for processing in your job. You may

    access your work directory by using the command:

{{< highlight bash >}}

$ cd $WORK

{{< /highlight >}}

Resources

---------

- ##### Crane - HCC's newest machine, Crane has 7232 Intel Xeon cores in 452 nodes with 64GB RAM per node.

- ##### Rhino - HCC's AMD-based cluster, intended for large RAM computing needs.

- ##### Red - This cluster is the resource for UNL's [USCMS](https://uscms.org/) Tier-2 site.

- ##### Anvil - HCC's cloud computing cluster based on Openstack

- ##### Glidein - A gateway to running jobs on the OSG, a collection of computing resources across the US.

Resource Capabilities

---------------------

| Cluster | Overview | Processors | RAM | Connection | Storage

| ------- | ---------| ---------- | --- | ---------- | ------

| **Crane**   | 548 node Production-mode LINUX cluster | 452 Intel Xeon E5-2670 2.60GHz 2 CPU/16 cores per node<br> <br>116 Intel Xeon E5-2697 v4 2.3GHz, 2 CPU/36 cores per node<br><br>("CraneOPA") | 452 nodes @ \*64GB<br><br>79 nodes @ \*\*\*256GB<br><br>37 nodes @ \*\*\*\*512GB | QDR Infiniband<br><br>EDR Omni-Path Architecture | ~1.8 TB local scratch per node<br><br>~4 TB local scratch per node<br><br>~1452 TB shared Lustre storage

| **Rhino** | 110 node Production-mode LINUX cluster | 110 AMD Interlagos CPUs (6272 / 6376), 4 CPU/64 cores per node | 106 nodes @ 192GB\*\*/256GB\*\*\* <br><br> 2 nodes @ 512GB\*\*\*\* <br><br> 2 nodes @ 1024GB\*\*\*\*\* | QDR Infiniband | ~1.5TB local scratch per node <br><br> ~360TB shared BeeGFS storage |

| **Red** | 344 node Production-mode LINUX cluster | Various Xeon and  Opteron processors 7,280 cores maximum, actual number of job slots depends on RAM usage | 1.5-4GB RAM per job slot | 1Gb, 10Gb, and 40Gb Ethernet | ~10.8PB of raw storage space |

| **Anvil** | 76 Compute nodes (Partially used for cloud, the rest used for general computing), 12 Storage nodes, 2 Network nodes Openstack cloud | 76 Intel Xeon E5-2650 v3 2.30GHz 2 CPU/20 cores per node | 76 nodes @ 256GB | 10Gb Ethernet | 528 TB Ceph shared storage (349TB available now) |

You may only request the following amount of RAM: <br>

\*62.5GB <br>

\*\*187.5GB <br>

\*\*\*250GB <br>

\*\*\*\*500GB <br>

\*\*\*\*\*1000GB

content/events/2012/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2012"

description = "Historical listing of various HCC events for the year 2012."

weight = "70"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/2013/HCC_Supercomputing_Symposium_2013.mddeleted100644 → 0

+0
−31

Original line number
Original line
Diff line number
Diff line

+++

title = "HCC Supercomputing Symposium 2013"

description = "HCC Supercomputing Symposium 2013"

+++

<strong>Location: Wick Alumni Center, Dana Board Room</strong><br>

The morning sessions will be interactive – attendees should bring a laptop if at all possible!  The first sessions will be primarily planned tutorial format, while the “Open Lab” will involve more individualized help of those in attendance as desired.

| Time    | Title                                                 | Description                                                      |

|---------|-------------------------------------------------------|------------------------------------------------------------------|

| 9 - 10am| Getting started at HCC        | HCC 101 (Linux Primer, Job submission) [Quick Start Guides](https://hcc.unl.edu/docs/quickstarts/)                 |

| 10 - 11am | Job management						  | Creating and deploying parallel jobs at HCC, OSG [Submitting Jobs](https://hcc.unl.edu/docs/guides/submitting_jobs/)  |

| 11 - 11:45am    | HCC Open Lab                                    |      Presentations and help from HCC staff                                          |

| 12 - 1pm     | LUNCH lecture with UNIT         | Carl Lundstedt: How Big Data Allows Us to Investigate the Smallest Things                 |

|1:15 – 2pm   | New User Spotlight – Steve Kolbe, Theatre                           | Media Arts Film Rendering @ HCC                   |

| 2 - 3pm     | State of HCC; Crane Announcement                 | David Swanson                           |

1.  <span style="line-height: 1.4285715;">Getting Started at HCC (9-10am</span>  

    1.  [Getting an account at HCC](https://hcc.unl.edu/new-user-request)

    2.  Logging into HCC resources for [windows](https://hcc.unl.edu/docs/quickstarts/connecting/for_windows_users/) or [Mac/Linux Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_maclinux_users/)

	3.	Basic Linux commands

	4.	[Transferring files to HCC](https://hcc.unl.edu/docs/guides/handling_data/)

2.  Job Management (10-11am)

    1.  [End to End MPI](https://hcc.unl.edu/docs/guides/running_applications/mpi_jobs_on_hcc/) example.  Including how to transfer source code and compile an MPI application on Tusker.

    2.  [Compiling](https://hcc.unl.edu/docs/guides/running_applications/compiling_source_code/compiling_an_openmp_application/) and [submitting](https://hcc.unl.edu/docs/guides/submitting_jobs/submitting_an_openmp_job/) multi-thread and OpenMP applications.

    3.  Using scheduler features such as [Arrays](https://hcc.unl.edu/docs/guides/submitting_jobs/submitting_a_job_array/

) and [Dependencies](https://hcc.unl.edu/docs/guides/submitting_jobs/job_dependencies/

).

    4.  Debugging on Tusker by using [interactive submission](https://hcc.unl.edu/docs/guides/submitting_jobs/submitting_an_interactive_job/). 

        <span style="line-height: 1.4285715;">&lt;lunch&gt;</span>

content/events/2013/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2013"

description = "Historical listing of various HCC events for the year 2013."

weight = "60"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/2013/hcc_condor_workshop_2013.mddeleted100644 → 0

+0
−50

Original line number
Original line
Diff line number
Diff line

+++

title = "HCC Condor Workshop 2013"

description = "HCC Condor Workshop 2013."

+++

Condor

============

Workshop presentation slides

&lt;[condor\_workshop.pdf](https://unl.box.com/s/ou8tf62bqkbrh7yx0cl4me1zbp3z0j08)&gt;

| Time    | Title                   | Description                                    |

|---------|-------------------------|------------------------------------------------|

| 1pm-2pm | Demo/Hand-On Practice   | Executing Condor jobs on HCC supertcomputers   |

| 2pm-3pm | Individual Consultation | Bring your research code on HCC supercomputers |

Password for all demo accounts is **HCC\_condor2013**.

Replace `<group name>` with `demo` and `<username>` with `demoXX` (e.g. `demo01`).

-   Get started with [For Windows Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_windows_users/)

    or [For Mac/Linux Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_maclinux_users/).

-   Sample code for Fortran/C on HCC:

    &lt;[serial\_dir.zip](https://unl.box.com/s/khkpt68pe3k0lu2ythn9kzjeva40itdd)&gt;

-   Sample code for Condor on HCC:

    &lt;[condor\_dir.zip](https://unl.box.com/s/qpvgnqr9ukjcmt0d2trde4qcp5e5ez17)&gt;

-   For more details of this demonstration, see [Condor Jobs on

    HCC](https://hcc.unl.edu/docs/guides/submitting_jobs/condor_jobs_on_hcc/).

MATLAB

============

Workshop presentation slides

&lt;[matlab\_workshop.pdf](https://unl.box.com/s/ou8tf62bqkbrh7yx0cl4me1zbp3z0j08)&gt;

| Time    | Title                   | Description                                       |

|---------|-------------------------|---------------------------------------------------|

| 1pm-2pm | Demo/Hand-On Practice   | Implementing a MATLAB code on HCC supertcomputers |

| 2pm-3pm | Individual Consultation | Bring your MATLAB code on HCC supercomputers      |

Password for all demo accounts is **HCC\_matlab2013**.

Replace `<group name>` with `demo` and `<username>` with `demoXX` (e.g. `demo01`).

-   Get started with [For Windows Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_windows_users/)

    or [For Mac/Linux Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_maclinux_users/).

-   Hands On: [MATLAB on HCC](https://hcc.unl.edu/docs/guides/submitting_jobs/submitting_matlab_jobs/)

-   Sample code for MATLAB on HCC:

    &lt;[matlab\_dir.zip](https://unl.box.com/s/u19fy7cjeswfl1wi7h1nkeeie8gl3z90)&gt;

-   Sample code for Parallel MATLAB Job:

    &lt;[parallel\_matlab\_dir.zip](https://unl.box.com/s/fhl3kf6hg8dmtozkphq3u2r58yexrppe)&gt;

content/events/2013/hcc_matlab_workshop_2013.mddeleted100644 → 0

+0
−29

Original line number
Original line
Diff line number
Diff line

+++

title = "HCC Matlab Workshop 2013"

description = "HCC Matlab Workshop 2013."

+++

Workshop presentation slides

&lt;[matlab\_workshop.pdf](https://unl.box.com/s/lulbnbrnr7xqwufrx5s20gxnyxpykn7j)&gt;

| Time        | Title                   | Description                                      |

|-------------|-------------------------|--------------------------------------------------|

| UNMC        |                         |                                                  |

| 10am - 11am | Demo/Hand-On Practice   | Implementing a MATLAB code on HCC supercomputers |

| 11am - 12pm | Individual Consultation | Bring your MATLAB code on HCC supercomputers     |

| UNO         |                         |                                                  |

| 2pm - 3pm   | Demo/Hand-On Practice   | Implementing a MATLAB code on HCC supercomputers |

| 3pm - 4pm   | Individual Consultation | Bring your MATLAB code on HCC supercomputers     |

Password for all demo accounts is **HCC\_matlab2013**.

Replace `<group name>` with `demo` and `<username>` with `demoXX`

(e.g. `demo01`).

-   Get started with [For Windows Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_windows_users/)

    or [For Mac/Linux Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_maclinux_users/).

-   Hands On: [MATLAB on HCC](https://hcc.unl.edu/docs/guides/submitting_jobs/submitting_matlab_jobs/)

-   Sample code for MATLAB on HCC:

    &lt;[matlab\_dir.zip](https://unl.box.com/s/u19fy7cjeswfl1wi7h1nkeeie8gl3z90)&gt;

-   Sample code for Parallel MATLAB Job:

    &lt;[parallel\_matlab\_dir.zip](https://unl.box.com/s/fhl3kf6hg8dmtozkphq3u2r58yexrppe)&gt;

content/events/2013/hcc_mpi_workshop_2013.mddeleted100644 → 0

+0
−25

Original line number
Original line
Diff line number
Diff line

+++

title = "HCC MPI Workshop 2013"

description = "HCC MPI Workshop 2013."

+++

Workshop presentation slides &lt;[mpi\_workshop.pdf](https://unl.box.com/s/7rufq2a39n2vektg19ko9fjcbios4kqp)&gt;

| Time    | Title                   | Description                                    |

|---------|-------------------------|------------------------------------------------|

| 1pm-2pm | Demo/Hand-On Practice   | Implementing a MPI code on HCC supertcomputers |

| 2pm-3pm | Individual Consultation | Bring your research code on HCC supercomputers |

Password for all demo accounts is **HCC\_mpi2013**.

Replace `<group name>` with `demo` and `<username>` with `demoXX` (e.g. `demo01`).

-   Get started with [For Windows Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_windows_users/)

    or [For Mac/Linux Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_maclinux_users/).

-   Sample code for Fortran/C on HCC:

    &lt;[serial\_dir.zip](https://unl.box.com/s/khkpt68pe3k0lu2ythn9kzjeva40itdd)&gt;

-   Sample code for MPI on HCC:

    &lt;[mpi\_dir.zip](https://unl.box.com/s/fekjdxx82gxp3x75nbsrqmrri52d4zlp)&gt;

-   For more details of this demonstration, see [MPI Jobs on

    HCC](https://hcc.unl.edu/docs/guides/submitting_jobs/submitting_an_mpi_job/).  

content/events/2014/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2014"

description = "Historical listing of various HCC events for the year 2014."

weight = "50"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/2015/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2015"

description = "Historical listing of various HCC events for the year 2015."

weight = "40"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/2016/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2016"

description = "Historical listing of various HCC events for the year 2016."

weight = "30"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/2016/hcc_qiime_workshop_2016.mddeleted100644 → 0

+0
−71

Original line number
Original line
Diff line number
Diff line

+++

title = "HCC QIIME Workshop 2016"

description = "HCC QIIME Workshop 2016."

+++

When:   October 20, 2016

Where:  NIC Food Innovation Campus room 277

             1901 N. 21st St., Lincoln, 68588

[Details and Agenda](https://hcc.unl.edu/hcc-qiime-workshop)

------------------------------------------------------------------------

Materials

---------

Slides for morning

introduction: [QIIME\_2016\_intro.pdf](https://unl.box.com/s/d2o92uoar7x59a7rxl0f3y7o3ncy0swq)

Software Carpentry Unix Shell

Lessons: [http://eharstad.github.io/shell-novice/](http://eharstad.github.io/shell-novice/)

Slides for HCC

Overview: [2016\_Qiime.pdf](https://unl.box.com/s/4gryu7ny8lo95fs8clyu566k2ubbhtef)

QIIME

slides: [QIIME\_Talk\_Oct20.pdf](https://unl.box.com/s/h4zi0h27t1t16u9due3ybfrxmsdebzin)

QIIME

tutorial: [http://bioinformatics.unl.edu/qiime_tutorial.pdf](http://bioinformatics.unl.edu/qiime_tutorial.pdf)

------------------------------------------------------------------------

**What you need to do before Thursday October 20th**

----------------------------------------------------

*Participants will need to bring a laptop with them to the workshop.* In addition, there are a few things that are required to be completed before you arrive:

--------------------------------------------------------------------------------------------------------------------------------------------------------------

1.  To participate in this workshop, you will need to

    [sign up for an HCC account](http://hcc.unl.edu/new-user-request)

    (if you do not already have one). Please complete the account

    application and [DUO two-factor authentication

    setup](https://hcc-docs.unl.edu/display/HCCDOC/Setting+up+and+using+Duo)

    before Thursday. This process requires advisor approval (via email)

    and a visit to the HCC offices in the Schorr Center to activate your

    account, so it can sometimes take a day or more to complete. Please

    plan accordingly. If you still do not have your account set up by

    Thursday, please arrive at the workshop by 8:30AM to get help.

2.  Once you have your HCC account and DUO set up, please make sure that

    you are able to log into the HCC clusters (instructions for [Windows

    Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_windows_users/)

    and [Mac/Linux

    Users](https://hcc.unl.edu/docs/quickstarts/connecting/for_maclinux_users/).

    If you have any problems logging in, please email us at

    [hcc-support@unl.edu](mailto:hcc-support@unl.edu)

3.  On Thursday we will be demonstrating how to transfer files using

    Globus Connect. Before arriving at the workshop, please log in to

    the

    [Globus Web App](https://www.globus.org/app/transfer)

    using your My.UNL credentials (choose *University of

    Nebraska-Lincoln* from the drop down menu).  Next, install the

    Globus Connect Personal app on your laptop (directions for

    [Mac OS X](https://docs.globus.org/how-to/globus-connect-personal-mac/),

    [Linux](https://docs.globus.org/how-to/globus-connect-personal-linux/),

    and

    [Windows](https://docs.globus.org/how-to/globus-connect-personal-windows/)

    so that you can begin using Globus to transfer data to and from your

    laptop.

content/events/2017/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2017"

description = "Historical listing of various HCC events for the year 2017."

weight = "20"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/2017/unk_linear_algebra_feb_28th_2017.mddeleted100644 → 0

+0
−75

Original line number
Original line
Diff line number
Diff line

+++

title = "UNK Linear Algebra, Feb. 28, 2017"

description = "UNK Linear Algebra, Feb. 28, 2017."

+++

**If at any time today you have difficulties or become lost, please

place the <span style="color: rgb(255,0,0);">red</span> sticky note on

top of your monitor and a helper will be around to assist you.**

    For these instructions, any commands to be typed into the terminal will be formatted like this.

**What is a cluster:**

----------------------

![cluster image](/images/cluster_small.png)

(picture courtesy of:

[http://training.h3abionet.org/technical_workshop_2015/?page_id=403](http://training.h3abionet.org/technical_workshop_2015/?page_id=403))

**  

**

**To connect to the Crane cluster:**

------------------------------------

-   insert the Yubikey into the computer's USB drive. There should be a

    small green light in the middle of the Yubikey to indicate it is

    inserted correctly.

-   Open your preferred web browser and navigate to

    [http://go.unl.edu/cranessh](http://go.unl.edu/cranessh)

-   Click "Start SSH session to crane.unl.edu"  

{{% notice info %}}

The link above is no longer available. If you wish to use a terminal in your browser, Sandstone is an option [https://hcc.unl.edu/docs/guides/sandstone/](https://hcc.unl.edu/docs/guides/sandstone/)

{{% /notice %}}

   ![](/images/ssh.png)

-   Click the "Terminal: SSH" icon to begin the SSH session  

    ![](/images/terminalSSH.png)

-   Type in the provided Username and Password. Note that the password

    will not display on screen, but rest assured that even though

    nothing is being output, your password is being entered as you type.

-   At the "Passcode:" prompt, press your finger to the gold circle in

    the middle of the Yubikey until a string of characters appears on

    screen.  

    ![](/images/yubikey.png)

-   If you logged in successfully, your screen should look similar to

    the one belo  

    ![](/images/crane_login.png)

**Linux Commands Reference List:**

----------------------------------

https://hcc.unl.edu/docs/quickstarts/connecting/basic_linux_commands/

**To run MATLAB interactively:**

--------------------------------

-   After logging into the cluster, navigate to your $WORK directory:

    -   cd $WORK

-   Request an interactive job:

    -   srun --reservation=unk --mem=4096 --pty $SHELL

-   Load the MATLAB module:

    -   module load matlab

-   Run MATLAB:

    -   matlab

**To access the MATLAB Tutorial:**

----------------------------------

-   Navigate to your $WORK directory: 

    -   cd $WORK

-   Clone the github repo containing the tutorial files:

    -   git clone https://github.com/unlhcc/HCCWorkshops.git

content/events/2017/unl_r_for_biologists_class_march_8_2017.mddeleted100644 → 0

+0
−114

Original line number
Original line
Diff line number
Diff line

+++

title = "R for Biologists, March 8, 2017"

description = "R for Biologists, March 8, 2017."

+++

**We will be utilizing <span style="color: rgb(255,0,0);">red</span> and

<span style="color: rgb(51,153,102);">green</span> sticky notes today.

If you run into problems or have questions,**

**please place the <span style="color: rgb(255,0,0);">red</span> sticky

note to the back of your computer screen and a helper will assist you.**

If you have not already requested an HCC account under the rcourse998

group, please do so

[here](https://hcc.unl.edu/new-user-request)

If you already have an HCC account and need to be added to the

rcourse998 group, please let us know.

If you have not previously set up Duo Authentication, please ask for

assistance.

**Set up Instructions:**

**Windows:**

For Windows will we use two third party

application **PuTTY** and **WinSCP** for demonstration.

PuTTY:  

&lt;[http://www.putty.org/](http://www.putty.org/)&gt;

WinSCP:

&lt; [http://winscp.net/eng/download.php](http://winscp.net/eng/download.php)&gt;

**Mac/Linux:**

Mac and Linux users will need to download and install **Cyberduck**.

Detailed information for downloading and setting up Cyberduck can be

found here: [For Mac/Linux Users](https://cyberduck.io/)

**Linux Commands Reference List:**

[https://hcc.unl.edu/docs/quickstarts/connecting/basic_linux_commands/](https://hcc.unl.edu/docs/quickstarts/connecting/basic_linux_commands/)  

  **R core and R Studio:**

We will be writing scripts offline in RStudio and then uploading them to

execute them on the cluster. This lesson assumes you have the R core and

RStudio installed. If you do not you can install them here:

R

core: [https://cloud.r-project.org/](https://cloud.r-project.org/)

RStudio: [https://www.rstudio.com/products/rstudio/download/](https://www.rstudio.com/products/rstudio/download/)

**Required Packages:**

We will also be using the dplyr, ggplot2 and maps package. If you do not

have these installed, please install them now. You can do so using the

following commands inside the RStudio console:

    install.packages("dplyr")

    install.packages("ggplot2")

    install.packages("maps")

**What is a cluster:**

![](/images/cluster_small.png)

(picture courtesy

of: [http://training.h3abionet.org/technical_workshop_2015/?page_id=403](http://training.h3abionet.org/technical_workshop_2015/?page_id=403))

### To download the tutorial files:

-   Navigate to your $WORK directory: 

    -   cd $WORK

-   Clone the github repo containing the tutorial files:

    -   git clone https://github.com/unlhcc/HCCWorkshops.git

Take Home Exercise:

[Data Analysis in R](https://unl.box.com/s/8i647f8are21tc11la0jqk2xddlg19wy) - Please note

that the on the bottom of page three, there is a missing parenthesis at

the end of the last command.

The final code chunk should read:

    # Calculate flight age using birthmonth

    age <- data.frame(names(acStart), acStart, stringsAsFactors=FALSE)

    colnames(age) <- c("TailNum", "acStart")

    flights <- left_join(flights, age, by="TailNum")

    flights <- mutate(flights, Age = (flights$Year * 12) + flights$Month - flights$acStart)

 No newline at end of file

content/events/2018/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "2018"

description = "Listing of various HCC events for the year 2018."

weight = "10"

+++

Historical listing of HCC Events

----------

{{% children %}}

content/events/_index.mddeleted100644 → 0

+0
−10

Original line number
Original line
Diff line number
Diff line

+++

title = "Events"

description = "Historical listing of various HCC events."

weight = "30"

+++

Historical listing of HCC Events

----------

{{% children sort="weight" description="true" %}}

content/facilities.mddeleted100644 → 0

+0
−137

Original line number
Original line
Diff line number
Diff line

---

title: "Facilities of the Holland Computing Center"

---

This document details the equipment resident in the Holland Computing Center (HCC) as of November 2018.

HCC has two primary locations directly interconnected by a pair of 10 Gbps fiber optic links (20 Gbps total). The 1800 sq. ft. HCC machine room at the Peter Kiewit Institute (PKI) in Omaha can provide up to 500 kVA in UPS and genset protected power, and 160 ton cooling. A 2200 sq. ft. second machine room in the Schorr Center at the University of Nebraska-Lincoln (UNL) can currently provide up to 100 ton cooling with up to 400 kVA of power. One Brocade MLXe router and two Dell Z9264F-ON core switches in each location provide both high WAN bandwidth and Software Defined Networking (SDN) capability. The Schorr machine room connects to campus and Internet2/ESnet at 100 Gbps while the PKI machine room connects at 10 Gbps. HCC uses multiple data transfer nodes as well as a FIONA (flash IO network appliance) to facilitate end-to-end performance for data intensive workflows.

HCC's resources at UNL include two distinct offerings: Rhino and Red. Rhino is a linux cluster dedicated to general campus usage with 7,040 compute cores interconnected by low-latency Mellanox QDR InfiniBand networking. 360 TB of BeeGFS storage is complemented by 50 TB of NFS storage and 1.5 TB of local scratch per node.  Each compute node is a Dell R815 server with at least 192 GB RAM and 4 Opteron 6272 / 6376 (2.1 / 2.3 GHz) processors.

The largest machine on the Lincoln campus is Red, with 9,536 job slots interconnected by a mixture of 1, 10, and 40 Gbps ethernet. More importantly, Red serves up over 6.6 PB of storage using the Hadoop Distributed File System (HDFS). Red is integrated with the Open Science Grid (OSG), and serves as a major site for storage and analysis in the international high energy physics project known as CMS (Compact Muon Solenoid).

HCC's resources at PKI (Peter Kiewit Institute) in Omaha include Crane, Anvil, Attic, and Common storage.

Crane debuted at 474 on the Top500 list with an HPL benchmark or 121.8 TeraFLOPS. Intel Xeon chips (8-core, 2.6 GHz) provide the processing with 4 GB RAM available per core and a total of 12,236 cores. The cluster shares 1.5 PetaBytes of Lustre storage and contains HCC's GPU resources. We have since expanded the existing cluster: 96 nodes with new Intel Xeon E5-2697 v4 chips and 100GB Intel Omni-Path interconnect were added to Crane. Moreover, Crane has 21 GPU nodes with 57 NVIDIA GPUs in total which enables the most state-of-art research, from drug discovery to deep learning.

Anvil is an OpenStack cloud environment consisting of 1,520 cores and 400TB of CEPH storage all connected by 10 Gbps networking. The Anvil cloud exists to address needs of NU researchers that cannot be served by traditional scheduler-based HPC environments such as GUI applications, Windows based software, test environments, and persistent services. In addition, a project to expand Ceph storage by 1.1 PB is in progress.

Attic and Silo form a near line archive with 1.0 PB of usable storage. Attic is located at PKI in Omaha, while Silo acts as an online backup located in Lincoln. Both Attic and Silo are connected with 10 Gbps network connections.

In addition to the cluster specific Lustre storage, a shared common storage space exists between all HCC resources with 1.9PB capacity.

These resources are detailed further below.

# 1. HCC at UNL Resources

## 1.1 Rhino

* 107 4-socket Opteron 6172 / 6376 (16-core, 2.1 / 2.3 GHz) with 192 or 256 GB RAM

    * 2x with 512 GB RAM, 2x with 1024 GB RAM

* Mellanox QDR InfiniBand

* 1 and 10 GbE networking

    * 5x Dell N3048 switches

* 50TB shared storage (NFS) -> /home

* 360TB BeeGFS storage over Infiniband -> /work

* 1.5TB local scratch

## 1.2 Red

* USCMS Tier-2 resource, available opportunistically via the Open Science Grid

* 60 2-socket Xeon E5530 (2.4GHz) (16 slots per node)

* 16 2-socket Xeon E5520 (2.27 GHz) (16 slots per node)

* 36 2-socket Xeon X5650 (2.67GHz) (24 slots per node)

* 16 2-socket Xeon E5-2640 v3 (2.6GHz) (32 slots per node)

* 40 2-socket Xeon E5-2650 v3 (2.3GHz) (40 slots per node)

* 24 4-socket Opteron 6272 (2.1 GHz) (64 slots per node)

* 28 2-socket Xeon E5-2650 v2 (2.6GHz) (32 slots per node)

* 48 2-socket Xeon E5-2660 (2.2GHz) (32 slots per node)

* 24 2-socket Xeon E5-2660 v4 (2.0GHz) (56 slots per node)

* 2 2-socket Xeon E5-1660 v3 (3.0GHz) (16 slots per node)

* 10.8 PB HDFS storage

* Mix of 1, 10, and 40 GbE networking

    * 1x Dell S6000-ON switch

    * 2x Dell S4048-ON switch

    * 5x Dell S3048-ON switches

    * 2x Dell S4810 switches

    * 2x Dell N3048 switches

## 1.3 Silo (backup mirror for Attic)

* 1 Mercury RM216 2U Rackmount Server 2 Xeon E5-2630 (12-core, 2.6GHz)

* 10 Mercury RM445J 4U Rackmount JBOD with 45x 4TB NL SAS Hard Disks

# 2. HCC at PKI Resources

## 2.1 Crane

* 452 Relion 2840e systems from Penguin

    * 452x with 64 GB RAM

    * 2-socket Intel Xeon E5-2670 (8-core, 2.6GHz)

    * Intel QDR InfiniBand

* 96 nodes from multiple vendor

    * 59x with 256 GB RAM

    * 37x with 512 GB RAM

    * 2-socket Intel Xeon E5-2697 v4 (18-core, 2.3GHz)

    * Intel Omni-Path

* 1 and 10 GbE networking

    * 4x 10 GbE switch

    * 14x 1 GbE switches

* 1500 TB Lustre storage over InfiniBand

* 3 Supermicro SYS-6016GT systems

    * 48 GB RAM

    * 2-socket Intel Xeon E5620 (4-core, 2.4GHz)

    * 2 Nvidia M2070 GPUs

* 3 Supermicro SYS-1027GR-TSF systems

    * 128 GB RAM

    * 2-socket Intel Xeon E5-2630 (6-core, 2.3GHz)

    * 3 Nvidia K20M GPUs

* 1 Supermicro SYS-5017GR-TF systems

    * 32 GB RAM

    * 1-socket Intel Xeon E5-2650 v2 (8-core, 2.6GHz)

    * 2 Nvidia K40C GPUs

* 5 Supermicro SYS-2027GR-TRF systems

    * 64 GB RAM

    * 2-socket Intel Xeon E5-2650 v2 (8-core, 2.6GHz)

    * 4 Nvidia K40M GPUs

* 2 Supermicro SYS-5018GR-T systems

    * 64 GB RAM

    * 2-socket Intel Xeon E5-2620 v4 (8-core, 2.1GHz)

    * 2 Nvidia P100 GPUs

## 2.2 Attic

* 1 Mercury RM216 2U Rackmount Server 2-socket Xeon E5-2630 (6-core, 2.6GHz)

* 10 Mercury RM445J 4U Rackmount JBOD with 45x 4TB NL SAS Hard Disks

## 2.3 Anvil

* 76 PowerEdge R630 systems

    * 76x with 256 GB RAM

    * 2-socket Intel Xeon E5-2650 v3 (10-core, 2.3GHz)

    * Dual 10Gb Ethernet

* 12 PowerEdge R730xd systems

    * 12x with 128 GB RAM

    * 2-socket Intel Xeon E5-2630L v3 (8-core, 1.8GHz)

    * 12x 4TB NL SAS Hard Disks and 2x200 GB SSD

    * Dual 10 Gb Ethernet

* 2 PowerEdge R320 systems

    * 2x with 48 GB RAM

    * 1-socket Intel E5-2403 v3 (4-core, 1.8GHz)

    * Quad 10Gb Ethernet

* 10 GbE networking

    * 6x Dell S4048-ON switches

## 2.4 Shared Common Storage

* Storage service providing 1.9PB usable capacity

* 6 SuperMicro 1028U-TNRTP+ systems

    * 2-socket Intel Xeon E5-2637 v4 (4-core, 3.5GHz)

    * 256 GB RAM

    * 120x 4TB SAS Hard Disks

* 2 SuperMicro 1028U-TNRTP+ systems

    * 2-socket Intel Xeon E5-2637 v4 (4-core, 3.5GHz)

    * 128 GB RAM

    * 6x 200 GB SSD

* Intel Omni-Path

* 10 GbE networking

content/faq/_index.mddeleted100644 → 0

+0
−177

Original line number
Original line
Diff line number
Diff line

+++

title = "FAQ"

description = "HCC Frequently Asked Questions"

weight = "20"

+++

- [I have an account, now what?](#i-have-an-account-now-what)

- [How do I change my password?](#how-do-i-change-my-password)

- [I forgot my password, how can I retrieve it?](#i-forgot-my-password-how-can-i-retrieve-it)

- [I just deleted some files and didn't mean to! Can I get them back?](#i-just-deleted-some-files-and-didn-t-mean-to-can-i-get-them-back)

- [How do I (re)activate Duo?](#how-do-i-re-activate-duo)

- [How many nodes/memory/time should I request?](#how-many-nodes-memory-time-should-i-request)

- [I am trying to run a job but nothing happens?](#i-am-trying-to-run-a-job-but-nothing-happens)

- [I keep getting the error "slurmstepd: error: Exceeded step memory limit at some point." What does this mean and how do I fix it?](#i-keep-getting-the-error-slurmstepd-error-exceeded-step-memory-limit-at-some-point-what-does-this-mean-and-how-do-i-fix-it)

- [I want to talk to a human about my problem. Can I do that?](#i-want-to-talk-to-a-human-about-my-problem-can-i-do-that)

---

#### I have an account, now what?

Congrats on getting an HCC account! Now you need to connect to a Holland

cluster. To do this, we use an SSH connection. SSH stands for Secure

Shell, and it allows you to securely connect to a remote computer and

operate it just like you would a personal machine.

Depending on your operating system, you may need to install software to

make this connection. Check out on Quick Start Guides for information on

how to install the necessary software for your operating system

- [For Mac/Linux Users]({{< relref "for_maclinux_users" >}})

- [For Windows Users]({{< relref "for_windows_users" >}})

#### How do I change my password?

#### I forgot my password, how can I retrieve it?

Information on how to change or retrieve your password can be found on

the documentation page: [How to change your

password]({{< relref "how_to_change_your_password" >}})

All passwords must be at least 8 characters in length and must contain

at least one capital letter and one numeric digit. Passwords also cannot

contain any dictionary words. If you need help picking a good password,

consider using a (secure!) password generator such as

[this one provided by Random.org](https://www.random.org/passwords)

To preserve the security of your account, we recommend changing the

default password you were given as soon as possible.

#### I just deleted some files and didn't mean to! Can I get them back?

That depends. Where were the files you deleted?

**If the files were in your $HOME directory (/home/group/user/):** It's

possible.

$HOME directories are backed up daily and we can restore your files as

they were at the time of our last backup. Please note that any changes

made to the files between when the backup was made and when you deleted

them will not be preserved. To have these files restored, please contact

HCC Support at

{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)

as soon as possible.

**If the files were in your $WORK directory (/work/group/user/):** No.

Unfortunately, the $WORK directories are created as a short term place

to hold job files. This storage was designed to be quickly and easily

accessed by our worker nodes and as such is not conducive to backups.

Any irreplaceable files should be backed up in a secondary location,

such as Attic, the cloud, or on your personal machine. For more

information on how to prevent file loss, check out [Preventing File

Loss]({{< relref "preventing_file_loss" >}}).

#### How do I (re)activate Duo?

**If you have not activated Duo before:**

Please stop by

[our offices](http://hcc.unl.edu/location)

along with a photo ID and we will be happy to activate it for you. If

you are not local to Omaha or Lincoln, contact us at

{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)

and we will help you activate Duo remotely.

**If you have activated Duo previously but now have a different phone

number:**

Stop by our offices along with a photo ID and we can help you reactivate

Duo and update your account with your new phone number.

**If you have activated Duo previously and have the same phone number:**

Email us at

{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)

from the email address your account is registered under and we will send

you a new link that you can use to activate Duo.

#### How many nodes/memory/time should I request?

**Short answer:** We don’t know.

**Long answer:** The amount of resources required is highly dependent on

the application you are using, the input file sizes and the parameters

you select. Sometimes it can help to speak with someone else who has

used the software before to see if they can give you an idea of what has

worked for them.

But ultimately, it comes down to trial and error; try different

combinations and see what works and what doesn’t. Good practice is to

check the output and utilization of each job you run. This will help you

determine what parameters you will need in the future.

For more information on how to determine how many resources a completed

job used, check out the documentation on [Monitoring Jobs]({{< relref "monitoring_jobs" >}}).

#### I am trying to run a job but nothing happens?

Where are you trying to run the job from? You can check this by typing

the command \`pwd\` into the terminal.

**If you are running from inside your $HOME directory

(/home/group/user/)**:

Move your files to your $WORK directory (/work/group/user) and resubmit

your job.

The worker nodes on our clusters have read-only access to the files in

$HOME directories. This means that when a job is submitted from $HOME,

the scheduler cannot write the output and error files in the directory

and the job is killed. It appears the job does nothing because no output

is produced.

**If you are running from inside your $WORK directory:**

Contact us at

{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)

with your login, the name of the cluster you are running on, and the

full path to your submit script and we will be happy to help solve the

issue.

##### I keep getting the error "slurmstepd: error: Exceeded step memory limit at some point." What does this mean and how do I fix it?

This error occurs when the job you are running uses more memory than was

requested in your submit script.

If you specified `--mem` or `--mem-per-cpu` in your submit script, try

increasing this value and resubmitting your job.

If you did not specify `--mem` or `--mem-per-cpu` in your submit script,

chances are the default amount allotted is not sufficient. Add the line

{{< highlight batch >}}

#SBATCH --mem=<memory_amount>

{{< /highlight >}}

to your script with a reasonable amount of memory and try running it again. If you keep

getting this error, continue to increase the requested memory amount and

resubmit the job until it finishes successfully.

For additional details on how to monitor usage on jobs, check out the

documentation on [Monitoring Jobs]({{< relref "monitoring_jobs" >}}).

If you continue to run into issues, please contact us at

{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)

for additional assistance.

#### I want to talk to a human about my problem. Can I do that?

Of course! We have an open door policy and invite you to stop by

[either of our offices](http://hcc.unl.edu/location)

anytime Monday through Friday between 9 am and 5 pm. One of the HCC

staff would be happy to help you with whatever problem or question you

have.  Alternatively, you can drop one of us a line and we'll arrange a

time to meet:  [Contact Us](https://hcc.unl.edu/contact-us).

content/guides/_index.mddeleted100644 → 0

+0
−9

Original line number
Original line
Diff line number
Diff line

+++

title = "Guides"

weight = "20"

+++

In-depth guides to using HCC resources

--------------------------------------

{{% children description="true" %}}

content/guides/anvil/available_images.mddeleted100644 → 0

+0
−36

Original line number
Original line
Diff line number
Diff line

+++

title = "Available images"

description = "HCC-provided images for Anvil"

+++

HCC provides pre-configured images available to researchers.  Below is a

list of available images.

|       Image Name         |  Username to Connect |        Access instructions           |                                 Description                                       |

| ------------------------ | -------------------- | ------------------------------------ | --------------------------------------------------------------------------------- |

| Cloudera 5.12 GNOME      |  `cloudera`          | [X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | Cloudera 5.12 QuickStart VM. *Note*: Follow the X2Go instructions, but choose GNOME for the Session type instead of Xfce.|

| CentOS 7.4               |   `centos`           | `ssh -l centos@<ip address>`         | The CentOS Linux distribution is a stable, predictable, manageable and reproducible platform derived from the sources of Red Hat Enterprise Linux (RHEL).|

| CentOS 6.9	           |   `centos`           | `ssh -l centos@<ip address>`         | **HCC Standard OS**. The CentOS Linux distribution is a stable, predictable, manageable and reproducible platform derived from the sources of Red Hat Enterprise Linux (RHEL).|

| Fedora 26 Cloud	   |   `fedora`           | `ssh -l fedora <ipaddress>`          | Fedora is a Linux distribution developed by the community-supported Fedora Project and sponsored by the Red Hat company.|

| Fedora 26 RStudio (Xfce) |   `fedora`           |[X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | Fedora 26 with the Xfce Desktop Environment pre-installed.|

| CentOS 7.4 Xfce          |   `centos`           |[X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | CentOS 7.4 with the Xfce Desktop Environment pre-installed.|

| CentOS 6.9 Xfce          |   `centos`           |[X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | CentOS 6.9 with the Xfce Desktop Environment pre-installed.|

| Ubuntu 14.04 Xfce        |   `ubuntu`           |[X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | Ubuntu 14.04 with the Xfce Desktop Environment pre-installed.|

| Ubuntu 16.04 Xfce        |   `ubuntu`           |[X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | Ubuntu 16.04 with the Xfce Desktop Environment pre-installed.|

| Ubuntu 17.04 Xfce        |   `ubuntu`           |[X2Go instructions]({{< relref "connecting_to_linux_instances_using_x2go" >}}) | Ubuntu 17.04 with the Xfce Desktop Environment pre-installed.|

| Windows 7	           |   `cloud-user`       |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 7 Enterprise edition with remote desktop access.|

| Windows 10	           |   `cloud-user`       |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 10 LTSB edition with remote desktop access.|

| Windows 7 Matlab         |  `cloud-user`        |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 7 Enterprise with Matlab r2013b, r2014b, r2015b, r2016b, r2017a pre-installed.|

| Windows 10 Matlab	   |  `cloud-user`        |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 10 LTSB with Matlab r2013b, r2014b, r2015b, r2016b, r2017a pre-installed.|

| Windows 7 SAS	           |  `cloud-user`        |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 7 Enterprise with SAS 9.3, 9.4 pre-installed.|

| Windows 10 SAS	   |  `cloud-user`        |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 10 LTSB with Matlab r2013b, r2014b, r2015b, r2016b, r2017a pre-installed.|

| Windows 7 Mathematica	   |  `cloud-user`        |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 7 Enterprise with Mathematica 10.4 and 11.0 pre-installed.|

| Windows 10 Mathematica   |  `cloud-user`        |[Windows instructions]({{< relref "connecting_to_windows_instances" >}})       | Windows 10 LTSB with Mathematica 10.4 and 11.0 pre-installed.|

| Ubuntu Cloud 14.04 LTS   |  `ubuntu`            | `ssh -l ubuntu <ipaddress>`         | Ubuntu Cloud Image from the 14.04 Long Term Support release.|

| Ubuntu Cloud 16.04 LTS   |  `ubuntu`            | `ssh -l ubuntu <ipaddress>`         | Ubuntu Cloud Image from the 16.04 Long Term Support release.|

| Ubuntu Cloud 17.04 LTS   |  `ubuntu`            | `ssh -l ubuntu <ipaddress>`         | Ubuntu Cloud Image from the 17.04 Long Term Support release.|

Additional images can be produced by HCC staff by request at

{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu).

content/guides/anvil/formatting_and_mounting_a_volume_in_linux.mddeleted100644 → 0

+0
−90

Original line number
Original line
Diff line number
Diff line

+++

title = "Formatting and mounting a volume in Linux"

description = "How to format and mount volume as a hard drive in Linux."

+++

{{% notice info %}}

This guide assumes you associated your SSH Key Pair with the instance

when it was created, and that you are connected to the [Anvil VPN]({{< relref "connecting_to_the_anvil_vpn" >}}).

{{% /notice %}}

Once you have [created and attached]({{< relref "creating_and_attaching_a_volume" >}})

your volume, it must be formatted and mounted in your Linux instance to be usable.  This

procedure is identical to what would be done when attaching a second

hard drive to a physical machine.  In this example, a 1GB volume was

created and attached to the instance.  Note that the majority of this

guide is for a newly created volume.  

{{% notice note %}}

**If you are attaching an existing volume with data already on it,

skip to [creating a directory and mounting the volume](#mounting-the-volume).**

{{% /notice %}}

#### Formatting the volume

Follow the relevant guide

([Windows]({{< relref "connecting_to_linux_instances_from_windows">}})

| [Mac]({{< relref "connecting_to_linux_instances_from_mac" >}})) for your

operating system to connect to your instance.  Formatting and mounting

the volume requires root privileges, so first run the

command `sudo su -` to get a root shell.

{{% panel theme="danger" header="**Running commands as root**" %}}**Extreme care should be taken when running commands as `root.`** It is very easy to permanently delete data or cause irreparable damage to your instance.{{% /panel %}}

{{< figure src="/images/anvil-volumes/1-sudo.png" width="576" >}}

Next, you will need to determine what device the volume is presented as

within Linux.  Typically this will be `/dev/vdb`, but it is necessary to

verify this to avoid mistakes, especially if you have more than one

volume attached to an instance.  The command `lsblk` will list the

hard drive devices and partitions.

{{< figure src="/images//anvil-volumes/2-lsblk.png" width="576" >}}

Here there is a completely empty (no partitions) disk device matching

the 1GB size of the volume, so `/dev/vdb` is the correct device.

The `parted` utility will first be used to label the device and then create a partition.

{{< highlight bash >}}

parted /dev/vdb mklabel gpt

parted /dev/vdb mkpart primary 0% 100%

{{< /highlight >}}

{{< figure src="/images/anvil-volumes/3-mkpart.png" width="576" >}}

Now that a partition has been created, it can be formatted.  Here, the

ext4 filesystem will be used.  This is the default filesystem used by

many Linux distributions including CentOS and Ubuntu, and is a good

general choice.  An alternate filesystem may be used by running a

different format command.  To format the partition using ext4, run the

command `mkfs.ext4 /dev/vdb1`.  You will see a progress message and then

be returned to the shell prompt.

{{< figure src="/images/anvil-volumes/4-mkfs.png" width="576" >}}

#### Mounting the volume

{{% notice note %}}

If you are attaching a pre-existing volume, start here.

{{% /notice %}}

Finally, the formatted partition must be mounted as a directory to be

used.  By convention this is done under `/mnt`, but you may choose to

mount it elsewhere depending on the usage.  Here, a directory

called `myvolume` will be created and the volume mounted there.  Run the

following commands to make the directory and mount the volume:

{{< highlight bash >}}

mkdir /mnt/myvolume

mount /dev/vdb1 /mnt/myvolume

{{< /highlight >}}

{{< figure src="/images/anvil-volumes/5-mount.png" width="576" >}}

Running the command `df -h` should then show the new mounted empty

volume.  

{{< figure src="/images/anvil-volumes/6-df.png" width="576" >}}

The volume can now be used.

content/guides/handling_data/_index.mddeleted100644 → 0

+0
−130

Original line number
Original line
Diff line number
Diff line

+++

title = "Handling Data"

description = "How to work with and transfer data to/from HCC resources."

weight = "30"

+++

{{% panel theme="danger" header="**Sensitive and Protected Data**" %}}HCC currently has *no storage* that is suitable for **HIPAA** or other **PID** data sets.  Users are not permitted to store such data on HCC machines.{{% /panel %}}

All HCC machines have three separate areas for every user to store data,

each intended for a different purpose.   In addition, we have a transfer

service that utilizes [Globus Connect]({{< relref "globus_connect" >}}).

{{< figure src="/images/35325560.png" height="500" class="img-border">}}

---

### Home Directory

{{% notice info %}}

You can access your home directory quickly using the $HOME environmental

variable (i.e. '`cd $HOME'`).

{{% /notice %}}

Your home directory (i.e. `/home/[group]/[username]`) is meant for items

that take up relatively small amounts of space.  For example:  source

code, program binaries, configuration files, etc.  This space is

quota-limited to **20GB per user**.  The home directories are backed up

for the purposes of best-effort disaster recovery.  This space is not

intended as an area for I/O to active jobs.  **/home** is mounted

**read-only** on cluster worker nodes to enforce this policy.

---

### Common Directory

{{% notice info %}}

You can access your common directory quickly using the $COMMON

environmental variable (i.e. '`cd $COMMON`')

{{% /notice %}}

The common directory operates similarly to work and is mounted with

**read and write capability to worker nodes all HCC Clusters**. This

means that any files stored in common can be accessed from Crane or Rhino,

making this directory ideal for items that need to be

accessed from multiple clusters such as reference databases and shared

data files.

{{% notice warning %}}

Common is not designed for heavy I/O usage. Please continue to use your

work directory for active job output to ensure the best performance of

your jobs.

{{% /notice %}}

Quotas for common are **30 TB per group**, with larger quotas available

for purchase if needed. However, files stored here will **not be backed

up** and are **not subject to purge** at this time. Please continue to

backup your files to prevent irreparable data loss.

Additional information on using the common directories can be found in

the documentation on [Using the /common File System]({{< relref "using_the_common_file_system" >}})

---

### High Performance Work Directory

{{% notice info %}}

You can access your work directory quickly using the $WORK environmental

variable (i.e. '`cd $WORK'`).

{{% /notice %}}

{{% panel theme="danger" header="**File Loss**" %}}The `/work` directories are **not backed up**. Irreparable data loss is possible with a mis-typed command. See [Preventing File Loss]({{< relref "preventing_file_loss" >}}) for strategies to avoid this.{{% /panel %}}

Every user has a corresponding directory under /work using the same

naming convention as `/home` (i.e. `/work/[group]/[username]`).  We

encourage all users to use this space for I/O to running jobs.  This

directory can also be used when larger amounts of space are temporarily

needed.  There is a **50TB per group quota**; space in /work is shared

among all users.  It should be treated as short-term scratch space, and

**is not backed up**.  **Please use the `hcc-du` command to check your

own and your group's usage, and back up and clean up your files at

reasonable intervals in $WORK.**

---

### Purge Policy

HCC has a **purge policy on /work** for files that become dormant.

 After **6 months of inactivity on a file (26 weeks)**, an automated

purge process will reclaim the used space of these dormant files.  HCC

provides the **`hcc-purge`** utility to list both the summary and the

actual file paths of files that have been dormant for **24 weeks**.

 This list is periodically generated; the timestamp of the last search

is included in the default summary output when calling `hcc-purge` with

no arguments.  No output from `hcc-purge` indicates the last scan did

not find any dormant files.  `hcc-purge -l` will use the less pager to

list the matching files for the user.  The candidate list can also be

accessed at the following path:` /lustre/purge/current/${USER}.list`.

 This list is updated twice a week, on Mondays and Thursdays.

{{% notice warning %}}

`/work` is intended for recent job output and not long term storage. Evidence of circumventing the purge policy by users will result in consequences including account lockout.

{{% /notice %}}

If you have space requirements outside what is currently provided,

please

email <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a> and

we will gladly discuss alternatives.

---

### [Attic]({{< relref "using_attic" >}})

Attic is a near line archive available for purchase at HCC.  Attic

provides reliable large data storage that is designed to be more

reliable then `/work`, and larger than `/home`. Access to Attic is done

through [Globus Connect]({{< relref "globus_connect" >}}).

More details on Attic can be found on HCC's

<a href="https://hcc.unl.edu/attic" class="external-link">Attic</a>

website.

---

### [Globus Connect]({{< relref "globus_connect" >}})

For moving large amounts of data into or out of HCC resources, users are

highly encouraged to consider using [Globus

Connect]({{< relref "globus_connect" >}}).

---

### Using Box

You can use your [UNL

Box.com]({{< relref "integrating_box_with_hcc" >}}) account to download and

upload files from any of the HCC clusters.

content/guides/handling_data/data_for_unmc_users_only.mddeleted100644 → 0

+0
−47

Original line number
Original line
Diff line number
Diff line

+++ 

title = "Data for UNMC Users Only"

description= "Data storage options for UNMC users"

weight = 50

+++

{{% panel theme="danger" header="Sensitive and Protected Data" %}} HCC currently has no storage that is suitable for HIPAA or other PID

data sets.  Users are not permitted to store such data on HCC machines.

Crane have a special directory, only for UNMC users. Please

note that this filesystem is still not suitable for HIPAA or other PID

data sets.

{{% /panel %}}

---

### Transferring files to this machine from UNMC.

You will need to email us

at <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a> to

gain access to this machine. Once you do, you can sftp to 10.14.250.1

and upload your files. Note that sftp is your only option. You may use

different sftp utilities depending on your platform you are logging in

from. Email us if you need help with this. Once you are logged in, you

should be at `/volumes/UNMC1ZFS/[group]/[username]`, or

`/home/[group]/[username]`. Both are the same location and you will be

allowed to write files there.

For Windows, learn more about logging in and uploading files

[here](https://hcc-docs.unl.edu/display/HCCDOC/For+Windows+Users).

Using your uploaded files on Crane.

---------------------------------------------

Using your

uploaded files is easy. Just go to

`/shared/unmc1/[group]/[username]` and your files will be in the same

place. You may notice that the directory is not available at times. This

is because the unmc1 directory is automounted. This means, if you try to

go to the directory, it will show up. Just "`cd`" to

`/shared/unmc1/[group]/[username]` and all of the files will be

there.

If you have space requirements outside what is currently provided,

please

email <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a> and

we will gladly discuss alternatives.

content/guides/handling_data/globus_connect/activating_hcc_cluster_endpoints.mddeleted100644 → 0

+0
−39

Original line number
Original line
Diff line number
Diff line

+++

title = "Activating HCC Cluster Endpoints"

description = "How to activate HCC endpoints on Globus"

weight = 20

+++

You will not be able to transfer files to or from an HCC endpoint using Globus Connect without first activating the endpoint.  Endpoints are available for Crane (`hcc#crane`), Rhino, (`hcc#rhino`), and Attic (`hcc#attic`).  Follow the instructions below to activate any of these endpoints and begin making transfers.

1.  [Sign in](https://www.globus.org/SignIn) to your Globus account using your campus credentials or your Globus ID (if you have one).  Then click on 'Endpoints' in the left sidebar.  

{{< figure src="/images/Glogin.png" >}}    

{{< figure src="/images/endpoints.png" >}}

2.  Find the endpoint you want by entering '`hcc#crane`', '`hcc#rhino`', or '`hcc#attic`' in the search box and hit 'enter'.  Once you have found and selected the endpoint, click the green 'activate' icon. On the following page, click 'continue'.

{{< figure src="/images/activateEndpoint.png" >}}

{{< figure src="/images/EndpointContinue.png" >}}

3.  You will be redirected to the HCC Globus Endpoint Activation page.  Enter your *HCC* username and password (the password you usually use to log into the HCC clusters).

{{< figure src="/images/hccEndpoint.png" >}}

4.  Next you will be prompted to

    provide your *Duo* credentials.  If you use the Duo Mobile app on

    your smartphone or tablet, select 'Duo Push'.  Once you approve the notification that is sent to your phone,

    the activation will be complete.  If you use a Yubikey for

    authentication, select the 'Passcode' option and then press your

    Yubikey to complete the activation. Upon successful activation, you

    will be redirected to your Globus *Manage Endpoints* page.  

{{< figure src="/images/EndpointPush.png" >}}

{{< figure src="/images/endpointComplete.png" >}}

The endpoint should now be ready

and will not have to be activated again for the next 7 days.  

To transfer files between any two HCC clusters, you will need to

activate both endpoints individually. 

Next, learn how to [make file transfers between HCC endpoints]({{< relref "file_transfers_between_endpoints" >}}) or how to [transfer between HCC endpoints and a personal computer]({{< relref "file_transfers_to_and_from_personal_workstations" >}}).

---

content/guides/handling_data/globus_connect/file_sharing.mddeleted100644 → 0

+0
−55

Original line number
Original line
Diff line number
Diff line

+++

title = "File Sharing"

description = "How to share files using Globus"

weight = 50

+++

If you would like another colleague or researcher to have access to your

data, you may create a shared endpoint on Crane, Rhino, or Attic. You can personally manage access to this endpoint and

give access to anybody with a Globus account (whether or not

they have an HCC account).  *Please use this feature responsibly by

sharing only what is necessary and granting access only to trusted

users.*

{{% notice info %}}

Shared endpoints created in your `home` directory on HCC servers (with

the exception of Attic) are *read-only*. You may create readable and

writable shared endpoints in your `work` directory (or `/shared`).

{{% /notice %}}

1.  Sign in to your Globus account, click on the 'Endpoints' tab

    and search for the endpoint that you will use to host your shared

    endpoint.  For example, if you would like to share data in your

    Crane `work` directory, search for the `hcc#crane` endpoint.  Once

    you have found the endpoint, it will need to be activated if it has

    not been already (see [endpoint activation instructions

    here]({{< relref "activating_hcc_cluster_endpoints" >}})).

     If it is already activated, select the endpoint by clicking on the

    name.  Then select the 'share' button on the right sidebar.

{{< figure src="/images/sharedEndpoint.png" >}}

{{< figure src="/images/shareButton.png" >}}

2.  In the 'Path' box, enter the full path to the directory you

    would like to share.  Only files under this directory will be shared

    to the endpoint users you grant access.  Enter a descriptive endpoint

    name and provide a

    short description of the endpoint if you wish.  Finally, click 'Create Share'.

{{< figure src="/images/createShare.png" >}}

3.  Type the Globus ID (or group name) of the user (or group) to whom you would like to grant

    access to this endpoint.  Next enter the *relative path* of the

    directory that this user should be able to access.  For example, if

    the source path of your shared endpoint

    is `/work/<groupid>/<userid>/share` but you would like your

    colleague to only have access

    to `/work/<groupid>/<userid>/share/dataX`, then the 'Path' should be

    entered as simply `/dataX`.  Finally, click the blue 'Add Permission' button.

     You should see the user or group added to the list.

{{< figure src="/images/addPermission.png" >}}

{{< figure src="/images/sharedGroup.png" >}}

---

content/guides/handling_data/globus_connect/file_transfers_between_endpoints.mddeleted100644 → 0

+0
−50

Original line number
Original line
Diff line number
Diff line

+++

title = "File Transfers Between Endpoints"

description = "How to transfer files between HCC clusters using Globus"

weight = 30

+++

To transfer files between HCC clusters, you will first need to

[activate]({{< relref "activating_hcc_cluster_endpoints" >}}) the

two endpoints you would like to use (the available endpoints

are: `hcc#crane` `hcc#rhino`, and `hcc#attic`).  Once

that has been completed, follow the steps below to begin transferring

files.  (Note: You can also transfer files between an HCC endpoint and

any other Globus endpoint for which you have authorized access.  That

may include a [personal

endpoint]({{< relref "file_transfers_to_and_from_personal_workstations" >}}),

a [shared

endpoint]({{< relref "file_sharing" >}}),

or an endpoint on another computing resource or cluster.  Once the

endpoints have been activated, the file transfer process is generally

the same regardless of the type of endpoints you use.  For demonstration

purposes we use two HCC endpoints.)

1.  Once both endpoints for the desired file transfer have been

    activated, [sign in](https://www.globus.org/SignIn) to

    your Globus account (if you are not already) and select

	"Transfer or Sync to.." from the right sidebar. If you have

	a small screen, you may have to click the menu icon 

	first.

{{< figure src="/images/Transfer.png">}}

2.  Enter the names of the two endpoints you would like to use, or

    select from the drop-down menus (for

    example, `hcc#attic` and `hcc#crane`).  Enter the

    directory paths for both the source and destination (the 'from' and

    'to' paths on the respective endpoints). Press 'Enter' to view files

    under these directories.  Select the files or directories you would

    like to transfer (press *shift* or *control* to make multiple

    selections) and click the blue highlighted arrow to start the

    transfer.  

{{< figure src="/images/startTransfer.png" >}}

3.  Globus will display a message when your transfer has completed 

	(or in the unlikely event that it was unsuccessful), and you will 

	also receive an email. Select the 'refresh' icon to see your file

	in the destination folder.

{{< figure src="/images/transferComplete.png" >}}      

--- 

content/guides/handling_data/integrating_box_with_hcc.mddeleted100644 → 0

+0
−92

Original line number
Original line
Diff line number
Diff line

+++

title = "Integrating Box with HCC"

description = "How to integrate Box with HCC"

weight = 30

+++

UNL has come to an arrangement

with <a href="https://www.box.com/" class="external-link">Box.com</a> to

provide unlimited cloud storage to every student, staff, and faculty

member.  This can be useful when used with jobs to automatically upload

results when the job has completed.  Combined with

<a href="https://sites.box.com/sync4/" class="external-link">Box Sync</a>,

the uploaded files can be sync'd to your laptop or desktop upon job

completion. The upload and download speed of Box is about 20 to 30 MB/s

in good network traffic conditions.  Users can use a tool called lftp to transfer files between HCC clusters and their Box accounts.

---

### Step-by-step guide for Lftp

1.  You need to create your UNL [Box.com](https://www.box.com/) account [here](https://box.unl.edu/).

2.  Since we are going to be using [webdav](https://en.wikipedia.org/wiki/WebDAV) protocol to access your [Box.com](https://www.box.com/) storage, you need to create an **External Password**.  In the [Box.com](https://www.box.com/) interface, you can create it at **[Account Settings](https://unl.app.box.com/settings) > Create External Password.**

{{< figure src="/images/box_create_external_password.png" class="img-border" >}}

3.  After logging into the cluster of your choice, load the `lftp` module by entering the command below at the prompt:

{{% panel theme="info" header="Load the lftp module" %}}

{{< highlight bash >}}

module load lftp

{{< /highlight >}}

{{% /panel %}}

4.  Connect to Box using your full email as the username and external password you created:

{{% panel theme="info" header="Connect to Box" %}}

{{< highlight bash >}}

lftp -u <username>,<password> ftps://ftp.box.com

{{< /highlight >}}

{{% /panel %}}

5.  Test the connection by running the `ls` command.  You should see a listing of your Box files.  Assuming it works, add a bookmark named "box" to use when connecting later:

{{% panel theme="info" header="Add lftp bookmark" %}}

{{< highlight bash >}}

lftp demo2@unl.edu@ftp.box.com:/> bookmark add box

{{< /highlight >}}

{{% /panel %}}

6.  Exit `lftp` by typing `quit`.  To reconnect later, use bookmark name:

{{% panel theme="info" header="Connect using bookmark name" %}}

{{< highlight bash >}}

lftp box

{{< /highlight >}}

{{% /panel %}}

7.  To upload or download files, use `get` and `put` commands.  For example:

{{% panel theme="info" header="Transferring files" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ lftp box

lftp demo2@unl.edu@ftp.box.com:/> put myfile.txt

lftp demo2@unl.edu@ftp.box.com:/> get my_other_file.txt

{{< /highlight >}}

{{% /panel %}}

8.  To download directories, use the `mirror` command.  To upload directories, use the `mirror` command with the `-R` option.  For example, to download a directory named `my_box-dir` to your current directory:

{{% panel theme="info" header="Download a directory from Box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ lftp box

lftp demo2@unl.edu@ftp.box.com:/> mirror my_box_dir

{{< /highlight >}}

{{% /panel %}}

To upload a directory named `my_hcc_dir` to Box, use `mirror` with the `-R` option:

{{% panel theme="info" header="Upload a directory to Box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ lftp box

lftp demo2@unl.edu@ftp.box.com:/> mirror -R my_hcc_dir 

{{< /highlight >}}

{{% /panel %}}

9.  Lftp also supports using scripts to transfer files.  This can be used to automatically download or upload files during jobs.  For example, create a file called "transfer.sh" with the following lines:

{{% panel theme="info" header="transfer.sh" %}}

{{< highlight bash >}}

open box

get some_input_file.tar.gz

put my_output_file.tar.gz

{{< /highlight >}}

{{% /panel %}}

To run this script, do:

{{% panel theme="info" header="Run transfer.sh" %}}

{{< highlight bash >}}

module load lftp

lftp -f transfer.sh 

{{< /highlight >}}

{{% /panel %}}

content/guides/handling_data/using_rclone_with_hcc.mddeleted100755 → 0

+0
−139

Original line number
Original line
Diff line number
Diff line

+++

title = "Using Rclone for File Transfer"

description = "How to use Rclone with HCC"

weight = 9

+++

Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box, Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task. Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/). 

This tool can be used to transfer files between HCC clusters and outside cloud providers, such as Box.

---

### Setup RClone

1.  You need to create your UNL [Box.com](https://www.box.com/) account [here](https://box.unl.edu/).

2.  Due to the clusters being remote machines, Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize box. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.

3.  After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:

{{% panel theme="info" header="Load the Rclone module" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ module load rclone

{{< /highlight >}}

{{% /panel %}}

4.  We will need to start the basic configuration for box. To do this run `rclone config`:

{{% panel theme="info" header="Load the rclone config" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone config

{{< /highlight >}}

{{% /panel %}}

5.  In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "UNLBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. When you are prompted for auto config, select `N` and switch to a terminal on your local machine:

{{% panel theme="info" header="Configure box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone config

No remotes found - make a new one

n) New remote

s) Set configuration password

q) Quit config

n/s/q> n

name> UNLBox

Type of storage to configure.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value

 6 / Box

   \ "box"

Storage> 6

Box App Client Id.

Leave blank normally.

Enter a string value. Press Enter for the default ("").

client_id>

Box App Client Secret

Leave blank normally.

Enter a string value. Press Enter for the default ("").

client_secret>

Edit advanced config? (y/n)

y) Yes

n) No

y/n> y

Cutoff for switching to multipart upload (>= 50MB).

Enter a size with suffix k,M,G,T. Press Enter for the default ("50M").

upload_cutoff> 15G

Max number of times to try committing a multipart file.

Enter a signed integer. Press Enter for the default ("100").

commit_retries>

Remote config

Use auto config?

 * Say Y if not sure

 * Say N if you are working on a remote or headless machine

y) Yes

n) No

y/n> n

For this to work, you will need Rclone available on a machine that has a web browser available.

Execute the following on your machine:

        rclone authorize "box"

Then paste the result below:

result>

{{< /highlight >}}

{{% /panel %}}

6.  Run `rclone authorize "box"` on the local machine. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your UNL e-mail address. You will be taken to sign into UNL's Box using your **Canvas** credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.

{{% panel theme="info" header="List contents of Box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone authorize "box"

{{< /highlight >}}

{{% /panel %}}

{{< figure src="/images/BoxSSO.png" height="500" class="img-border">}}

{{% panel theme="info" header="Local Config" %}}

{{< highlight bash >}}

[demo2@local.machine ~]$ rclone authorize "box"

If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth

Log in and authorize Rclone for access

Waiting for code...

{{< /highlight >}}

{{% /panel %}}

For other services, please refer to the [rclone documentation](https://rclone.org/). 

7.  Test the connection by running the `ls` command.  You should see a listing of your Box files.

{{% panel theme="info" header="List contents of Box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone ls UNLBox:/

{{< /highlight >}}

{{% /panel %}}

8.  To upload or download files, use the `clone` command.  For example:

{{% panel theme="info" header="Transferring files" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone copy UNLBox:/SomeFile.txt ./

[demo2@login.crane ~]$ rclone copy ./SomeFile.txt UNLBox:/

{{< /highlight >}}

{{% /panel %}}

9.  To download directories, use the `clone` command and use directory names over file.  This copies the contents of the folders, so you need to specify a destination folder.

{{% panel theme="info" header="Download a directory from Box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone copy UNLBox:/my_hcc_dir ./my_hcc_dir

{{< /highlight >}}

{{% /panel %}}

To upload a directory named `my_hcc_dir` to Box, use `clone`.

{{% panel theme="info" header="Upload a directory to Box" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone copy ./my_hcc_dir UNLBox:/my_hcc_dir

{{< /highlight >}}

{{% /panel %}}

11.  Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The exmaple below would sync the files of the local directory to the remote directory on box.

{{% panel theme="info" header="transfer.sh" %}}

{{< highlight bash >}}

[demo2@login.crane ~]$ rclone sync ./my_hcc_dir UNLBox:/my_hcc_dir

{{< /highlight >}}

{{% /panel %}}

content/guides/handling_data/using_the_common_file_system.mddeleted100644 → 0

+0
−87

Original line number
Original line
Diff line number
Diff line

+++

title = "Using the /common File System"

description = "How to use HCC's /common file system"

weight = 70

+++

### Quick overview: 

-   Connected read/write to all HCC HPC cluster resources – you will see

    the same files "in common" on any HCC cluster (i.e. Crane and Tukser).

-   30 TB Per-group quota at no charge – larger quota available for

    $100/TB/year

-   No backups are made!  Don't be silly!  Precious data should still be

    stored / backed up elsewhere such as on Attic.  Please. 

-   No purge!  So, while your files just might be lost in the case of

    disk failure or user error, they won't be removed by the purge

    scripts. 

---

### Accessing common

{{% notice info %}}

Your `/common` directory can be accessed via the `$COMMON` environment variable, i.e. `cd $COMMON.`

{{% /notice %}}

---

### How should I use `/common`? 

-   Store things that are routinely needed on multiple clusters.

-   `/common` is a network attached FS, so limit the number of files per

    directory (1 million files in a directory is a very bad idea).

-   **If you are accessing `/common` for a job, you will need to add a

    line to your submission script!  **

    -   We have each user check out a "license" to access `/common` for a

        given job.

    -   This allows us to know exactly who is accessing it, and for how

        long, in case of the need for a shut down so we can try to avoid

        killing jobs whenever possible.

    -   It also allows us to limit how many jobs can hammer this single

        filesystem so it remains healthy and happy. 

To gain access to the path on worker

nodes, a job must be submitted

with the following SLURM directive:

{{% panel theme="info" header="**SLURM Submit File**" %}}

{{< highlight bash >}}

#SBATCH --licenses=common

{{< /highlight >}}

{{% /panel %}}

If a job lacks the above SLURM directive, `/common` will not be accessible from the worker nodes.  (Briefly, this construct will allow us

to quickly do maintenance on a single cluster without having to unmount

`$COMMON` from all HCC resources).

---

### What should I **not** do when using `/common`?

-   Don't use it for high I/O work flows, use `/work` for that – `/common`

    should mostly be used to read largely static files or data.

-   Do not expect your compiled program binaries to work everywhere!

     `/common` is available on machines with different cpu architecture,

    different network connections, and so on.  *caveat emptor!*

    -   Serial codes will not be optimized for all clusters.

    -   MPI codes, in particular, will likely not work unless recompiled

        for each cluster.

    -   If you use  `module` things should be just fine!

---

### /common and used space reporting ###

The /common file system has the capability to compress files so they store

less data on the underlying disk storage.  Tools like `du` will report the

true amount of space consumed by files by default.  If the files have been

compressed before being stored to disk, the report will appear smaller

than what may be expected.  Passing the `--apparent-size` argument to

`du` will cause the report to be the uncompressed size of consumed space.

{{< highlight bash >}}

$ pwd

/common/demo/demo01

$ python -c 'print "Hello World!\n" * 2**20,' > hello_world.txt

$ ls -lh hello_world.txt 

-rw-r--r-- 1 demo01 demo 13M Mar  7 12:55 hello_world.txt

$ du -sh hello_world.txt 

2.0K    hello_world.txt

$ du -sh --apparent-size hello_world.txt 

13M     hello_world.txt

{{< /highlight >}}

content/guides/running_applications/Jupyter.mddeleted100644 → 0

+0
−58

Original line number
Original line
Diff line number
Diff line

+++

title = "Jupyter Notebooks on Crane"

description = "How to access and use a Jupyter Notebook"

weight = 20

+++

- [Connecting to Crane] (#connecting-to-crane)

- [Running Code] (#running-code)

- [Opening a Terminal] (#opening-a-terminal)

- [Using Custom Packages] (#using-custom-packages)

## Connecting to Crane

-----------------------

 Jupyter defines it's notebooks ("Jupyter Notebooks") as 

	an open-source web application that allows you to create and share documents that contain live code,

	equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation,

	statistical modeling, data visualization, machine learning, and much more.

1.  To open a Jupyter notebook, [Sign in](https://crane.unl.edu) to crane.unl.edu using your hcc credentials (NOT your 

	campus credentials).

{{< figure src="/images/jupyterLogin.png" >}}

2.	Select your preferred authentication method.

	{{< figure src="/images/jupyterPush.png" >}}  

3.	Choose a job profile. Select "Noteboook via SLURM Job | Small (1 core, 4GB RAM, 8 hours)" for light tasks such as debugging or small-scale testing.

Select the other options based on your computing needs. Note that a SLURM Job will save to your "work" directory.

{{< figure src="/images/jupyterjob.png" >}}

## Running Code

1.  Select the "New" dropdown menu and select the file type you want to create.   

{{< figure src="/images/jupyterNew.png" >}}

2.	A new tab will open, where you can enter your code. Run your code by selecting the "play" icon.

{{< figure src="/images/jupyterCode.png">}}

## Opening a Terminal

1.	From your user home page, select "terminal" from the "New" drop-down menu.

{{< figure src="/images/jupyterTerminal.png">}}

2.	A terminal opens in a new tab. You can enter [Linux commands] ({{< relref "basic_linux_commands" >}})

 at the prompt.

{{< figure src="/images/jupyterTerminal2.png">}}

## Using Custom Packages

Many popular `python` and `R` packages are already installed and available within Jupyter Notebooks. 

However, it is possible to install custom packages to be used in notebooks by creating a custom Anaconda 

Environment. Detailed information on how to create such an environment can be found at

 [Using an Anaconda Environment in a Jupyter Notebook on Crane]({{< relref "using_anaconda_package_manager/#using-an-anaconda-environment-in-a-jupyter-notebook-on-crane" >}}).

---

content/guides/running_applications/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Running Applications"

description = "How to run various applications on HCC resources."

weight = "20"

+++

{{% children %}}

content/guides/running_applications/allinea_profiling_and_debugging/_index.mddeleted100644 → 0

+0
−18

Original line number
Original line
Diff line number
Diff line

+++

title = "Allinea Profiling & Debugging Tools"

description = "How to use the Allinea suite of tools for profiling and debugging."

+++

HCC provides both the Allinea Forge suite and Performance Reports to

assist with debugging and profiling C/C++/Fortran code.  These tools

support single-threaded, multi-threaded (pthreads/OpenMP), MPI, and CUDA

code.  The Allinea Forge suite consists of two programs:  DDT for

debugging and MAP for profiling.  The Performance Reports software

provides a convenient way to profile HPC applications.  It generates an

easy-to-read single-page HTML report.

For information on using each tool, see the following pages.

[Using Allinea Forge via Reverse Connect]({{< relref "using_allinea_forge_via_reverse_connect" >}})

[Allinea Performance Reports]({{< relref "allinea_performance_reports" >}})

content/guides/running_applications/available_software_for_crane.mddeleted100644 → 0

+0
−45

Original line number
Original line
Diff line number
Diff line

+++

title = "Available Software for Crane"

description = "List of available software for crane.unl.edu."

scripts = ["https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.min.js", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-pager.min.js","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-filter.min.js","/js/sort-table.js"]

css = ["http://mottie.github.io/tablesorter/css/theme.default.css","https://mottie.github.io/tablesorter/css/theme.dropbox.css", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/jquery.tablesorter.pager.min.css","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/filter.formatter.min.css"]

+++

{{% notice tip %}}

HCC provides some software packages via the Singularity container

software. If you do not see a desired package in the module list below,

please check the [Using Singularity]({{< relref "using_singularity" >}})

page for the software list there.

{{% /notice %}}

{{% panel theme="warning" header="Module prerequisites" %}}

If a module lists one or more prerequisites, the prerequisite module(s)

must be loaded before or along with, that module.

For example, the `cdo/2.1` modules requires `compiler/pgi/13.` To load

the cdo module, doing either

`module load compiler/pgi/13`

`module load cdo/2.1`

or

`module load compiler/pgi/13 cdo/2.1` (Note the prerequisite module

**must** be first.)

is acceptable.

{{% /panel %}}

{{% panel theme="info" header="Multiple versions" %}}

Some packages list multiple compilers for prerequisites. This means that

the package has been built with each version of the compilers listed.

{{% /panel %}}

{{% panel theme="warning" header="Custom GPU Anaconda Environment" %}}

If you are using custom GPU Anaconda Environment, the only module you need to load is `anaconda`:

`module load anaconda`

{{% /panel %}}

{{< table url="http://crane-head.unl.edu:8192/lmod/spider/json" >}}

content/guides/running_applications/available_software_for_rhino.mddeleted100644 → 0

+0
−45

Original line number
Original line
Diff line number
Diff line

+++

title = "Available Software for Rhino"

description = "List of available software for rhino.unl.edu."

scripts = ["https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.min.js", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-pager.min.js","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/widgets/widget-filter.min.js","/js/sort-table.js"]

css = ["http://mottie.github.io/tablesorter/css/theme.default.css","https://mottie.github.io/tablesorter/css/theme.dropbox.css", "https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/jquery.tablesorter.pager.min.css","https://cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/css/filter.formatter.min.css"]

+++

{{% notice tip %}}

HCC provides some software packages via the Singularity container

software. If you do not see a desired package in the module list below,

please check the [Using Singularity]({{< relref "using_singularity" >}})

page for the software list there.

{{% /notice %}}

{{% panel theme="warning" header="Module prerequisites" %}}

If a module lists one or more prerequisites, the prerequisite module(s)

must be loaded before or along with, that module.

For example, the `cdo/2.1` modules requires `compiler/pgi/13.` To load

the cdo module, doing either

`module load compiler/pgi/13`

`module load cdo/2.1`

or

`module load compiler/pgi/13 cdo/2.1` (Note the prerequisite module

**must** be first.)

is acceptable.

{{% /panel %}}

{{% panel theme="info" header="Multiple versions" %}}

Some packages list multiple compilers for prerequisites. This means that

the package has been built with each version of the compilers listed.

{{% /panel %}}

{{% panel theme="warning" header="Custom GPU Anaconda Environment" %}}

If you are using custom GPU Anaconda Environment, the only module you need to load is `anaconda`:

`module load anaconda`

{{% /panel %}}

{{< table url="http://rhino-head.unl.edu:8192/lmod/spider/json" >}}

content/guides/running_applications/bioinformatics_tools/alignment_tools/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Alignment Tools"

description = "How to use various alignment tools on HCC machines"

weight = "52"

+++

{{% children %}}

 No newline at end of file

content/guides/running_applications/bioinformatics_tools/alignment_tools/blast/_index.mddeleted100644 → 0

+0
−16

Original line number
Original line
Diff line number
Diff line

+++

title = "BLAST"

description = "How to use BLAST on HCC machines"

weight = "52"

+++

[BLAST] (https://blast.ncbi.nlm.nih.gov/Blast.cgi) is a local alignment tool that finds similarity between sequences. This tool compares nucleotide or protein sequences to sequence databases, and calculates significance of matches. Sometimes these input sequences are large and using the command-line BLAST is required.

The following pages, [Create Local BLAST Database]({{<relref "create_local_blast_database" >}}) and [Running BLAST Alignment]({{<relref "running_blast_alignment" >}}) describe how to run some of the most common BLAST executables as a single job using the SLURM scheduler on HCC.

### Useful Information

In order to test the BLAST (blast/2.2) performance on Crane, we aligned three nucleotide query datasets, `small.fasta`, `medium.fasta` and `large.fasta`, against the non-redundant nucleotide **nt.fasta** database from NCBI. Some statistics about the query datasets and the time and memory resources used for the alignment are shown on the table below:

{{< readfile file="/static/html/blast.html" >}}

content/guides/running_applications/bioinformatics_tools/data_manipulation_tools/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Data Manipulation Tools"

description = "How to use data manipulation tools on HCC machines"

weight = "52"

+++

{{% children %}}

 No newline at end of file

content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "De Novo Assembly Tools"

description = "How to use de novo assembly tools on HCC machines"

weight = "52"

+++

{{% children %}}

content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/trinity/running_trinity_in_multiple_steps.mddeleted100644 → 0

+0
−93

Original line number
Original line
Diff line number
Diff line

+++

title = "Running Trinity in Multiple Steps"

description =  "How to run Trinity in multiple steps on HCC resources"

weight = "10"

+++

## Running Trinity with Paired-End fastq data with 8 CPUs and 100GB of RAM

The first step of running Trinity is to run Trinity with the option **--no_run_chrysalis**:

{{% panel header="`trinity_step1.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Trinity_Step1

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Trinity_Step1.%J.out

#SBATCH --error=Trinity_Step1.%J.err

module load trinity/2.6

Trinity --seqType fq --JM 100G --left input_reads_pair_1.fastq --right input_reads_pair_2.fastq --SS_lib_type FR --output trinity_out/ --CPU $SLURM_NTASKS_PER_NODE --no_run_chrysalis

{{< /highlight >}}

{{% /panel %}}

The second step of running Trinity is to run Trinity with the option **--no_run_quantifygraph**:

{{% panel header="`trinity_step2.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Trinity_Step2

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Trinity_Step2.%J.out

#SBATCH --error=Trinity_Step2.%J.err

module load trinity/2.6

Trinity --seqType fq --JM 100G --left input_reads_pair_1.fastq --right input_reads_pair_2.fastq --SS_lib_type FR --output trinity_out/ --CPU $SLURM_NTASKS_PER_NODE --no_run_quantifygraph

{{< /highlight >}}

{{% /panel %}}

The third step of running Trinity is to run Trinity with the option **--no_run_butterfly**:

{{% panel header="`trinity_step3.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Trinity_Step3

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Trinity_Step3.%J.out

#SBATCH --error=Trinity_Step3.%J.err

module load trinity/2.6

Trinity --seqType fq --JM 100G --left input_reads_pair_1.fastq --right input_reads_pair_2.fastq --SS_lib_type FR --output trinity_out/ --CPU $SLURM_NTASKS_PER_NODE --no_run_butterfly

{{< /highlight >}}

{{% /panel %}}

The fourth step of running Trinity is to run Trinity without any additional option:

{{% panel header="`trinity_step4.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Trinity_Step4

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Trinity_Step4.%J.out

#SBATCH --error=Trinity_Step4.%J.err

module load trinity/2.6

Trinity --seqType fq --JM 100G --left input_reads_pair_1.fastq --right input_reads_pair_2.fastq --SS_lib_type FR --output trinity_out/ --CPU $SLURM_NTASKS_PER_NODE

{{< /highlight >}}

{{% /panel %}}

### Trinity Output

Trinity outputs number of files in its `trinity_out/` output directory after each executed step. The output file `Trinity.fasta` is the final Trinity output that contains the assembled transcripts.

{{% notice tip %}}

The Inchworm (step 1) and Chrysalis (step 2) steps can be memory intensive. A basic recommendation is to have **1GB of RAM per 1M ~76 base Illumina paired-end reads**.

{{% /notice %}}

content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/velvet/_index.mddeleted100644 → 0

+0
−22

Original line number
Original line
Diff line number
Diff line

+++

title = "Velvet"

description = "How to use Velvet on HCC machines"

weight = "52"

+++

[Velvet] (https://www.ebi.ac.uk/~zerbino/velvet/) is a general sequence assembler designed to produce assembly from short, as well as long reads. Running Velvet consists of a sequence of two commands **velveth** and **velvetg**. **velveth** produces a hash table of k-mers, while **velvetg** constructs the genome assembly. The k-mer length, also known as hash length corresponds to the length, in base pairs, of the words of the reads being hashed.

Velvet has lots of parameters that can be found in its [manual] (https://www.ebi.ac.uk/~zerbino/velvet/Manual.pdf). However, the k-mer value is crucial in obtaining optimal assemblies. Higher k-mer values increase the specificity, and lower k-mer values increase the sensitivity.

Velvet supports multiple file formats: `fasta`, `fastq`, `fasta.gz`, `fastq.gz`, `sam`, `bam`, `eland`, `gerald`. Velvet also supports different read categories for different sequencing technologies and libraries, e.g. `short`, `shortPaired`, `short2`, `shortPaired2`, `long`, `longPaired`.

Each step of Velvet (**velveth** and **velvetg**) may be run as its own job. The following pages describe how to run Velvet in this manner on HCC and provide example submit scripts:

{{% children %}}

### Useful Information

In order to test the Velvet (velvet/1.2) performance on Tusker, we used three paired-end input fastq files, `small_1.fastq` and `small_2.fastq`, `medium_1.fastq` and `medium_2.fastq`, and `large_1.fastq` and `large_2.fastq`. Some statistics about the input files and the time and memory resources used by Velvet on Tusker are shown in the table below:

{{< readfile file="/static/html/velvet.html" >}}

content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/velvet/running_velvet_with_paired_end_data.mddeleted100644 → 0

+0
−68

Original line number
Original line
Diff line number
Diff line

+++

title = "Running Velvet with Paired-End Data"

description =  "How to run velvet with paired-end data on HCC resources"

weight = "10"

+++

## Running Velvet with Paired-End long fastq data with k-mer=43, 8 CPUs and 100GB of RAM

The first step of running Velvet is to run **velveth**:

{{% panel header="`velveth.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Velvet_Velveth

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=10gb

#SBATCH --output=Velveth.%J.out

#SBATCH --error=Velveth.%J.err

module load velvet/1.2

export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE

velveth output_directory/ 43 -fastq -longPaired -separate input_reads_pair_1.fastq input_reads_pair_2.fastq

{{< /highlight >}}

{{% /panel %}}

After running **velveth**, the next step is to run **velvetg** on the `output_directory/` and files generated from **velveth**:

{{% panel header="`velvetg.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Velvet_Velvetg

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Velvetg.%J.out

#SBATCH --error=Velvetg.%J.err

module load velvet/1.2

export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE

velvetg output_directory/ -min_contig_lgth 200

{{< /highlight >}}

{{% /panel %}}

Both **velveth** and **velvetg** are multi-threaded.

### Velvet Output

{{% panel header="`Output directory after velveth`"%}}

{{< highlight bash >}}

$ ls output_directory/

Log  Roadmaps  Sequences

{{< /highlight >}}

{{% /panel %}}

{{% panel header="`Output directory after velvetg`"%}}

{{< highlight bash >}}

$ ls output_directory/

contigs.fa  Graph  LastGraph  Log  PreGraph  Roadmaps  Sequences  stats.txt

{{< /highlight >}}

{{% /panel %}}

The output fasta file `contigs.fa` is the final Velvet output that contains the assembled contigs. More information about the output files is provided in the Velvet manual.

content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/velvet/running_velvet_with_single_end_and_paired_end_data.mddeleted100644 → 0

+0
−69

Original line number
Original line
Diff line number
Diff line

+++

title = "Running Velvet with Single-End and Paired-End Data"

description =  "How to run velvet with single-end and paired-end data on HCC resources"

weight = "10"

+++

## Running Velvet with Single-End and Paired-End short fasta data with k-mer=51, 8 CPUs and 100GB of RAM

The first step of running Velvet is to run **velveth**:

{{% panel header="`velveth.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Velvet_Velveth

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=10gb

#SBATCH --output=Velveth.%J.out

#SBATCH --error=Velveth.%J.err

module load velvet/1.2

export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE

velveth output_directory/ 51 -fasta -short input_reads.fasta -fasta -shortPaired2 -separate input_reads_pair_1.fasta input_reads_pair_2.fasta

{{< /highlight >}}

{{% /panel %}}

After running **velveth**, the next step is to run **velvetg** on the `output_directory/` and files generated from **velveth**:

{{% panel header="`velvetg.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Velvet_Velvetg

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Velvetg.%J.out

#SBATCH --error=Velvetg.%J.err

module load velvet/1.2

export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE

velvetg output_directory/ -min_contig_lgth 200

{{< /highlight >}}

{{% /panel %}}

Both **velveth** and **velvetg** are multi-threaded.

### Velvet Output

{{% panel header="`Output directory after velveth`"%}}

{{< highlight bash >}}

$ ls output_directory/

Log  Roadmaps  Sequences

{{< /highlight >}}

{{% /panel %}}

{{% panel header="`Output directory after velvetg`"%}}

{{< highlight bash >}}

$ ls output_directory/

contigs.fa  Graph  LastGraph  Log  PreGraph  Roadmaps  Sequences  stats.txt

{{< /highlight >}}

{{% /panel %}}

The output fasta file `contigs.fa` is the final Velvet output that contains the assembled contigs. More information about the output files is provided in the Velvet manual.

content/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/velvet/running_velvet_with_single_end_data.mddeleted100644 → 0

+0
−68

Original line number
Original line
Diff line number
Diff line

+++

title = "Running Velvet with Single-End Data"

description =  "How to run velvet with single-end data on HCC resources"

weight = "10"

+++

## Running Velvet with Single-End short fasta data with k-mer=31, 8 CPUs and 100GB of RAM

The first step of running Velvet is to run **velveth**:

{{% panel header="`velveth.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Velvet_Velveth

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=10gb

#SBATCH --output=Velveth.%J.out

#SBATCH --error=Velveth.%J.err

module load velvet/1.2

export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE

velveth output_directory/ 31 -fasta -short input_reads.fasta

{{< /highlight >}}

{{% /panel %}}

After running **velveth**, the next step is to run **velvetg** on the `output_directory/` and files generated from **velveth**:

{{% panel header="`velvetg.submit`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --job-name=Velvet_Velvetg

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=8

#SBATCH --time=168:00:00

#SBATCH --mem=100gb

#SBATCH --output=Velvetg.%J.out

#SBATCH --error=Velvetg.%J.err

module load velvet/1.2

export OMP_NUM_THREADS=$SLURM_NTASKS_PER_NODE

velvetg output_directory/ -min_contig_lgth 200

{{< /highlight >}}

{{% /panel %}}

Both **velveth** and **velvetg** are multi-threaded.

### Velvet Output

{{% panel header="`Output directory after velveth`"%}}

{{< highlight bash >}}

$ ls output_directory/

Log  Roadmaps  Sequences

{{< /highlight >}}

{{% /panel %}}

{{% panel header="`Output directory after velvetg`"%}}

{{< highlight bash >}}

$ ls output_directory/

contigs.fa  Graph  LastGraph  Log  PreGraph  Roadmaps  Sequences  stats.txt

{{< /highlight >}}

{{% /panel %}}

The output fasta file `contigs.fa` is the final Velvet output that contains the assembled contigs. More information about the output files is provided in the Velvet manual.

content/guides/running_applications/bioinformatics_tools/downloading_sra_data_from_ncbi.mddeleted100644 → 0

+0
−43

Original line number
Original line
Diff line number
Diff line

+++

title = "Downloading SRA data from NCBI"

description = "How to download data from NCBI"

weight = "52"

+++

One way to download high-volume data from NCBI is to use command line

utilities, such as **wget**, **ftp** or Aspera Connect **ascp**

plugin. The Aspera Connect plugin is commonly used high-performance transfer

plugin that provides the best transfer speed.

This plugin is available on our clusters as a module. In order to use it, load the appropriate module first:

{{< highlight bash >}}

$ module load aspera-cli

{{< /highlight >}}

The basic usage of the Aspera plugin is

{{< highlight bash >}}

$ ascp -i $ASPERA_PUBLIC_KEY -k 1 -T -l <max_download_rate_in_Mbps>m anonftp@ftp.ncbi.nlm.nih.gov:/<files_to_transfer> <local_work_output_directory>

{{< /highlight >}}

where **-k 1** enables resume of partial transfers, **-T** disables encryption for maximum throughput, and **-l** sets the transfer rate.

**\<files_to_transfer\>** mentioned in the basic usage of Aspera

plugin has a specifically defined pattern that needs to be followed:

{{< highlight bash >}}

<files_to_transfer> = /sra/sra-instant/reads/ByRun/sra/SRR|ERR|DRR/<first_6_characters_of_accession>/<accession>/<accession>.sra

{{< /highlight >}}

where **SRR\|ERR\|DRR** should be either **SRR**, **ERR **or **DRR** and should match the prefix of the target **.sra** file.

More **ascp** options can be seen by using:

{{< highlight bash >}}

$ ascp --help

{{< /highlight >}}

For example, if you want to download the **SRR304976** file from NCBI in your $WORK **data/** directory with downloading speed of **1000 Mbps**, you should use the following command:

{{< highlight bash >}}

$ ascp -i $ASPERA_PUBLIC_KEY -k 1 -T -l 1000m anonftp@ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR304/SRR304976/SRR304976.sra /work/[groupname]/[username]/data/

{{< /highlight >}}

content/guides/running_applications/bioinformatics_tools/pre_processing_tools/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Pre-processing Tools"

description = "How to use pre-processing tools on HCC machines"

weight = "52"

+++

{{% children %}}

 No newline at end of file

content/guides/running_applications/bioinformatics_tools/reference_based_assembly_tools/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Reference-Based Assembly Tools"

description = "How to use reference based assembly tools on HCC machines"

weight = "52"

+++

{{% children %}}

 No newline at end of file

content/guides/running_applications/bioinformatics_tools/removing_detecting_redundant_sequences/_index.mddeleted100644 → 0

+0
−7

Original line number
Original line
Diff line number
Diff line

+++

title = "Tools for Removing/Detecting Redundant Sequences"

description = "How to use tools for removing/detecting redundant sequences on HCC machines"

weight = "52"

+++

{{% children %}}

content/guides/running_applications/module_commands.mddeleted100644 → 0

+0
−235

Original line number
Original line
Diff line number
Diff line

+++

title = "Module Commands"

description = "How to use the module utility on HCC resources."

+++

`module` commands provide an HPC system user the capability to compile

into their source code using any type of library that is

available on the server. The `module` command gives each user the

ability to modify their environmental `PATH` and `LD_LIBRARY_PATH`

variables.

{{% notice info %}}

Please note that if you compile your application using a particular

module, you must include the appropriate module load statement in your

submit script.

{{% /notice %}}

### List Modules Loaded

{{% panel theme="info" header="Example Usage: module list" %}}

{{< highlight bash >}}

module list

No Modulefiles Currently Loaded.

echo $PATH

/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin

{{< /highlight >}}

{{% /panel %}}

### List Modules Available

{{% panel theme="info" header="Example Usage: Listing Available Modules" %}}

{{< highlight bash >}}

module avail

---------------------------------------------- /util/opt/Modules/modulefiles ----------------------------------------------

NCL/6.0             bowtie/2.0.0-beta6  compiler/pgi/12     hdfeos5/1.14        mplus/7.0           szip/2.1            

NCL/6.0dist         compiler/gcc/4.6    cufflinks/2.0.2     hugeseq/1.0         netcdf/4.1          tophat/2.0.5        

NCO/4.1             compiler/gcc/4.7    deprecated          intel-mkl/11        netcdf/4.2          udunits/2.1         

R/2.15              compiler/intel/11   hdf4/4.2            intel-mkl/12        openmpi/1.5         zlib/1.2            

WRF/WRF             compiler/intel/12   hdf5/1.8            lsdyna/5.1.1        openmpi/1.6         

acml/5.1            compiler/open64/4.5 hdf5/1.8.6          lsdyna/6.0.0        samtools/0.1        

bowtie/0.12.8       compiler/pgi/11     hdfeos2/2.18        mplus/6.12          sas/9.3    

{{< /highlight >}}

{{% /panel %}}

#### module load \<module-name\>

Places the binaries and libraries for \<module-name\> into your `PATH` and `LD_LIBRARY_PATH`.

{{% panel theme="info" header="Example Usage: Loading Desired Module" %}}

{{< highlight bash >}} 

module load compiler/pgi/11

module list

Currently Loaded Modulefiles:

 1) compiler/pgi/11 

echo $PATH

/util/comp/pgi/linux86-64/11/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin

{{< /highlight >}}

{{% /panel %}}

#### module unload \<module-name\>

Removes the binaries and libraries associated with \<module-name\> from your PATH and `LD_LIBRARY_PATH`.

{{% panel theme="info" header="Example Usage: module unload" %}}

{{< highlight bash >}}

module unload compiler/pgi/11

module list

No Modulefiles Currently Loaded.

echo $PATH

/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin 

{{< /highlight >}}

{{% /panel %}} 

#### module purge

**Purges** all previously **loaded** module libraries and binaries from

your `PATH` and `LD_LIBRARY_PATH`.

{{% panel theme="info" header="Example Usage: module purge" %}}

{{< highlight bash >}}

module load compiler/open64

module load zlib/1.2

module list

Currently Loaded Modulefiles:

 1) zlib/1.2             2) compiler/open64/4.5 

module purge

module list

No Modulefiles Currently Loaded.

{{< /highlight >}}

{{% /panel %}}

#### module help

To see a complete list of module commands/options.

**Example Usage: module help**

{{% panel theme="info" header="Example Usage: module help" %}}

{{< highlight bash >}}

Usage: module [options] sub-command [args ...]

Options:

  -h -? -H --help                   This help message

  -s availStyle --style=availStyle  Site controlled avail style: system en_grouped (default: en_grouped)

  --regression_testing              Lmod regression testing

  -D                                Program tracing written to stderr

  --debug=dbglvl                    Program tracing written to stderr

  --pin_versions=pinVersions        When doing a restore use specified version, do not follow defaults

  -d --default                      List default modules only when used with avail

  -q --quiet                        Do not print out warnings

  --expert                          Expert mode

  -t --terse                        Write out in machine readable format for commands: list, avail, spider, savelist

  --initial_load                    loading Lmod for first time in a user shell

  --latest                          Load latest (ignore default)

  --ignore_cache                    Treat the cache file(s) as out-of-date

  --novice                          Turn off expert and quiet flag

  --raw                             Print modulefile in raw output when used with show

  -w twidth --width=twidth          Use this as max term width

  -v --version                      Print version info and quit

  -r --regexp                       use regular expression match

  --gitversion                      Dump git version in a machine readable way and quit

  --dumpversion                     Dump version in a machine readable way and quit

  --check_syntax --checkSyntax      Checking module command syntax: do not load

  --config                          Report Lmod Configuration

  --config_json                     Report Lmod Configuration in json format

  --mt                              Report Module Table State

  --timer                           report run times

  --force                           force removal of a sticky module or save an empty collection

  --redirect                        Send the output of list, avail, spider to stdout (not stderr)

  --no_redirect                     Force output of list, avail and spider to stderr

  --show_hidden                     Avail and spider will report hidden modules

  --spider_timeout=timeout          a timeout for spider

  -T --trace

module [options] sub-command [args ...]

Help sub-commands:

------------------

  help                              prints this message

  help                module [...]  print help message from module(s)

Loading/Unloading sub-commands:

-------------------------------

  load | add          module [...]  load module(s)

  try-load | try-add  module [...]  Add module(s), do not complain if not found

  del | unload        module [...]  Remove module(s), do not complain if not found

  swap | sw | switch  m1 m2         unload m1 and load m2

  purge                             unload all modules

  refresh                           reload aliases from current list of modules.

  update                            reload all currently loaded modules.

Listing / Searching sub-commands:

---------------------------------

  list                              List loaded modules

  list                s1 s2 ...     List loaded modules that match the pattern

  avail | av                        List available modules

  avail | av          string        List available modules that contain "string".

  spider                            List all possible modules

  spider              module        List all possible version of that module file

  spider              string        List all module that contain the "string".

  spider              name/version  Detailed information about that version of the module.

  whatis              module        Print whatis information about module

  keyword | key       string        Search all name and whatis that contain "string".

Searching with Lmod:

--------------------

  All searching (spider, list, avail, keyword) support regular expressions:

  spider -r           '^p'          Finds all the modules that start with `p' or `P'

  spider -r           mpi           Finds all modules that have "mpi" in their name.

  spider -r           'mpi$         Finds all modules that end with "mpi" in their name.

Handling a collection of modules:

--------------------------------

  save | s                          Save the current list of modules to a user defined "default" collection.

  save | s            name          Save the current list of modules to "name" collection.

  reset                             The same as "restore system"

  restore | r                       Restore modules from the user's "default" or system default.

  restore | r         name          Restore modules from "name" collection.

  restore             system        Restore module state to system defaults.

  savelist                          List of saved collections.

  describe | mcc      name          Describe the contents of a module collection.

Deprecated commands:

--------------------

  getdefault          [name]        load name collection of modules or user's "default" if no name given.

                                    ===> Use "restore" instead  <====

  setdefault          [name]        Save current list of modules to name if given, otherwise save as the default list for you the user.

                                    ===> Use "save" instead. <====

Miscellaneous sub-commands:

---------------------------

  show                modulefile    show the commands in the module file.

  use [-a]            path          Prepend or Append path to MODULEPATH.

  unuse               path          remove path from MODULEPATH.

  tablelist                         output list of active modules as a lua table.

Important Environment Variables:

--------------------------------

  LMOD_COLORIZE                     If defined to be "YES" then Lmod prints properties and warning in color.

    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Lmod Web Sites

  Documentation:    http://lmod.readthedocs.org

  Github:           https://github.com/TACC/Lmod

  Sourceforge:      https://lmod.sf.net

  TACC Homepage:    https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

  To report a bug please read http://lmod.readthedocs.io/en/latest/075_bug_reporting.html

    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Modules based on Lua: Version 7.4.16  2017-05-23 11:10 -05:00

    by Robert McLay mclay@tacc.utexas.edu

{{< /highlight >}}

{{% /panel %}}

content/guides/running_applications/mpi_jobs_on_hcc.mddeleted100644 → 0

+0
−322

Original line number
Original line
Diff line number
Diff line

+++

title = "MPI Jobs on HCC"

description = "How to compile and run MPI programs on HCC machines"

weight = "52"

+++

This quick start demonstrates how to implement a parallel (MPI)

Fortran/C program on HCC supercomputers. The sample codes and submit

scripts can be downloaded from [mpi_dir.zip](/attachments/mpi_dir.zip).

#### Login to a HCC Cluster

Log in to a HCC cluster through PuTTY ([For Windows Users]({{< relref "/quickstarts/connecting/for_windows_users">}})) or Terminal ([For Mac/Linux

Users]({{< relref "/quickstarts/connecting/for_maclinux_users">}})) and make a subdirectory called `mpi_dir` under the `$WORK` directory.

{{< highlight bash >}}

$ cd $WORK

$ mkdir mpi_dir

{{< /highlight >}}

In the subdirectory `mpi_dir`, save all the relevant codes. Here we

include two demo programs, `demo_f_mpi.f90` and `demo_c_mpi.c`, that

compute the sum from 1 to 20 through parallel processes. A

straightforward parallelization scheme is used for demonstration

purpose. First, the master core (i.e. `myid=0`) distributes equal

computation workload to a certain number of cores (as specified by

`--ntasks `in the submit script). Then, each worker core computes a

partial summation as output. Finally, the master core collects the

outputs from all worker cores and perform an overall summation. For easy

comparison with the serial code ([Fortran/C on HCC]({{< relref "fortran_c_on_hcc">}})), the

added lines in the parallel code (MPI) are marked with "!=" or "//=".

{{%expand "demo_f_mpi.f90" %}}

{{< highlight fortran >}}

Program demo_f_mpi

!====== MPI =====

    use mpi     

!================

    implicit none

    integer, parameter :: N = 20

    real*8 w

    integer i

    common/sol/ x

    real*8 x

    real*8, dimension(N) :: y 

!============================== MPI =================================

    integer ind

    real*8, dimension(:), allocatable :: y_local                    

    integer numnodes,myid,rc,ierr,start_local,end_local,N_local     

    real*8 allsum                                                   

!====================================================================

!============================== MPI =================================

    call mpi_init( ierr )                                           

    call mpi_comm_rank ( mpi_comm_world, myid, ierr )               

    call mpi_comm_size ( mpi_comm_world, numnodes, ierr )           

                                                                                                                                        !

    N_local = N/numnodes                                            

    allocate ( y_local(N_local) )                                   

    start_local = N_local*myid + 1                                  

    end_local =  N_local*myid + N_local                             

!====================================================================

    do i = start_local, end_local

        w = i*1d0

        call proc(w)

        ind = i - N_local*myid

        y_local(ind) = x

!       y(i) = x

!       write(6,*) 'i, y(i)', i, y(i)

    enddo   

!       write(6,*) 'sum(y) =',sum(y)

!============================================== MPI =====================================================

    call mpi_reduce( sum(y_local), allsum, 1, mpi_real8, mpi_sum, 0, mpi_comm_world, ierr )             

    call mpi_gather ( y_local, N_local, mpi_real8, y, N_local, mpi_real8, 0, mpi_comm_world, ierr )     

    if (myid == 0) then                                                                                 

        write(6,*) '-----------------------------------------'                                          

        write(6,*) '*Final output from... myid=', myid                                                  

        write(6,*) 'numnodes =', numnodes                                                               

        write(6,*) 'mpi_sum =', allsum  

        write(6,*) 'y=...'

        do i = 1, N

            write(6,*) y(i)

        enddo                                                                                       

        write(6,*) 'sum(y)=', sum(y)                                                                

    endif                                                                                               

    deallocate( y_local )                                                                               

    call mpi_finalize(rc)                                                                               

!========================================================================================================

Stop

End Program

Subroutine proc(w)

    real*8, intent(in) :: w

    common/sol/ x

    real*8 x

    x = w

Return

End Subroutine

{{< /highlight >}}

{{% /expand %}}

{{%expand "demo_c_mpi.c" %}}

{{< highlight c >}}

//demo_c_mpi

#include <stdio.h>

//======= MPI ========

#include "mpi.h"    

#include <stdlib.h>   

//====================

double proc(double w){

        double x;       

        x = w;  

        return x;

}

int main(int argc, char* argv[]){

    int N=20;

    double w;

    int i;

    double x;

    double y[N];

    double sum;

//=============================== MPI ============================

    int ind;                                                    

    double *y_local;                                            

    int numnodes,myid,rc,ierr,start_local,end_local,N_local;    

    double allsum;                                              

//================================================================

//=============================== MPI ============================

    MPI_Init(&argc, &argv);

    MPI_Comm_rank( MPI_COMM_WORLD, &myid );

    MPI_Comm_size ( MPI_COMM_WORLD, &numnodes );

    N_local = N/numnodes;

    y_local=(double *) malloc(N_local*sizeof(double));

    start_local = N_local*myid + 1;

    end_local = N_local*myid + N_local;

//================================================================

    for (i = start_local; i <= end_local; i++){        

        w = i*1e0;

        x = proc(w);

        ind = i - N_local*myid;

        y_local[ind-1] = x;

//      y[i-1] = x;

//      printf("i,x= %d %lf\n", i, y[i-1]) ;

    }

    sum = 0e0;

    for (i = 1; i<= N_local; i++){

        sum = sum + y_local[i-1];   

    }

//  printf("sum(y)= %lf\n", sum);    

//====================================== MPI ===========================================

    MPI_Reduce( &sum, &allsum, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD );

    MPI_Gather( &y_local[0], N_local, MPI_DOUBLE, &y[0], N_local, MPI_DOUBLE, 0, MPI_COMM_WORLD );

    if (myid == 0){

    printf("-----------------------------------\n");

    printf("*Final output from... myid= %d\n", myid);

    printf("numnodes = %d\n", numnodes);

    printf("mpi_sum = %lf\n", allsum);

    printf("y=...\n");

    for (i = 1; i <= N; i++){

        printf("%lf\n", y[i-1]);

    }   

    sum = 0e0;

    for (i = 1; i<= N; i++){

        sum = sum + y[i-1]; 

    }

    printf("sum(y) = %lf\n", sum);

    }

    free( y_local );

    MPI_Finalize ();

//======================================================================================        

return 0;

}

{{< /highlight >}}

{{% /expand %}}

---

#### Compiling the Code

The compiling of a MPI code requires first loading a compiler "engine"

such as `gcc`, `intel`, or `pgi` and then loading a MPI wrapper

`openmpi`. Here we will use the GNU Complier Collection, `gcc`, for

demonstration.

{{< highlight bash >}}

$ module load compiler/gcc/6.1 openmpi/2.1

$ mpif90 demo_f_mpi.f90 -o demo_f_mpi.x  

$ mpicc demo_c_mpi.c -o demo_c_mpi.x

{{< /highlight >}}

The above commends load the `gcc` complier with the `openmpi` wrapper.

The compiling commands `mpif90` or `mpicc` are used to compile the codes

to`.x` files (executables). 

### Creating a Submit Script

Create a submit script to request 5 cores (with `--ntasks`). A parallel

execution command `mpirun ./` needs to enter to last line before the

main program name.

{{% panel header="`submit_f.mpi`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --ntasks=5

#SBATCH --mem-per-cpu=1024

#SBATCH --time=00:01:00

#SBATCH --job-name=Fortran

#SBATCH --error=Fortran.%J.err

#SBATCH --output=Fortran.%J.out

mpirun ./demo_f_mpi.x 

{{< /highlight >}}

{{% /panel %}}

{{% panel header="`submit_c.mpi`"%}}

{{< highlight bash >}}

#!/bin/sh

#SBATCH --ntasks=5

#SBATCH --mem-per-cpu=1024

#SBATCH --time=00:01:00

#SBATCH --job-name=C

#SBATCH --error=C.%J.err

#SBATCH --output=C.%J.out

mpirun ./demo_c_mpi.x 

{{< /highlight >}}

{{% /panel %}}

#### Submit the Job

The job can be submitted through the command `sbatch`. The job status

can be monitored by entering `squeue` with the `-u` option.

{{< highlight bash >}}

$ sbatch submit_f.mpi

$ sbatch submit_c.mpi

$ squeue -u <username>

{{< /highlight >}}

Replace `<username>` with your HCC username.

Sample Output

-------------

The sum from 1 to 20 is computed and printed to the `.out` file (see

below). The outputs from the 5 cores are collected and processed by the

master core (i.e. `myid=0`).

{{%expand "Fortran.out" %}}

{{< highlight batchfile>}}

 -----------------------------------------

 *Final output from... myid=           0

 numnodes =           5

 mpi_sum =   210.00000000000000     

 y=...

   1.0000000000000000     

   2.0000000000000000     

   3.0000000000000000     

   4.0000000000000000     

   5.0000000000000000     

   6.0000000000000000     

   7.0000000000000000     

   8.0000000000000000     

   9.0000000000000000     

   10.000000000000000     

   11.000000000000000     

   12.000000000000000     

   13.000000000000000     

   14.000000000000000     

   15.000000000000000     

   16.000000000000000     

   17.000000000000000     

   18.000000000000000     

   19.000000000000000     

   20.000000000000000     

 sum(y)=   210.00000000000000     

{{< /highlight >}}

{{% /expand %}} 

{{%expand "C.out" %}}

{{< highlight batchfile>}}

-----------------------------------

*Final output from... myid= 0

numnodes = 5

mpi_sum = 210.000000

y=...

1.000000

2.000000

3.000000

4.000000

5.000000

6.000000

7.000000

8.000000

9.000000

10.000000

11.000000

12.000000

13.000000

14.000000

15.000000

16.000000

17.000000

18.000000

19.000000

20.000000

sum(y) = 210.000000

{{< /highlight >}}

{{% /expand %}}

content/guides/running_applications/running_ocean_land_atmosphere_model_olam.mddeleted100644 → 0

+0
−237

Original line number
Original line
Diff line number
Diff line

+++

title = "Running OLAM at HCC"

description = "How to run the OLAM (Ocean Land Atmosphere Model) on HCC resources."

+++

### OLAM compilation on Tusker

##### pgi/11 compilation with mpi and openmp enabled

1.  Load modules:

    {{< highlight bash >}}

    module load compiler/pgi/11 openmpi/1.6 szip/2.1 zlib/1.2 NCL/6.1dist

    {{< /highlight >}}

2.  Edit the `include.mk` file.

{{% panel theme="info" header="include.mk" %}}

{{< highlight batch >}}

#-----------------  LINUX Intel Fortran ifort/gcc ---------------

F_COMP=mpif90

# If the compiler supports (and the user wants to use)

# the module IEEE_ARITHMETIC, uncomment below

IEEE_ARITHMETIC=yes

# If using MPI libraries:

OLAM_MPI=yes

# If parallel hdf5 is supported, uncomment the next line

OLAM_PARALLEL_HDF5=yes

# If you use the ED2 model, uncomment the next line

#USE_ED2=yes

MPI_PATH=/util/opt/openmpi/1.6/pgi/11

PAR_INCS=-I$(MPI_PATH)/include:$(MPI_PATH)/lib

PAR_LIBS=-L$(MPI_PATH)/lib -lmpi

# OPTIMIZED:

F_OPTS=-O3 -traceback -mp

#F_OPTS=-xHost -O3 -fno-alias -ip -openmp -traceback

#F_OPTS=-g -O3 -xHost -traceback

# DEBUG:

#F_OPTS=-g -fp-model precise -check bounds -traceback \

#        -debug extended -check uninit -ftrapuv

# FORTRAN FLAGS FOR BIG FILES WHICH WOULD HAVE EXCESSIVE COMPILATION TIME

#SLOW_FFLAGS=-O1 -g -no-ip -traceback

C_COMP=mpicc

#C_COMP=mpicc

C_OPTS=-DUNDERSCORE -DLITTLE

NCARG_DIR=/util/src/ncl_ncarg/ncl_ncarg-6.1.2/lib

LIBNCARG=-L$(NCARG_DIR) -lncarg -lncarg_gks -lncarg_c \

          -L/usr/lib64 -lX11 -ldl -lpthread -lgfortran -lcairo

HDF5_LIBS=-L/util/opt/hdf5/1.8.13/openmpi/1.6/pgi/11/lib -lhdf5_fortran -lhdf5 -lz -lm

HDF5_INCS=-I/util/opt/hdf5/1.8.13/openmpi/1.6/pgi/11/include

NETCDF_LIBS=-L/util/opt/netcdf/4.2/pgi/11/lib -lnetcdf

NETCDF_INCS=-I/util/opt/netcdf/4.2/pgi/11/include

LOADER=$(F_COMP)

LOADER_OPTS=-mp

#LOADER_OPTS=-static-intel $(F_OPTS)

# For Apple OSX: the stack size needs to be increased at link time

# LOADER_OPTS=-static-intel $(F_OPTS) -Wl,-stack_size -Wl,0x10000000

# to allow ifort compiler to link with pg-compiled ncar graphics:

# LIBS=-z muldefs -L/opt/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc

## IMPORTANT:  Need to specify this flag in ED2

#USE_HDF5=1

{{< /highlight >}}

{{% /panel %}}

3. Command: `make clean`

4. Command: `make -j 8`

##### intel/12 compilation with mpi and openmp enabled

1.  Load modules:

   {{< highlight bash >}}

    module load compiler/intel/12 openmpi/1.6 szip/2.1 zlib/1.2

    {{< /highlight >}}

2.  Edit the `include.mk` file.

{{% panel theme="info" header="include.mk" %}}

{{< highlight batch >}}

#-----------------  LINUX Intel Fortran ifort/gcc ---------------

F_COMP=mpif90

# If the compiler supports (and the user wants to use)

# the module IEEE_ARITHMETIC, uncomment below

IEEE_ARITHMETIC=yes

# If using MPI libraries:

OLAM_MPI=yes

# If parallel hdf5 is supported, uncomment the next line

OLAM_PARALLEL_HDF5=yes

# If you use the ED2 model, uncomment the next line

#USE_ED2=yes

MPI_PATH=/util/opt/openmpi/1.6/intel/12

PAR_INCS=-I$(MPI_PATH)/include:$(MPI_PATH)/lib

PAR_LIBS=-L$(MPI_PATH)/lib -lmpi

# OPTIMIZED:

F_OPTS=-O3 -traceback -openmp

#F_OPTS=-xHost -O3 -fno-alias -ip -openmp -traceback

#F_OPTS=-g -O3 -xHost -traceback

# DEBUG:

#F_OPTS=-g -fp-model precise -check bounds -traceback \

#        -debug extended -check uninit -ftrapuv

# FORTRAN FLAGS FOR BIG FILES WHICH WOULD HAVE EXCESSIVE COMPILATION TIME

#SLOW_FFLAGS=-O1 -g -no-ip -traceback

C_COMP=mpicc

#C_COMP=mpicc

C_OPTS=-DUNDERSCORE -DLITTLE

NCARG_DIR=/util/src/ncl_ncarg/ncl_ncarg-6.1.2/lib

LIBNCARG=-L$(NCARG_DIR) -lncarg -lncarg_gks -lncarg_c \

          -L/usr/lib64 -lX11 -ldl -lpthread -lgfortran -lcairo

HDF5_LIBS=-L/util/opt/hdf5/1.8.13/openmpi/1.6/intel/12/lib -lhdf5_fortran -lhdf5 -lz -lm

HDF5_INCS=-I/util/opt/hdf5/1.8.13/openmpi/1.6/intel/12/include

NETCDF_LIBS=-L/util/opt/netcdf/4.2/intel/12/lib -lnetcdf

NETCDF_INCS=-I/util/opt/netcdf/4.2/intel/12/include

LOADER=$(F_COMP)

LOADER_OPTS=-openmp

#LOADER_OPTS=-static-intel $(F_OPTS)

# For Apple OSX: the stack size needs to be increased at link time

# LOADER_OPTS=-static-intel $(F_OPTS) -Wl,-stack_size -Wl,0x10000000

# to allow ifort compiler to link with pg-compiled ncar graphics:

# LIBS=-z muldefs -L/opt/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc

## IMPORTANT:  Need to specify this flag in ED2

#USE_HDF5=1

{{< /highlight >}}

{{% /panel %}}

3. Command: `make clean`

4. Command: `make -j 8`

### OLAM compilation on Crane

##### Intel/15 compiler with OpenMPI/1.10

1.  Load modules:

    {{< highlight bash >}}

    module load compiler/intel/15 openmpi/1.10 NCL/6.1 netcdf/4.4 phdf5/1.8 szip/2.1 zlib/1.2

    {{< /highlight >}}

2.  Edit the `include.mk` file:

{{% panel theme="info" header="include.mk" %}}

{{< highlight batch >}}

#-----------------  LINUX Intel Fortran ifort/gcc ---------------

F_COMP=/util/opt/hdf5/1.8/openmpi/1.10/intel/15/bin/h5pfc

# If the compiler supports (and the user wants to use)

# the module IEEE_ARITHMETIC, uncomment below

IEEE_ARITHMETIC=yes

# If using MPI libraries:

OLAM_MPI=yes

# If parallel hdf5 is supported, uncomment the next line

OLAM_PARALLEL_HDF5=yes

# If you use the ED2 model, uncomment the next line

#USE_ED2=yes

#MPI_PATH=/usr/local/mpich

PAR_INCS=-I/util/opt/openmpi/1.10/intel/15/include

PAR_LIBS=-L/util/opt/openmpi/1.10/intel/15/lib

# OPTIMIZED:

F_OPTS=-xHost -O3 -fno-alias -ip -openmp -traceback

#F_OPTS=-g -O3 -xHost -traceback

# DEBUG:

#F_OPTS=-g -fp-model precise -check bounds -traceback \

#        -debug extended -check uninit -ftrapuv

# EXTRA OPTIONS FOR FIXED-SOURCE CODE

FIXED_SRC_FLAGS=-fixed -132

# FORTRAN FLAGS FOR BIG FILES WHICH WOULD HAVE EXCESSIVE COMPILATION TIME

SLOW_FFLAGS=-O1 -g -no-ip -traceback

#C_COMP=icc

C_COMP=mpicc

C_OPTS=-O3 -DUNDERSCORE -DLITTLE

NCARG_DIR=/util/opt/NCL/6.1/lib

LIBNCARG=-L$(NCARG_DIR) -lncarg -lncarg_gks -lncarg_c \

-L/usr/lib64 -lX11 -ldl -lpng -lpthread -lgfortran -lcairo

HDF5_LIBS=-L/util/opt/hdf5/1.8/openmpi/1.10/intel/15/lib

HDF5_INCS=-I/util/opt/hdf5/1.8/openmpi/1.10/intel/15/include

NETCDF_LIBS=-L/util/opt/netcdf/4.4/intel/15/lib -lnetcdf

NETCDF_INCS=-I/util/opt/netcdf/4.4/intel/15/include

LOADER=$(F_COMP)

LOADER_OPTS=-static-intel $(F_OPTS)

# For Apple OSX: the stack size needs to be increased at link time

# LOADER_OPTS=-static-intel $(F_OPTS) -Wl,-stack_size -Wl,0x10000000

# to allow ifort compiler to link with pg-compiled ncar graphics:

# LIBS=-z muldefs -L/opt/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc

## IMPORTANT:  Need to specify this flag in ED2

USE_HDF5=1

{{< /highlight >}}

{{% /panel %}}

3.  Command: `make clean`

4.  Command: `make -j 8`

### Sample SLURM submit scripts

##### PGI compiler:

{{% panel theme="info" header="Sample submit script for PGI compiler" %}}

{{< highlight batch >}}

#!/bin/sh

#SBATCH --ntasks=8                                          # 8 cores

#SBATCH --mem-per-cpu=1024                                  # Minimum memory required per CPU (in megabytes)

#SBATCH --time=03:15:00                                     # Run time in hh:mm:ss

#SBATCH --error=/work/[groupname]/[username]/job.%J.err

#SBATCH --output=/work/[groupname]/[username]/job.%J.out

module load compiler/pgi/11 openmpi/1.6 szip/2.1 zlib/1.2

mpirun /path/to/olam-4.2c-mpi

{{< /highlight >}}

{{% /panel %}}

##### Intel compiler:

{{% panel theme="info" header="Sample submit script for Intel compiler" %}}

{{< highlight batch >}}

#!/bin/sh

#SBATCH --ntasks=8                                          # 8 cores

#SBATCH --mem-per-cpu=1024                                  # Minimum memory required per CPU (in megabytes)

#SBATCH --time=03:15:00                                     # Run time in hh:mm:ss

#SBATCH --error=/work/[groupname]/[username]/job.%J.err

#SBATCH --output=/work/[groupname]/[username]/job.%J.out

module load compiler/intel/12 openmpi/1.6 szip/2.1 zlib/1.2

mpirun /path/to/olam-4.2c-mpi

{{< /highlight >}}

{{% /panel %}}

content/guides/running_applications/running_theano.mddeleted100644 → 0

+0
−89

Original line number
Original line
Diff line number
Diff line

+++

title = "Running Theano"

description = "How to run the Theano on HCC resources."

+++

Theano is available on HCC resources via the modules system. Both CPU and GPU

versions are available on Crane.  Additionally, installs for both Python

2.7 and 3.6 are provided.

### Initial Setup

Theano attempts to write to a `~/.theano` directory in some

circumstances, which can cause errors as the `/home` filesystem is

read-only on HCC machines.  As a workaround, create the directory on

`/work` and make a symlink from `/home`:

{{% panel theme="info" header="Create & symlink .theano directory" %}}

{{< highlight bash >}}

mkdir -p $WORK/.theano

ln -s $WORK/.theano $HOME/.theano

{{< /highlight >}}

{{% /panel %}}

This only needs to be done once on each HCC machine.

### Running the CPU version

To use the CPU version, simply load the module and run your Python code.

 You can choose between the Python 2.7, 3.5 or 3.6 environments:

{{% panel theme="info" header="Python 2.7 version" %}}

{{< highlight bash >}}

module load theano/py27/1.0

python my_python2_script.py

{{< /highlight >}}

{{% /panel %}}

or

{{% panel theme="info" header="Python 3.5 version" %}}

{{< highlight bash >}}

module load theano/py35/1.0

python my_python3_script.py

{{< /highlight >}}

{{% /panel %}}

or

{{% panel theme="info" header="Python 3.6 version" %}}

{{< highlight bash >}}

module load theano/py36/1.0

python my_python3_script.py

{{< /highlight >}}

{{% /panel %}}

### Running the GPU version

To use the GPU version, first create a `~/.theanorc` file with the

following contents (or append to an existing file as needed):

{{% panel theme="info" header="~/.theanorc" %}}

{{< highlight batch >}}

[global]

device = cuda

{{< /highlight >}}

{{% /panel %}}

Next, load the theano module:

{{% panel theme="info" header="Load the theano module" %}}

{{< highlight bash >}}

module load theano/py27/0.9

{{< /highlight >}}

{{% /panel %}}

To test the GPU support, start an interactive job on a GPU node and

import the theano module within the Python interpreter.  You should see

output similar to the following:

{{% panel theme="info" header="GPU support test" %}}

{{< highlight python >}}

Python 2.7.15 | packaged by conda-forge | (default, May  8 2018, 14:46:53)

[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import theano

Using cuDNN version 7005 on context None

Mapped name None to device cuda: Tesla K20m (0000:03:00.0)

{{< /highlight >}}

{{% /panel %}}

content/guides/running_applications/using_anaconda_package_manager.mddeleted100644 → 0

+0
−294

Original line number
Original line
Diff line number
Diff line

+++

title = "Using Anaconda Package Manager"

description = "How to use the Anaconda Package Manager on HCC resources."

+++

[Anaconda](https://www.anaconda.com/what-is-anaconda),

from [Anaconda, Inc](https://www.anaconda.com)

is a completely free enterprise-ready distribution for large-scale data

processing, predictive analytics, and scientific computing. It includes

over 195 of the most popular Python packages for science, math,

engineering, and data analysis. **It also offers the ability to easily

create custom _environments_ by mixing and matching different versions

of Python and/or R and other packages into isolated environments that

individual users are free to create.**  Anaconda includes the `conda`

package and environment manager to make managing these environments

straightforward.

- [Using Anaconda](#using-anaconda)

- [Creating custom Anaconda Environment](#creating-custom-anaconda-environment)

- [Creating custom GPU Anaconda Environment](#creating-custom-gpu-anaconda-environment)

- [Adding Packages to an Existing Environment](#adding-packages-to-an-existing-environment)

- [Using an Anaconda Environment in a Jupyter Notebook on Crane](#using-an-anaconda-environment-in-a-jupyter-notebook-on-crane)

### Using Anaconda

While the standard methods of installing packages via `pip`

and `easy_install` work with Anaconda, the preferred method is using

the `conda` command.  

{{% notice info %}}

Full documentation on using Conda is available

at http://conda.pydata.org/docs/

A [cheatsheet](/attachments/11635089.pdf) is also provided.

{{% /notice %}}

A few examples of the basic commands are provided here.  For a full

explanation of all of Anaconda/Conda's capabilities, see the

documentation linked above. 

Anaconda is provided through the `anaconda` module on HCC machines.  To

begin using it, load the Anaconda module.

{{% panel theme="info" header="Load the Anaconda module to start using Conda" %}}

{{< highlight bash >}}

module load anaconda

{{< /highlight >}}

{{% /panel %}}

To display general information about Conda/Anaconda, use the `info` subcommand.

{{% panel theme="info" header="Display general information about Conda/Anaconda" %}}

{{< highlight bash >}}

conda info

{{< /highlight >}}

{{% /panel %}}

Conda allows the easy creation of isolated, custom environments with

packages and versions of your choosing.  To show all currently available

environments, and which is active, use the `info `subcommand with the

`-e` option.

{{% panel theme="info" header="List available environments" %}}

{{< highlight bash >}}

conda info -e

{{< /highlight >}}

{{% /panel %}}

The active environment will be marked with an asterisk (\*) character.

The `list` command will show all packages installed

in the currently active environment.

{{% panel theme="info" header="List installed packages in current environment" %}}

{{< highlight bash >}}

conda list

{{< /highlight >}}

{{% /panel %}}

To find the names of packages, use the `search` subcommand.

{{% panel theme="info" header="Search for packages" %}}

{{< highlight bash >}}

conda search numpy

{{< /highlight >}}

{{% /panel %}}

If the package is available, this will also display available package

versions and compatible Python versions the package may be installed

under.

### Creating Custom Anaconda Environment

The `create` command is used to create a new environment.  It requires

at a minimum a name for the environment, and at least one package to

install.  For example, suppose we wish to create a new environment, and

need version 1.8 of NumPy.

{{% notice info %}}

The `conda create` command must be run on the login node.

{{% /notice %}}

{{% panel theme="info" header="Create a new environment by providing a name and package specification" %}}

{{< highlight bash >}}

conda create -n mynumpy numpy=1.8 

{{< /highlight >}}

{{% /panel %}}

This will create a new environment called 'mynumpy' and installed NumPy

version 1.8, along with any required dependencies.  

To use the environment, we must first *activate* it.

{{% panel theme="info" header="Activate environment" %}}

{{< highlight bash >}}

source activate mynumpy

{{< /highlight >}}

{{% /panel %}}

Our new environment is now active, and we can use it.  The shell prompt

will change to indicate this as well (this can be disable if desired).

### Creating Custom GPU Anaconda Environment

We provide GPU versions of various frameworks such as `tensorflow`, `keras`, `theano`, via [modules](../module_commands). However, sometimes you may need additional libraries or packages that are not available as part of these modules. In this case, you will need to create your own GPU Anaconda environment.

To do this, you need to first clone one of our GPU modules to a new Anaconda environment, and then install the desired packages in this new environment.

The reason for this is that the GPU modules we support are built using the specific CUDA drivers our GPU nodes have. If you just create custom GPU environment without cloning the module, your code will not utilize the GPUs.

For example, if you want to use `tensorflow` with additional packages, first do:

{{% panel theme="info" header="Cloning GPU module to a new Anaconda environment" %}}

{{< highlight bash >}}

module load tensorflow-gpu/py36/1.12 anaconda

conda create -n tensorflow-gpu-1.12-custom --clone $CONDA_DEFAULT_ENV

module purge

{{< /highlight >}}

{{% /panel %}}

This will create a new `tensorflow-gpu-1.12-custom` environment in your home directory that is a copy of the `tensorflow-gpu` module. Then, you can install the additional packages you need in this environment.

{{% panel theme="info" header="Install new packages in the currently active environment" %}}

{{< highlight bash >}}

module load anaconda

source activate tensorflow-gpu-1.12-custom

conda install <packages>

{{< /highlight >}}

{{% /panel %}}

Next, whenever you want to use this custom GPU Anaconda environment, you need to add these two lines in your submit script:

{{< highlight bash >}}

module load anaconda

source activate tensorflow-gpu-1.12-custom

{{< /highlight >}}

{{% notice info %}}

If you have custom GPU Anaconda environment please only use the two lines from above and **DO NOT** load the module you have cloned earlier. Using `module load tensorflow-gpu/py36/1.12` and `source activate tensorflow-gpu-1.12-custom` in the same script is **wrong** and may give you various errors and incorrect results.

{{% /notice %}}

### Adding Packages to an Existing Environment

To install additional packages in an environment, use the `install`

subcommand.  Suppose we want to install iPython in our 'mynumpy'

environment.  While the environment is active, use `install `with no

additional arguments.  

{{% panel theme="info" header="Install a new package in the currently active environment" %}}

{{< highlight bash >}}

conda install ipython

{{< /highlight >}}

{{% /panel %}}

{{% notice info %}}

The `conda install` command must be run on the login node.

{{% /notice %}}

If you aren't currently in the environment you wish to install the

package in, add the `-n `option to specify the name.

{{% panel theme="info" header="Install new packages in a specified environment" %}}

{{< highlight bash >}}

conda install -n mynumpy ipython

{{< /highlight >}}

{{% /panel %}}

The `remove` subcommand to uninstall a package functions similarly.

{{% panel theme="info" header="Remove package from currently active environment" %}}

{{< highlight bash >}}

conda remove ipython

{{< /highlight >}}

{{% /panel %}}

{{% panel theme="info" header="Remove package from environment specified by name" %}}

{{< highlight bash >}}

conda remove -n mynumpy ipython

{{< /highlight >}}

{{% /panel %}}

To exit an environment, we *deactivate* it.

{{% panel theme="info" header="Exit current environment" %}}

Newer versions of anaconda:

{{< highlight bash >}}

conda deactivate

{{< /highlight >}}

Older versions of anaconda:

{{< highlight bash >}}

source deactivate

{{< /highlight >}}

{{% /panel %}}

Finally, to completely remove an environment, add the `--all `option

to `remove`.

{{% panel theme="info" header="Completely remove an environment" %}}

{{< highlight bash >}}

conda remove -n mynumpy --all

{{< /highlight >}}

{{% /panel %}}

### Using an Anaconda Environment in a Jupyter Notebook on Crane

It is not difficult to make an Anaconda environment available to a

Jupyter Notebook. To do so, follow the steps below, replacing

`myenv` with the name of the Python or R environment you wish to use:

1.  Stop any running Jupyter Notebooks and ensure you are logged out of

    the JupyterHub instance at https://crane.unl.edu

    1.  If you are not logged out, please click the Control Panel button

        located in the top right corner.

    2.  Click the "Stop My Server" Button to terminate the Jupyter

        server.

    3.  Click the logout button in the top right corner.  

2.  Using the command-line environment, load the target conda

    environment:

    {{< highlight bash >}}source activate myenv{{< /highlight >}}

3.  Install the Jupyter kernel and add the environment:

    1.  For a **Python** conda environment, install the IPykernel

        package, and then the kernel specification:

        {{< highlight bash >}}

        # Install ipykernel

        conda install ipykernel

        # Install the kernel specification

        python -m ipykernel install --user --name "$CONDA_DEFAULT_ENV" --display-name "Python ($CONDA_DEFAULT_ENV)"

        {{< /highlight >}}

    2.  For an **R** conda environment, install the jupyter\_client and

        IRkernel packages, and then the kernel specification:

        {{< highlight bash >}}

        # Install PNG support for R, the R kernel for Jupyter, and the Jupyter client

        conda install r-png

        conda install r-irkernel jupyter_client

        # Install jupyter_client 5.2.3 from anaconda channel for bug workaround

        conda install -c anaconda jupyter_client

        # Install the kernel specification

        R -e "IRkernel::installspec(name = '$CONDA_DEFAULT_ENV', displayname = 'R ($CONDA_DEFAULT_ENV)', user = TRUE)"

        {{< /highlight >}}

4.  Once you have the environment set up, deactivate it:

    {{< highlight bash >}}conda deactivate{{< /highlight >}}

5.  To make your conda environments accessible from the worker nodes,

    enter the following commands:

    {{< highlight bash >}}

    mkdir -p $WORK/.jupyter

    mv ~/.local/share/jupyter/kernels $WORK/.jupyter

    ln -s $WORK/.jupyter/kernels ~/.local/share/jupyter/kernels

    {{< /highlight >}}

	{{% notice note %}}

**Note**: Step 5 only needs to be done once. Any future created

environments will automatically be accessible from SLURM notebooks

once this is done.

**Note**: For older version of anaconda, use `source deactivate` to 

deactivate the environment.

{{% /notice %}}

6.  Login to JupyterHub at https://crane.unl.edu

    and create a new notebook using the environment by selecting the

    correct entry in the `New` dropdown menu in the top right

Source

Target

Files

Some changes are not shown.