Compare revisions

public/

.cache/

__pycache__/

site/

**/.DS_Store

variables:

  BASE_URL: "//hcc.unl.edu/docs"

  DEPLOY_ROOT: "/var/www/html/hcc-docs"

  HUGO_TARBALL:  "https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_Linux-64bit.tar.gz"

  GIT_SUBMODULE_STRATEGY: recursive

  MKDOCS_PREFIX: /tmp/gitlab-${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}/mkdocs

  MICROMAMBA_ROOT_PREFIX: /tmp/gitlab-${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}/micromamba

.fetch_external_files: &fetch_external_files

  - curl -L https://raw.githubusercontent.com/unlhcc/singularity-dockerfiles/master/IMAGELIST.md > docs/static/markdown/singularity-images.md

  - curl -L http://swan-head.unl.edu:8192/bio/data/json > docs/static/json/biodata.json

  - curl -L http://swan-head.unl.edu:8192/lmod/spider/json > docs/static/json/lmod.json

  - curl -L http://swan-head.unl.edu:8192/slurm/partitions/json > docs/static/json/swan_partitions.json

stages:

  - test

test:

  stage: test

  image: unlhcc/docker-glibc

  image: python:3.11

  except:

    - master

  tags:

    - docker

  before_script:

    - curl -L -o - ${HUGO_TARBALL} | tar -zxv -C /usr/local/bin

    - apt-get -q update

    - apt-get -q -y install zip pandoc

    - pip install -q -r requirements.txt

    - *fetch_external_files

  script:

    - hugo --ignoreCache -v

    - mkdocs build

    - ./pandoc-build.sh

deploy:

  stage: deploy

    - master

  tags:

    - hcc-docs-prod

  before_script:

    - mkdir -p ${MICROMAMBA_ROOT_PREFIX} ${MKDOCS_PREFIX}

    - *fetch_external_files

    - micromamba create -y -q -p ${MKDOCS_PREFIX}/mkdocs python=3.11 pip cairo pandoc

    - micromamba run -p ${MKDOCS_PREFIX}/mkdocs pip install -q -r requirements.txt

  script:

    - export HUGOTMP=`mktemp -d`

    - curl -L -o - ${HUGO_TARBALL} | tar -zx -C ${HUGOTMP}

    - curl -L https://raw.githubusercontent.com/unlhcc/singularity-dockerfiles/master/IMAGELIST.md > static/markdown/singularity-images.md

    - ${HUGOTMP}/hugo --ignoreCache -b ${BASE_URL} -v

    - pandoc -s content/facilities.md -o public/facilities.docx

    - rsync -avz --delete public/ $DEPLOY_ROOT

    - rm -rf ${HUGOTMP}

    - micromamba run -p ${MKDOCS_PREFIX}/mkdocs mkdocs build

    - find site/ -type f -name "*.html" -exec sed -i -e 's/src="\//src="\/docs\//g' -e 's/href="\//href="\/docs\//g' {} +

    - micromamba run -p ${MKDOCS_PREFIX}/mkdocs ./pandoc-build.sh

    - rsync -avz --delete site/ ${DEPLOY_ROOT}

  after_script:

    - rm -rf /tmp/gitlab-${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}

1.  Checking for outstanding [issues](https://git.unl.edu/hcc/hcc-docs/issues) is a quick way to identify contribution areas. Additionally,

    review an comments on existing merge requests are welcome.

2.  All documentation content can be found within the `content` folder. Subdirectories indicate menu items,

    with the top level document called `_index.md` within each subdirectory.

2.  All documentation content can be found within the `docs` folder. Subdirectories indicate menu items,

    with the top level document called `index.md` within each subdirectory.

Note: please do not submit modifications to theme or CSS items, `_header` or `_footer` files unless they 

Note: please do not submit modifications to theme or CSS items unless they 

adhere to [UNL style guidelines](https://wdn.unl.edu/unl-web-framework-5-standards-guide).

## What to Contribute

This is the repository for HCC's user documentation.  The documentation is a

combination of [Markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

for the source and the [Hugo](https://gohugo.io) static site generator.  Currently, 

the theme used is [docDock](https://themes.gohugo.io/docdock/).

for the source and the [MkDocs](https://www.mkdocs.org/) static site generator.  Currently, 

the theme used is [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/).

The website is viewable at https://hcc.unl.edu/docs.

-----------------

Since the documentation is a git repo, the standard git workflow of creating a branch for your

changes and opening a merge request applies.  The steps below detail how to install Hugo, 

changes and opening a merge request applies.  The steps below detail how to install MkDocs, 

edit/create content, and preview your changes locally before creating the MR/PR.

### Setup your environment

#### Clone the repo (including submodules) and create a new branch

Clone the repo somewhere convenient.  The DocDock theme is provided via a git submodule,

so after cloning, run `git submodule update --init --recursive` in the root of the repo.

Run `git checkout -b <mybranch>` to create a new branch for your changes.

Clone the repo somewhere convenient. 

#### Download and install Hugo

#### Download and install MkDocs

Download the appropriate Hugo tarball from the Hugo [releases page](https://github.com/gohugoio/hugo/releases).

MkDocs, the Material for MkDocs theme, and all plugins are installable with pip. 

(Direct links for [Linux](https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_Linux-64bit.tar.gz), [Mac](https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_macOS-64bit.tar.gz), and [Windows](https://github.com/gohugoio/hugo/releases/download/v0.51/hugo_0.51_Windows-64bit.zip))

The `hugo` program is a single binary with no dependencies and does not require root priviledges, 

so just place it somewhere convenient in your `PATH` (i.e. `~/bin`).

Verify it works by running the command `hugo version`:

```

bash-4.4# hugo version

Hugo Static Site Generator v0.51 linux/amd64 BuildDate: 2018-11-07T10:10:13Z

```bash

pip install mkdocs mkdocs-material mkdocs-nav-weight \

    mkdocs-material[imaging] mkdocs-awesome-nav mkdocs-macros-plugin \

    mkdocs-table-reader-plugin mkdocs-include-markdown-plugin

```

### Previewing the site and adding content

#### Use hugo server to preview the site

#### Use mkdocs serve to preview the site

The `hugo server` command will build the site and run a small webserver to allow you to

The `mkdocs serve` command will build the site and run a small webserver to allow you to

preview it.  Depending whether you're working on your local machine or a VM on Anvil,

the option you pass needs to be slightly different.  The `hugo server` command should be

run from the root of the repository (i.e. the same directory the `config.toml` file

the option you pass needs to be slightly different.  The `mkdocs serve` command should be

run from the root of the repository (i.e. the same directory the `mkdocs.yml` file

resides in).

**If you're working on your local laptop/deskop:**

This will allow you to view the webiste by going to `http://localhost:8080` in your web browser. 

Run the command:

```

hugo server -b http://127.0.0.1

```

Alternatively you can build the HTML files to view in a web browser using the `mkdocs build` command 

from the root of the repository. 

You should now be able to preview the website by going to `http://localhost:1313` in your web

browser. 

**If you're working in a VM on Anvil:**

You'll first need to allow port 1313.  If your VM is under the `swanson` project, there is

already an existing "Hugo" security group.  Just add that group to your VM.  Otherwise, first

create it and allow TCP port 1313 though.  You may also need to allow port 1313 through your

You'll first need to allow port 8080.  If your VM is under the `swanson` project, there is

already an existing "Port 8080" security group.  Just add that group to your VM.  Otherwise, first

create it and allow TCP port 8080 though.  You may also need to allow port 8080 through your

instance's firewall.

Once that is done, run the command:

```

hugo server -b http://<instance IP> --bind <instance IP>

mkdocs serve

```

where *\<instance IP\>* is the 10.71.x.x address of your instance.

You should now be able to preview the website by going to `http://<instance IP>:1313` in your web

browser.

You should now be able to preview the website by going to `http://<instance IP>:8080` in your web

browser, where *\<instance IP\>* is the 10.71.x.x address of your instance.

*Note:  The hugo server command will watch the filesystem for changes and force an update in your browswer,

*Note:  The mkdocs serve command will watch the filesystem for changes and force an update in your browswer,

so you can simply leave the server running as you work.*

**Note that the `mkdocs serve` command will not serve image files. If you need to preview images, you'll need to

run nginx via Docker as noted below.**

#### Running nginx using Docker

The built-in mkdocs server doesn't serve images. To fully preview the site either locally or on your VM

you'll need to run an nginx container. Install Docker and ensure port 8080 is allowed through to your VM as above if needed.

First build the site by running

```

mkdocs build

```

in the repo root. Still in the top-level folder of the repo, then run

```

docker run -d --name nginx-hcc-docs -v "${PWD}/site:/usr/share/nginx/html:ro" -p 8080:80 nginx:latest

```

This will start an nginx container in the background. You can then access the site at port 8080 either on localhost

or your VM's IP.  Note that the autorebuild function does not work here.

So you'll need to manually run `mkdocs rebuild` to pick up changes.

Once you're done, remove the container by running

```

docker rm -f nginx-docs

```

#### Adding content

The page content for the site is under the `content/` directory.  The directory structure will determine

The page content for the site is under the `docs/` directory.  The directory structure will determine

how the site itself is structured, so place content based on where in the site you would like it to be.

Each level in the tree will need to have an `_index.md` file to designate it as a new section.  The content

itself is almost all standard markdown.  However, there are a few things that are specific to Hugo that

Each level in the tree will need to have an `index.md` file to designate it as a new section.  The content

itself is almost all standard markdown.  However, there are a few things that are specific to MkDocs that

are used:

1.  *Front matter*  

    [Front matter](https://gohugo.io/content-management/front-matter) is Hugo-specific metadata added

    to the page.  The format can be TOML, YAML, or JSON.  The example here uses TOML, which is identified

    by opening and closing `+++`.  At minimum, each page should have a title and description:  

1.  *YAML Style Metadata*  

    [YAML Style Metadata](https://www.mkdocs.org/user-guide/writing-your-docs/#yaml-style-meta-data) is mkdocs-specific metadata added

    to the page.  The format needs to be YAML.  The example here is for our contact-us page, and is identified

    by opening and closing `---`  At minimum, each page should have a title and description:  

    ```

    +++

    title = "Contact Us"

    description = "Contact Information for HCC"

    +++

    ---

    title: "Contact Us"

    description: "Contact Information for HCC"

    ---

    ```  

    Another useful parameter is the `weight` option.  This will determine the order of display in the navigation tree;

    lower numbers are displayed before higher numbers.

2.  *Shortcodes*  

    [Shortcodes](https://gohugo.io/content-management/shortcodes) are "helper functions" to add content

    that would otherwise require using raw HTML.  Hugo provides some [built-in](https://gohugo.io/content-management/shortcodes/#use-hugo-s-built-in-shortcodes)

    shortcodes. The DocDock theme also provides its own [shortcodes](http://docdock.netlify.com/shortcodes).

    Shortcodes are added via a `{{% shortcodename parameters %}}` declaration.  For example, to embed a YouTube

    video with ID `-Vh7SyC-3mA`:  

    ```

    {{< youtube -Vh7SyC-3mA >}}

    ```  

    Other useful shortcodes include language-specific syntax highlighting, figures, and automatic listing

    of page children for navgiation/summary pages.

3.  *Using `relref` for link generation*  

    The `relref` shortcode should generally be used for linking to other pages within the site.  This will

    generate relative links that should work no matter what the base URL is.  For example, assume you're

    editing "my\_page.md" and want to link to the page "my\_other\_page.md" in the subdirectory "mystuff".

    To do so, add the line  

2.  *Macros and variables*  

    [Macros](https://mkdocs-macros-plugin.readthedocs.io/en/latest/) are "helper functions" to add content

    that would otherwise require using raw HTML.  

    Currently, a few custom ones are implemented. 

    - `{{ youtube('Emded_URL_OR_Share_Code') }}` - Embeds a youtube video into the docs

    - `{{ md_table('path_or_url') }}` - Embeds a markdown table as a sortable table. 

    - `{{ json_table('path_or_url') }}` - Embeds a JSON table as a sortable table. 

    Variables allow for the use of defining shorthand codes for using the same value in multiple places in the documentation.

    These are all defined in the `mkdocs.yml` file in the `extra:` section. 

    Variables are called using `{{ variable.path.to.value }}` where the path starts after the `extra:` block. 

    ```yaml

    extra:

      hcc:  

      # Swan Cluster Specific Information

        swan:

          xfer: "swan-xfer.unl.edu"

          ood:

            name: "Swan Open OnDemand"

            url: "https://swan-ood.unl.edu"

          work:

            block: "100 TiB"

    ```

    [My other page]({{< relref "mystuff/my_other_page" >}})

    Using a section of the YAML above we can get the following exampls:

    - `{{ hcc.swan.xfer }}` would yield the URL for Swan's high speed transfer node

    - `{{ hcc.swan.ood.url }}` would yield the block quota for Swan's $WORK filesystem

    - `{{ hcc.swan.work.block }}` would yield the Open OnDemand URL for Swan

    - ` {{ children('path_after_docs') }} ` - List child pages and Descriptions - do not add trailing `/`

    A good file for examples of the use of these variables is `docs/handling_data/index.md` where many variables 

    are used to discuss the filesystems.

3.  Code Blocks

    Code blocks are able to use language specific formatting and use the standard markdown format for codeblocks. 

    ```` markdown title="Example Code Block"

    ``` python

    print('Hello World!')

    ```

    ````

4.  Links 

    MkDocs uses standard markdown links, `[Text to Display](URL or path/to/file)`

    Note that you can omit the `.md` suffix from the filename.

4.  *Static content*  

    Hugo has special handling for static content (images, linked zip files, etc.).  In the root of the repo,

    there is a `static/` directory; all static content should go there.  Subdirectories can be made to keep

    things organized; there are existing directories for `images` and `attachments`.  Feel free to create

    additional directories if it makes sense.  To link to static content, you can use an absolute path from

    any page.  For example, assuming there is an image in the repo at `static/images/my_image.png`,

    you can link to it using the `figure` shortcode as `{{< figure src="/images/my_image.png" >}}`.

5.  *Static content*  

    1.  Images

        Images are stored in `docs/images`. 

        Images can then be embeded in one of two methods. The path starts from the `images` directory. 

        Using markdown: `![Optional Image Title](/image/picture.png)`

        Using HTML: `<img src="/images/picture.png">`

        - HTML will also allow the use of standard `img` embedding and customization. 

    2.  Markdown and HTML Files

        Sometimes there are cases where information is stored statically elsewhere. These files are organized in `docs/static`

        ``` markdown

        {% 

            include "relative/path/to/file.html"

        %}

        ```

    -  **Special note for .png files**:  In order to keep the repo size as small as possible, all images

       should be png's if possible, and further processed using the [pngquant](https://pngquant.org) compression

       tool.  This will reduce the size by ~3-4x while keeping good quality.  For example, to process and

       prior to committing the files to the repo, otherwise git will be forced to keep multiple copies as the files

       are binary.*

6. Callouts

    These are the callouts used to highlight information and have a few customization options. 

    Callout Template:

    ``` markdown

    !!! type "Optional Title"

        Content goes here. It is important to make sure to have space in the markdown file above and below the callout. 

        Content must also be indented 

    ```

    A full list of callout types and additional usage information is available in the [Material for MkDocs documentation page](https://squidfunk.github.io/mkdocs-material/reference/admonitions/#supported-types).

#### Adding your changes

Once you've got your set of changes, add and commit them to your branch.  Push the branch and

---

title: "{{ replace .Name "-" " " | title }}"

date: {{ .Date }}

draft: true

---

#baseURL = "http://hcc-docs.unl.edu/"

languageCode = "en-us"

title = "HCC-DOCS"

theme = "docdock"

contentdir    = "content"

layoutdir     = "layouts"

publishdir    = "public"

baseURL = "/"

canonifyURLs = "true"

relativeURLs = "true"

DefaultContentLanguage = "en"

pygmentsCodeFences = true

pygmentsStyle = "monokailight"

defaultContentLanguage = "en"

defaultContentLanguageInSubdir= false

enableMissingTranslationPlaceholders = false

[params]

editURL = "https://git.unl.edu/hcc/hcc-docs/edit/master/content/"

showVisitedLinks = true # default is false

themeStyle = "original" # "original" or "flex" # default "flex"

themeVariant = "blue" # choose theme variant "green", "gold" , "gray", "blue" (default)

ordersectionsby = "weight" # ordersectionsby = "title"

disableHomeIcon = false # default is false

disableSearch = false # default is false

disableNavChevron = false # set true to hide next/prev chevron, default is false

highlightClientSide = false # set true to use highlight.pack.js instead of the default hugo chroma highlighter

menushortcutsnewtab = true # set true to open shortcuts links to a new tab/window

enableGitInfo = true

[outputs]

home = [ "HTML", "RSS", "JSON"]

[[menu.shortcuts]]

name = "hcc.unl.edu"

identifier = "ds"

url = "https://hcc.unl.edu"

weight = 10

# Hugo v0.60+ switched renderer from blackfriday to goldmark

# - Allow unsafe for docdock template rendering

[markup.goldmark.renderer]

unsafe = true

+++

title = "2012"

description = "Historical listing of various HCC events for the year 2012."

weight = "70"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "2013"

description = "Historical listing of various HCC events for the year 2013."

weight = "60"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "2014"

description = "Historical listing of various HCC events for the year 2014."

weight = "50"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "2015"

description = "Historical listing of various HCC events for the year 2015."

weight = "40"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "2016"

description = "Historical listing of various HCC events for the year 2016."

weight = "30"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "2017"

description = "Historical listing of various HCC events for the year 2017."

weight = "20"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "2018"

description = "Listing of various HCC events for the year 2018."

weight = "10"

+++

Historical listing of HCC Events

----------

{{% children %}}

+++

title = "Events"

description = "Historical listing of various HCC events."

weight = "70"

hidden = true

+++

Historical listing of HCC Events

----------

{{% children sort="weight" description="true" %}}

+++

title = "FAQ"

description = "HCC Frequently Asked Questions"

weight = "95"

+++

- [I have an account, now what?](#i-have-an-account-now-what)

- [How do I change my password?](#how-do-i-change-my-password)

- [I forgot my password, how can I retrieve it?](#i-forgot-my-password-how-can-i-retrieve-it)

- [I just deleted some files and didn't mean to! Can I get them back?](#i-just-deleted-some-files-and-didn-t-mean-to-can-i-get-them-back)

- [How do I (re)activate Duo?](#how-do-i-re-activate-duo)

- [How many nodes/memory/time should I request?](#how-many-nodes-memory-time-should-i-request)

- [I am trying to run a job but nothing happens?](#i-am-trying-to-run-a-job-but-nothing-happens)

- [I keep getting the error "slurmstepd: error: Exceeded step memory limit at some point." What does this mean and how do I fix it?](#i-keep-getting-the-error-slurmstepd-error-exceeded-step-memory-limit-at-some-point-what-does-this-mean-and-how-do-i-fix-it)

- [I want to talk to a human about my problem. Can I do that?](#i-want-to-talk-to-a-human-about-my-problem-can-i-do-that)

---

#### I have an account, now what?

Congrats on getting an HCC account! Now you need to connect to a Holland

cluster. To do this, we use an SSH connection. SSH stands for Secure

Shell, and it allows you to securely connect to a remote computer and

operate it just like you would a personal machine.

Depending on your operating system, you may need to install software to

make this connection. Check out our documentation on [Connecting to HCC Clusters]

({{< relref "../connecting/" >}}).

#### How do I change my password?

#### I forgot my password, how can I retrieve it?

Information on how to change or retrieve your password can be found on

the documentation page: [How to change your

password]({{< relref "/accounts/how_to_change_your_password" >}})

All passwords must be at least 8 characters in length and must contain

at least one capital letter and one numeric digit. Passwords also cannot

contain any dictionary words. If you need help picking a good password,

consider using a (secure!) password generator such as

[this one provided by Random.org](https://www.random.org/passwords)

To preserve the security of your account, we recommend changing the

default password you were given as soon as possible.

#### I just deleted some files and didn't mean to! Can I get them back?

That depends. Where were the files you deleted?

**If the files were in your $HOME directory (/home/group/user/):** It's

possible.

$HOME directories are backed up daily and we can restore your files as

they were at the time of our last backup. Please note that any changes

made to the files between when the backup was made and when you deleted

them will not be preserved. To have these files restored, please contact

HCC Support at

{{< icon name="envelope" >}}[hcc-support@unl.edu](mailto:hcc-support@unl.edu)

as soon as possible.

**If the files were in your $WORK directory (/work/group/user/):** No.

Unfortunately, the $WORK directories are created as a short term place

to hold job files. This storage was designed to be quickly and easily

accessed by our worker nodes and as such is not conducive to backups.

Any irreplaceable files should be backed up in a secondary location,

such as Attic, the cloud, or on your personal machine. For more

information on how to prevent file loss, check out [Preventing File

Loss]({{< relref "preventing_file_loss" >}}).

#### How do I (re)activate Duo?

**If you have not activated Duo before:**

Please stop by

[our offices](http://hcc.unl.edu/location)

along with a photo ID and we will be happy to activate it for you. If

you are not local to Omaha or Lincoln, contact us at

{{< icon name="envelope" >}}[hcc-support@unl.edu](mailto:hcc-support@unl.edu)

and we will help you activate Duo remotely.

**If you have activated Duo previously but now have a different phone

number:**

Stop by our offices along with a photo ID and we can help you reactivate

Duo and update your account with your new phone number.

**If you have activated Duo previously and have the same phone number:**

Email us at

{{< icon name="envelope" >}}[hcc-support@unl.edu](mailto:hcc-support@unl.edu)

from the email address your account is registered under and we will send

you a new link that you can use to activate Duo.

#### How many nodes/memory/time should I request?

**Short answer:** We don’t know.

**Long answer:** The amount of resources required is highly dependent on

the application you are using, the input file sizes and the parameters

you select. Sometimes it can help to speak with someone else who has

used the software before to see if they can give you an idea of what has

worked for them.

Ultimately, it comes down to trial and error; try different

combinations and see what works and what doesn’t. Good practice is to

check the output and utilization of each job you run. This will help you

determine what parameters you will need in the future.

For more information on how to determine how many resources a completed

job used, check out the documentation on [Monitoring Jobs]({{< relref "monitoring_jobs" >}}).

#### I am trying to run a job but nothing happens?

Where are you trying to run the job from? You can check this by typing

the command \`pwd\` into the terminal.

**If you are running from inside your $HOME directory

(/home/group/user/)**:

Move your files to your $WORK directory (/work/group/user) and resubmit

your job.

The worker nodes on our clusters have read-only access to the files in

$HOME directories. This means that when a job is submitted from $HOME,

the scheduler cannot write the output and error files in the directory

and the job is killed. It appears the job does nothing because no output

is produced.

**If you are running from inside your $WORK directory:**

Contact us at

{{< icon name="envelope" >}}[hcc-support@unl.edu](mailto:hcc-support@unl.edu)

with your login, the name of the cluster you are running on, and the

full path to your submit script and we will be happy to help solve the

issue.

##### I keep getting the error "slurmstepd: error: Exceeded step memory limit at some point." What does this mean and how do I fix it?

This error occurs when the job you are running uses more memory than was

requested in your submit script.

If you specified `--mem` or `--mem-per-cpu` in your submit script, try

increasing this value and resubmitting your job.

If you did not specify `--mem` or `--mem-per-cpu` in your submit script,

chances are the default amount allotted is not sufficient. Add the line

{{< highlight batch >}}

#SBATCH --mem=<memory_amount>

{{< /highlight >}}

to your script with a reasonable amount of memory and try running it again. If you keep

getting this error, continue to increase the requested memory amount and

resubmit the job until it finishes successfully.

For additional details on how to monitor usage on jobs, check out the

documentation on [Monitoring Jobs]({{< relref "monitoring_jobs" >}}).

If you continue to run into issues, please contact us at

{{< icon name="envelope" >}}[hcc-support@unl.edu](mailto:hcc-support@unl.edu)

for additional assistance.

#### I want to talk to a human about my problem. Can I do that?

Of course! We have an open door policy and invite you to stop by

[either of our offices](http://hcc.unl.edu/location)

anytime Monday through Friday between 9 am and 5 pm. One of the HCC

staff would be happy to help you with whatever problem or question you

have.  Alternatively, you can drop one of us a line and we'll arrange a

time to meet:  [Contact Us](https://hcc.unl.edu/contact-us).

+++

title = "The Open Science Grid"

description = "How to utilize the Open Science Grid (OSG)."

weight = "80"

+++

If you find that you are not getting access to the volume of computing

resources needed for your research through HCC, you might also consider

submitting your jobs to the Open Science Grid (OSG). 

### What is the Open Science Grid?

The [Open Science Grid](http://opensciencegrid.org) advances

science through open distributed computing. The OSG is a

multi-disciplinary partnership to federate local, regional, community

and national cyber infrastructures to meet the needs of research and

academic communities at all scales. HCC participates in the OSG as a

resource provider and a resource user. We provide HCC users with a

gateway to running jobs on the OSG.

The map below shows the Open Science Grid sites located across the U.S.

{{< figure src="/images/17044917.png" >}}

This help document is divided into four sections, namely:

- [Characteristics of an OSG friendly job]({{< relref "characteristics_of_an_osg_friendly_job" >}})

- [How to submit an OSG Job with HTCondor]({{< relref "how_to_submit_an_osg_job_with_htcondor" >}})

- [A simple example of submitting an HTCondorjob]({{< relref "a_simple_example_of_submitting_an_htcondor_job" >}})

- [Using Distributed Environment Modules on OSG]({{< relref "using_distributed_environment_modules_on_osg" >}})

+++

title = "A simple example of submitting an HTCondor job"

description = "A simple example of submitting an HTCondor job."

weight=30

+++

This page describes a complete example of submitting an HTCondor job.

1.  SSH to Crane

    {{% panel theme="info" header="ssh command" %}}

    [apple@localhost]ssh apple@crane.unl.edu

    {{% /panel %}}

    {{% panel theme="info" header="output" %}}

    [apple@login.crane~]$

    {{% /panel %}}

2.  Write a simple python program in a file "hello.py" that we wish to

    run using HTCondor

    {{% panel theme="info" header="edit a python code named 'hello.py'" %}}

    [apple@login.crane ~]$ vim hello.py

    {{% /panel %}}

    Then in the edit window, please input the code below:

    {{% panel theme="info" header="hello.py" %}}

    #!/usr/bin/env python

    import sys

    import time

    i=1

    while i<=6:

            print i

            i+=1

            time.sleep(1)

    print 2**8

    print "hello world received argument = " +sys.argv[1]

    {{% /panel %}}

    This program will print 1 through 6 on stdout, then print the number

    256, and finally print  `hello world received argument = <Command

    Line Argument Sent to the hello.py>`

3.  Write an HTCondor submit script named "hello.submit"

    {{% panel theme="info" header="hello.submit" %}}

    Universe        = vanilla

    Executable      = hello.py

    Output          = OUTPUT/hello.out.$(Cluster).$(Process).txt

    Error           = OUTPUT/hello.error.$(Cluster).$(Process).txt

    Log             = OUTPUT/hello.log.$(Cluster).$(Process).txt

    notification = Never

    Arguments = $(Process)

    PeriodicRelease = ((JobStatus==5) && (CurentTime - EnteredCurrentStatus) > 30)

    OnExitRemove = (ExitStatus == 0)

    Queue 4

    {{% /panel %}}

4.  Create an OUTPUT directory to receive all output files that

    generated by your job (OUTPUT folder is used in the submit script

    above )

    {{% panel theme="info" header="create output directory" %}}

    [apple@login.crane ~]$ mkdir OUTPUT

    {{% /panel %}}

5.  Submit your job

    {{% panel theme="info" header="condor_submit" %}}

    [apple@login.crane ~]$ condor_submit hello.submit

    {{% /panel %}}

    {{% panel theme="info" header="Output of submit" %}}

    Submitting job(s)

    ....

    4 job(s) submitted to cluster 1013054.

    {{% /panel %}}

6.  Check status of `condor_q`

    {{% panel theme="info" header="condor_q" %}}

    [apple@login.crane ~]$ condor_q

    {{% /panel %}}

    {{% panel theme="info" header="Output of `condor_q`" %}}

    -- Schedd: login.crane.hcc.unl.edu : <129.93.227.113:9619?...

     ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

    720587.0   logan          12/15 10:48  33+14:41:17 H  0   0.0  continuous.cron 20

    720588.0   logan          12/15 10:48 200+02:40:08 H  0   0.0  checkprogress.cron

    1012864.0   jthiltge        2/15 16:48   0+00:00:00 H  0   0.0  test.sh

    1013054.0   jennyshao       4/3  17:58   0+00:00:00 R  0   0.0  hello.py 0

    1013054.1   jennyshao       4/3  17:58   0+00:00:00 R  0   0.0  hello.py 1

    1013054.2   jennyshao       4/3  17:58   0+00:00:00 I  0   0.0  hello.py 2

    1013054.3   jennyshao       4/3  17:58   0+00:00:00 I  0   0.0  hello.py 3

    7 jobs; 0 completed, 0 removed, 0 idle, 4 running, 3 held, 0 suspended