Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • FAQ
  • RDPv10
  • UNL_OneDrive
  • atticguidelines
  • data_share
  • globus-auto-backups
  • good-hcc-practice-rep-workflow
  • hchen2016-faq-home-is-full
  • ipynb-doc
  • master
  • rclone-fix
  • sislam2-master-patch-51693
  • sislam2-master-patch-86974
  • site_url
  • test
15 results

Target

Select target project
  • dweitzel2/hcc-docs
  • OMCCLUNG2/hcc-docs
  • salmandjing/hcc-docs
  • hcc/hcc-docs
4 results
Select Git revision
  • 26-add-screenshots-for-newer-rdp-v10-client
  • 28-overview-page-for-connecting-2
  • AddExamples
  • OMCCLUNG2-master-patch-74599
  • RDPv10
  • globus-auto-backups
  • gpu_update
  • master
  • mtanash2-master-patch-75717
  • mtanash2-master-patch-83333
  • mtanash2-master-patch-87890
  • mtanash2-master-patch-96320
  • patch-1
  • patch-2
  • patch-3
  • runTime
  • submitting-jobs-overview
  • tharvill1-master-patch-26973
18 results
Show changes
Showing
with 1140 additions and 225 deletions
---
title: Formatting and mounting a volume in Linux
summary: "How to format and mount volume as a hard drive in Linux."
---
!!! note
This guide assumes you associated your SSH Key Pair with the instance
when it was created, and that you are connected to the [Anvil VPN](../connecting_to_the_anvil_vpn/).
Once you have [created and attached](../creating_and_attaching_a_volume/)
your volume, it must be formatted and mounted in your Linux instance to be usable. This
procedure is identical to what would be done when attaching a second
hard drive to a physical machine. In this example, a 1GB volume was
created and attached to the instance. Note that the majority of this
guide is for a newly created volume.
!!! note
**If you are attaching an existing volume with data already on it,
skip to [creating a directory and mounting the volume](#mounting-the-volume).**
#### Formatting the volume
Follow the relevant guide
([Windows](../connecting_to_linux_instances_from_windows/)
| [Mac](../connecting_to_linux_instances_from_mac/)) for your
operating system to connect to your instance. Formatting and mounting
the volume requires root privileges, so first run the
command `sudo su -` to get a root shell.
!!! danger "**Running commands as root**"
<img src="/images/anvil-volumes/1-sudo.png" width="576">
Next, you will need to determine what device the volume is presented as
within Linux. Typically this will be `/dev/vdb`, but it is necessary to
verify this to avoid mistakes, especially if you have more than one
volume attached to an instance. The command `lsblk` will list the
hard drive devices and partitions.
<img src="/images//anvil-volumes/2-lsblk.png" width="576">
Here there is a completely empty (no partitions) disk device matching
the 1GB size of the volume, so `/dev/vdb` is the correct device.
The `parted` utility will first be used to label the device and then create a partition.
```bash
parted /dev/vdb mklabel gpt
parted /dev/vdb mkpart primary 0% 100%
```
<img src="/images/anvil-volumes/3-mkpart.png" width="576">
Now that a partition has been created, it can be formatted. Here, the
ext4 filesystem will be used. This is the default filesystem used by
many Linux distributions including CentOS and Ubuntu, and is a good
general choice. An alternate filesystem may be used by running a
different format command. To format the partition using ext4, run the
command `mkfs.ext4 /dev/vdb1`. You will see a progress message and then
be returned to the shell prompt.
<img src="/images/anvil-volumes/4-mkfs.png" width="576">
#### Mounting the volume
!!! note
If you are attaching a pre-existing volume, start here.
Finally, the formatted partition must be mounted as a directory to be
used. By convention this is done under `/mnt`, but you may choose to
mount it elsewhere depending on the usage. Here, a directory
called `myvolume` will be created and the volume mounted there. Run the
following commands to make the directory and mount the volume:
```bash
mkdir /mnt/myvolume
mount /dev/vdb1 /mnt/myvolume
```
<img src="/images/anvil-volumes/5-mount.png" width="576">
Running the command `df -h` should then show the new mounted empty
volume.
<img src="/images/anvil-volumes/6-df.png" width="576">
The volume can now be used.
+++
title = "Formatting and mounting a volume in Windows"
description = "How to format and mount volume as a hard drive in Windows."
+++
---
title: Formatting and mounting a volume in Windows
summary: "How to format and mount volume as a hard drive in Windows."
---
{{% notice info %}}
!!! note
This guide assumes you associated your SSH Key Pair with the instance
when it was created, and that you are connected to the [Anvil VPN]({{< relref "connecting_to_the_anvil_vpn" >}}).
{{% /notice %}}
when it was created, and that you are connected to the [Anvil VPN](../connecting_to_the_anvil_vpn/).
{{% notice note %}}
!!! note
**These steps are only for a newly created volume.** When attaching a
previously formatted volume to an instance, it will automatically be visible as a second drive.
{{% /notice %}}
---
Once you have [created and attached]({{< relref "creating_and_attaching_a_volume" >}}) your
Once you have [created and attached](../creating_and_attaching_a_volume/) your
volume, it must be formatted and mounted in your Windows instance to be
usable. This procedure is identical to what would be done when
attaching a second hard drive to a physical machine. In this example, a
......@@ -23,55 +23,55 @@ attaching a second hard drive to a physical machine.  In this example, a
From the *Start* menu, right-click on *Computer*, and select *Manage*.
{{< figure src="/images/13599562.png" >}}
<img src="/images/13599562.png">
This will open the *Computer Management* utility. On the left-hand side
menu, click *Disk Management*.
{{< figure src="/images/13599566.png" width="800" >}}
<img src="/images/13599566.png" width="800">
You should see two entries. One for the OS hard drive, and a second
unallocated disk.
{{< figure src="/images/13599569.png" width="800" >}}
<img src="/images/13599569.png" width="800">
Right click anywhere in the white rectangle for the unallocated disk and
choose *New Simple Volume...*.
{{< figure src="/images/13599574.png" width="800" >}}
<img src="/images/13599574.png" width="800">
This will bring up the *New Simple Volume Wizard*. Click *Next* to
continue.
{{< figure src="/images/13599577.png" >}}
<img src="/images/13599577.png">
The default is to format the entire volume. Unless there is a specific
reason not to, click *Next* to continue.
{{< figure src="/images/13599578.png" >}}
<img src="/images/13599578.png">
The default is to assign the volume a drive letter (typically *D*).
Click *Next* to continue.
{{< figure src="/images/13599579.png" >}}
<img src="/images/13599579.png">
Change the *Volume label* field if you wish the drive to be named
something other than the default, and click *Next* to continue.
{{< figure src="/images/13599580.png" >}}
<img src="/images/13599580.png">
Finally, click *Finish* to close the wizard.
{{< figure src="/images/13599581.png" >}}
<img src="/images/13599581.png">
In the *Computer Management* window, you should now see the formatted
and mounted volume.
{{< figure src="/images/13599584.png" width="800" >}}
<img src="/images/13599584.png" width="800">
If you open the *Computer* window, you should now see the volume as a
second hard drive.
{{< figure src="/images/13599585.png" >}}
<img src="/images/13599585.png">
The volume can now be used.
+++
title = "Anvil: HCC's Cloud"
description = "How to use Anvil, HCC's OpenStack-based cloud resource"
weight = "40"
+++
---
title: Anvil - HCC's Cloud
summary: How to use Anvil, HCC's OpenStack-based cloud resource
weight: 7
---
- [Overview](#overview)
- [Cloud Terms](#cloud-terms)
......@@ -10,21 +10,20 @@ weight = "40"
- [Backups](#backups)
{{% notice tip %}}
Have your account and ready to go? Visit the Anvil OpenStack web
interface at https://anvil.unl.edu.
{{% /notice %}}
!!! tip
Have your account and ready to go? Visit the Anvil OpenStack web interface at [https://anvil.unl.edu](https://anvil.unl.edu).
---
### Overview
Anvil is the Holland Computing Center's cloud computing resource, based
on the [OpenStack](https://www.openstack.org) software.
on the [OpenStack](https://www.openstack.org) software.
OpenStack is a free and open-source software platform for
cloud computing. Anvil was created to address the needs of NU's
cloud computing. Anvil was created to address the needs of NU's
research community that are not well served by a traditional
batch-scheduled Linux cluster environment. Examples of use cases that
batch-scheduled Linux cluster environment. Examples of use cases that
are well suited to Anvil include:
- A highly interactive environment, especially GUI applications
......@@ -39,80 +38,80 @@ are well suited to Anvil include:
web or database server
Using Anvil, one or more virtual machines (VMs) can be easily be created
via a user-friendly web dashboard. Created VMs are then accessible from
via a user-friendly web dashboard. Created VMs are then accessible from
HCC clusters, or your own workstation once connected to the Anvil
Virtual Private Network (VPN). Access is through standard means,
Virtual Private Network (VPN). Access is through standard means,
typically via SSH for Linux VMs and Remote Desktop for Windows VMs.
### Cloud Terms
There are a few terms used within the OpenStack interface and in the
instructions below that may be unfamiliar. The following brief
definitions may be useful. More detailed information is available in
the [OpenStack User Guide](http://docs.openstack.org/user-guide).
- **Project**: A project is the base unit of ownership in
OpenStack. Resources (CPUs, RAM, storage, etc.) are allocated and
user accounts are associated with a project. Within Anvil, each HCC
research group corresponds directly to a project. Similar to
instructions below that may be unfamiliar. The following brief
definitions may be useful. More detailed information is available in
the [OpenStack User Guide](http://docs.openstack.org/user-guide).
- **Project**: A project is the base unit of ownership in
OpenStack. Resources (CPUs, RAM, storage, etc.) are allocated and
user accounts are associated with a project. Within Anvil, each HCC
research group corresponds directly to a project. Similar to
resource allocation on HCC clusters, the members of a group share
the [project's resources]({{< relref "what_are_the_per_group_resource_limits" >}}).
- **Image**: An image corresponds to everything needed to create a
the [project's resources](what_are_the_per_group_resource_limits/).
- **Image**: An image corresponds to everything needed to create a
virtual machine for a specific operating system (OS), such as Linux
or Windows. HCC creates and maintains [basic Windows and Linux]({{< relref "available_images" >}})
or Windows. HCC creates and maintains [basic Windows and Linux](available_images/)
images for convenience.
Users can also create their own images that can then be uploaded to
OpenStack and used within the project.
- **Flavor**: A flavor (also known as *instance type*), defines the
parameters (i.e. resources) of the virtual machine. This includes
things such as number of CPUs, amount of RAM, storage, etc. There
are many instance types [available within Anvil]({{< relref "anvil_instance_types" >}}),
- **Flavor**: A flavor (also known as *instance type*), defines the
parameters (i.e. resources) of the virtual machine. This includes
things such as number of CPUs, amount of RAM, storage, etc. There
are many instance types [available within Anvil](anvil_instance_types/),
designed to meet a variety of needs.
- **Instance**: An instance is a running virtual machine, created
- **Instance**: An instance is a running virtual machine, created
by combining an image (the basic OS) with a flavor (resources).
That is, *Image + Flavor = Instance*.
- **Volume**: A volume is a means for persistent storage within
OpenStack. When an instance is destroyed, any additional data that
was on the OS hard drive is lost. A volume can be thought of
similar to an external hard drive. It can be attached to an
instance and accessed as a second drive. When the instance is
destroyed, data on the volume is retained. It can then be attached
That is, *Image + Flavor = Instance*.
- **Volume**: A volume is a means for persistent storage within
OpenStack. When an instance is destroyed, any additional data that
was on the OS hard drive is lost. A volume can be thought of
similar to an external hard drive. It can be attached to an
instance and accessed as a second drive. When the instance is
destroyed, data on the volume is retained. It can then be attached
and accessed from another instance later.
### Steps for Access
The guide below outlines the steps needed to begin using Anvil. Please
note that Anvil is currently in the *beta testing* phase. While
The guide below outlines the steps needed to begin using Anvil. Please
note that Anvil is currently in the *beta testing* phase. While
reasonable precautions are taken against data loss, **sole copies of
precious or irreproducible data should not be placed or left on Anvil**.
1. **Request access to Anvil**
1. **Request access to Anvil**
Access and resources are provided on a per-group basis, similar to
HCC clusters. For details, please see [What are the per group
resource limits?]({{< relref "what_are_the_per_group_resource_limits" >}})
HCC clusters. For details, please see [What are the per group
resource limits?](what_are_the_per_group_resource_limits/)
To begin using Anvil, user should fill out the short request form
at http://hcc.unl.edu/request-anvil-access.
at http://hcc.unl.edu/request-anvil-access.
An automated confirmation email will be sent. After group owner approves the request, an HCC staff
member will follow-up once access is available.
2. **Create SSH keys**
OpenStack uses SSH key pairs to identify users and control access to
the VMs themselves, as opposed to the traditional username/password
combination. SSH key pairs consist of two files, a public key and a
private key. The public file can be shared freely; this file will
be uploaded to OpenStack and associated with your account. The
private key file should be treated the same as a password. **Do not
combination. SSH key pairs consist of two files, a public key and a
private key. The public file can be shared freely; this file will
be uploaded to OpenStack and associated with your account. The
private key file should be treated the same as a password. **Do not
share your private key and always keep it in a secure location.**
Even if you have previously created a key pair for another purpose,
Even if you have previously created a key pair for another purpose,
it's best practice to create a dedicated pair for use with Anvil.
The process for creating key pairs is different between Windows and
Mac. Follow the relevant guide below for your operating system.
1. [Creating SSH key pairs on Windows]({{< relref "creating_ssh_key_pairs_on_windows" >}})
2. [Creating SSH key pairs on Mac]({{< relref "creating_ssh_key_pairs_on_mac" >}})
The process for creating key pairs is different between Windows and
Mac. Follow the relevant guide below for your operating system.
1. [Creating SSH key pairs on Windows](creating_ssh_key_pairs_on_windows/)
2. [Creating SSH key pairs on Mac](creating_ssh_key_pairs_on_mac/)
3. **Connect to the Anvil VPN**
The Anvil web portal is accessible from the Internet. On the other
......@@ -120,60 +119,60 @@ precious or irreproducible data should not be placed or left on Anvil**.
accessible from the Internet. In order to access the instances from
on and off-campus, you will need to first be connected to the Anvil
VPN. Follow the instructions below to connect.
1. [Connecting to the Anvil VPN]({{< relref "connecting_to_the_anvil_vpn" >}})
1. [Connecting to the Anvil VPN](connecting_to_the_anvil_vpn/)
4. **Add the SSH Key Pair to your account**
Before creating your first instance, you'll need to associate the
SSH key created in step 2 with your account. Follow the guide
SSH key created in step 2 with your account. Follow the guide
below to login to the web dashboard and add the key pair.
1. [Adding SSH Key Pairs]({{< relref "adding_ssh_key_pairs" >}})
1. [Adding SSH Key Pairs](adding_ssh_key_pairs/)
5. **Create an instance**
Once the setup steps above are completed, you can create an
instance within the web dashboard. Follow the guide below to create
instance within the web dashboard. Follow the guide below to create
an instance.
1. [Creating an Instance]({{< relref "creating_an_instance" >}})
1. [Creating an Instance](creating_an_instance/)
6. **Connect to your instance**
After an instance has been created, you can connect (login) and
begin to use it. Connecting is done via SSH or X2Go for Linux
begin to use it. Connecting is done via SSH or X2Go for Linux
instances and via Remote Desktop (RDP) for Windows instances.
Follow the relevant guide below for your instance and the type of
Follow the relevant guide below for your instance and the type of
OS you're connecting from.
1. [Connecting to Windows Instances]({{< relref "connecting_to_windows_instances" >}})
2. [Connecting to Linux Instances via SSH from Mac]({{< relref "connecting_to_linux_instances_from_mac" >}})
3. [Connecting to Linux instances via SSH from Windows]({{< relref "connecting_to_linux_instances_from_windows" >}})
4. [Connecting to Linux instances using X2Go (for images with Xfce)]({{< relref "connecting_to_linux_instances_using_x2go" >}})
1. [Connecting to Windows Instances](connecting_to_windows_instances/)
2. [Connecting to Linux Instances via SSH from Mac](connecting_to_linux_instances_from_mac/)
3. [Connecting to Linux instances via SSH from Windows](connecting_to_linux_instances_from_windows/)
4. [Connecting to Linux instances using X2Go (for images with Xfce)](connecting_to_linux_instances_using_x2go/)
7. **Create and attach a volume to your instance (optional)**
Volumes are a means within OpenStack for persistent storage. When
Volumes are a means within OpenStack for persistent storage. When
an instance is destroyed, all data that was placed on the OS hard
drive is lost. A volume can be thought of similar to an external
hard drive. It can be attached and detached from an instance as
needed. Data on the volume will persist until the volume itself is
destroyed. Creating a volume is an optional step, but may be useful
in certain cases. The process of creating and attaching a volume
drive is lost. A volume can be thought of similar to an external
hard drive. It can be attached and detached from an instance as
needed. Data on the volume will persist until the volume itself is
destroyed. Creating a volume is an optional step, but may be useful
in certain cases. The process of creating and attaching a volume
from the web dashboard is the same regardless of the type (Linux or
Windows) of instance it will be attached to. Once the volume is
Windows) of instance it will be attached to. Once the volume is
attached, follow the corresponding guide for your instance's OS to
format and make the volume usable within your instance.
1. [Creating and attaching a volume]({{< relref "creating_and_attaching_a_volume" >}})
2. [Formatting and mounting a volume in Windows]({{< relref "formatting_and_mounting_a_volume_in_windows" >}})
3. [Formatting and mounting a volume in Linux]({{< relref "formatting_and_mounting_a_volume_in_linux" >}})
1. [Creating and attaching a volume](creating_and_attaching_a_volume/)
2. [Formatting and mounting a volume in Windows](formatting_and_mounting_a_volume_in_windows/)
3. [Formatting and mounting a volume in Linux](formatting_and_mounting_a_volume_in_linux/)
8. **Transferring files to or from your instance (optional)**
Transferring files to or from an instance is similar to doing so
with a personal laptop or workstation. To transfer between an
instance and another HCC resource, both SCP and [Globus
Connect]({{< relref "guides/handling_data/globus_connect" >}}) can be used. For transferring
with a personal laptop or workstation. To transfer between an
instance and another HCC resource, both SCP and [Globus Connect](/handling_data/data_transfer/globus_connect/)
can be used. For transferring
between an instance and a laptop/workstation or another instance,
standard file sharing utilities such as Dropbox or Box can be used.
Globus may also be used, with one stipulation. In order to
Globus may also be used, with one stipulation. In order to
transfer files between two personal endpoints, a Globus Plus
subscription is required. As part of HCC's Globus Provider Plan,
HCC can provide this on a per-user basis free of charge. If you are
subscription is required. As part of HCC's Globus Provider Plan,
HCC can provide this on a per-user basis free of charge. If you are
interested in Globus Plus, please email
{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)
[hcc-support@unl.edu](mailto:hcc-support@unl.edu)
with your request and a brief explanation.
## Backups
......@@ -187,10 +186,11 @@ the down arrow next to the button “Edit Volume” of the volume you want
to make a snapshot, then, select “Create Snapshot”.
Please note the backup function is for disaster recovery use only. HCC
is unable to restore single files within instances. Further, HCC's
is unable to restore single files within instances. Further, HCC's
disaster recovery backups should not be the only source of backups for
important data. The backup policies are subject to change without prior
notice. To retrieve your backups, please contact HCC. If you have
special concerns please contact us at
{{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu).
[hcc-support@unl.edu](mailto:hcc-support@unl.edu).
+++
title = "Resizing an instance"
description = "How to change the size (flavor) of an existing instance."
+++
---
title: Resizing an instance
summary: "How to change the size (flavor) of an existing instance."
---
It is possible to resize an existing instance. To do so, please follow the
steps below.
{{% notice info %}}
!!! note
An instance can only be resized to a flavor with a disk size equal or
larger to the existing one. To use a flavor with a smaller disk size,
you will need to create a new instance.
{{% /notice %}}
---
......@@ -26,7 +26,7 @@ you will need to create a new instance.
3. Click *Instances* in the menu on the left and locate your
instance.
{{< figure src="/images/35325991.png" height="400" >}}
<img src="/images/35325991.png" height="400">
4. If you have important data in your instance and want to protect
against possible data loss, take a snapshot of the image.
......@@ -34,33 +34,33 @@ you will need to create a new instance.
1. To take a snapshot, clicking the *Create Snapshot* button on the
right hand side of your image in the list.
{{< figure src="/images/35325992.png" width="850" >}}
<img src="/images/35325992.png" width="850">
2. Enter an identifiable name for your snapshot and click *Create Snapshot*.
{{< figure src="/images/35325993.png" heigth="150" >}}
<img src="/images/35325993.png" heigth="150">
3. You will be taken to the *Images* page, where your snapshot will
be Queued. Wait until the image has finished saving, as
indicated by the Status changing to *Active*
{{< figure src="/images/35326002.png" width="850" >}}
<img src="/images/35326002.png" width="850">
4. Return to the Instance list by clicking *Instances* in the left
hand menu.
5. Click the down arrow next to the *Create Snapshot* button and click
*Resize Instance* in the menu.
{{< figure src="/images/35326003.png" width="200" >}}
<img src="/images/35326003.png" width="200">
6. Select the desired New Flavor and click the *Resize* button.
{{< figure src="/images/35326004.png" width="550" >}}
<img src="/images/35326004.png" width="550">
7. Wait until the instance is resized as indicated by the status
"Confirm or Revert Resize/Migrate". Click the *Confirm
Resize/Migrate* button to finalize the resize.
{{< figure src="/images/35326006.png" width="850" >}}
<img src="/images/35326006.png" width="850">
At this point, your instance should have a status of *Active*. Connect
to your instance and ensure that everything works as expected.
{{% notice warning %}}
If your instance ends in an "Error" state or you have any issues or questions with resizing, please contact us at [hcc-support@unl.edu] (mailto:hcc-support@unl.edu) for assistance.{{% /notice %}}
!!! warning
If your instance ends in an "Error" state or you have any issues or questions with resizing, please contact us at [hcc-support@unl.edu](mailto:hcc-support@unl.edu) for assistance.
---
title: Using MySQL instances
summary: "How to connect to and use MySQL-based instances"
---
!!! note
This guide assumes you associated your SSH Key Pair with the instance
when it was created, and that you are connected to the [Anvil VPN](../connecting_to_the_anvil_vpn/).
The MySQL images HCC provides are CentOS 7 based with MySQL/MariaDB installed
and configured to allow remote access for a MySQL user named `root`.
Access to MySQL is provided via this `root` user and a random password. When
the instance is created, the password is set randomly using your
SSH Key Pair. This password can be retrieved via the Dashboard web
interface, and is then used to login to MySQL.
!!! note
The `root` user is for connecting to MySQL only. Access via SSH is the same
as other CentOS-based images, using the `centos` user and SSH keys. Refer to
the instructions for [Windows](../connecting_to_linux_instances_from_windows/)
or [Mac](../connecting_to_linux_instances_from_mac/) as needed.
This guide assumes you have already created an instance using the `MariaDB Server` image.
In order to access the database, two additional steps are required. A
_Security Group_ must be created in OpenStack to allow access to the MySQL port (3306),
and the randomly generated password must be recovered.
### Create and attach a security group
Log into the Anvil web dashboard at [anvil.unl.edu](https://anvil.unl.edu) using
your HCC credentials. On the left-hand side navigation menu,
click *Access & Security*.
<img src="/images/13599031.png">
The *Security Groups* tab should already be selected. Click the *Create Security Group* button
in the upper right.
<img src="/images/security_groups_1.png" width="900">
A new window will open. Type in "MySQL" for the *Name* and then click *Create Security Group* to
save and close the window.
<img src="/images/security_groups_2.png" width="600">
Next, click *Manage Rules* for the newly created *MySQL* group.
<img src="/images/security_groups_3.png" width="900">
The details page for the group will open. Click the *Add Rule* button in the upper right.
<img src="/images/security_groups_4.png" width="900">
In the new window, open the *Rule* drop-down menu and select *MYSQL*. Click *Add* to finalize
the selection and close the window.
<img src="/images/security_groups_5.png" width="600">
You should now be returned to the details page for the security group. Verify that the
MYSQL rule is listed.
<img src="/images/security_groups_6.png" width="900">
Click the *Instances* button on the left, and locate your MySQL Server instance in the
list. Open the drop-down menu on the far right, and choose *Edit Security Groups*.
<img src="/images/security_groups_7.png" width="150">
A new window will open with the available security groups on the left, and the active ones
for the instance on the right. Click the `+` sign next to the *MySQL* group to add it to
your instance.
<img src="/images/security_groups_8.png" width="600">
It will move from the left side to the right.
Now click *Save* to apply the changes and close the window.
<img src="/images/security_groups_9.png" width="600">
You can now connect to the MySQL server in your instance.
### Recover the MySQL password
Once your instance is running, login to the Anvil web dashboard
at [anvil.unl.edu](https://anvil.unl.edu) and click the *Instances* menu option on left-hand side.
You should see an entry for your instance similar to the following:
<img src="/images/sql_server_1.png" width="900">
Click the down arrow next to *Create Snapshot* to open the drop-down
menu and select *Retrieve Password*:
<img src="/images/13042846.png" height="400">
This will open a new pop-up window where you will need to select
your **private** SSH key file. Click the *Choose File* button to open a
file explorer window.
<img src="/images/13042857.png" width="600">
Navigate to your private key file and choose to open the file. The
large text box should now have the contents of your private key. Click
the *Decrypt Password* button:
<img src="/images/13042860.png" width="600">
The randomly generated password should appear in the *Password* field.
<img src="/images/13042862.png" width="600">
Copy and paste this password into a convenient text editor.
### Connecting to MySQL
Determine the IP address of your instance by looking at the fourth
column entry on the *Instances* page:
<img src="/images/sql_server_2.png" width="900">
Using the username `root` along with the recovered password and the IP
address, you can now connect to MySQL using the program of your choice.
By default, MySQL runs on port 3306.
For example, using the command line client:
```bash
[demo20@login ~]$ mysql -u root -h 10.71.104.142 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.5.65-MariaDB MariaDB Server
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
```
### Using a volume to store the database
By default, the database is stored on the instance's virtual disk. The amount of space
available may not be sufficient for larger databases, so a _Volume_ may be used instead.
All of the following setup commands need to be run as `root` in the instance after
connecting via SSH from either [Windows](../connecting_to_linux_instances_from_windows/)
or [Mac](../connecting_to_linux_instances_from_mac/).
!!! danger "**Running commands as root**"
In order to use a volume, first follow the instructions to
[create and attach a volume](../creating_and_attaching_a_volume/) to your
instance MySQL instance. Next, proceed with [formatting and mounting the volume](../formatting_and_mounting_a_volume_in_linux/)
in your instance. If all the steps complete without errors, you should have your volume mounted at `/mnt/myvolume`.
Next, the MySQL server needs to be stopped and the existing contents of its storage directory copied to the volume:
```bash
service mysql stop
rsync -av /var/lib/mysql/ /mnt/myvolume/
```
Now the volume can be unmounted from `/mnt/myvolume` and remounted over the existing `/var/lib/mysql` location:
```bash
umount /mnt/myvolume
mount /dev/vdb1 /var/lib/mysql
```
Running the `df -h` command should show the `/var/lib/mysql` location as mounted from `/dev/vdb1` with the
increased storage amount:
```bash
/dev/vdb1 99G 183M 94G 1% /var/lib/mysql
```
In this example the volume size is 100GB. In general, it should be approximately the same size as your volume.
Finally, restart the MySQL server:
```bash
service mysql start
```
Assuming the `service` command doesn't return an error, the increased space is now available for use.
The last step is to ensure the volume gets mounted automatically after the instance is rebooted. To
enable this, add the following line to the `/etc/fstab` file:
```bash
/dev/vdb1 /var/lib/mysql ext4 defaults 0 0
```
+++
title = "What are the per-group resources limit?"
description = "Group resource limits for Anvil"
+++
---
title: What are the per-group resources limit?
summary: "Group resource limits for Anvil"
---
Any HCC research group may request access to Anvil by filling out
......@@ -25,7 +25,7 @@ The resources above are provided at no charge.  If a group requires more
resources, that can be accommodated on a fee basis, depending on the
amount needed. Please see the HCC
[Priority Access Pricing](http://hcc.unl.edu/priority-access-pricing) page for specific costs.
*By default, no public IP addresses are provided.*
Please contact {{< icon name="envelope" >}}[hcc-support@unl.edu] (mailto:hcc-support@unl.edu)
*No public IP addresses are provided and VPN use is required to access Anvil instances.*
Please contact [hcc-support@unl.edu](mailto:hcc-support@unl.edu)
for more details.
title: Running Applications
preserve_directory_names: false
+++
title = "BLAST with Allinea Performance Reports"
description = "Example of how to profile BLAST using Allinea Performance Reports."
+++
---
title: BLAST with Allinea Performance Reports
summary: "Example of how to profile BLAST using Allinea Performance Reports."
---
Simple example of using
[BLAST]({{< relref "/guides/running_applications/bioinformatics_tools/alignment_tools/blast/running_blast_alignment" >}})
[BLAST](/applications/app_specific/bioinformatics_tools/alignment_tools/blast/running_blast_alignment)
with Allinea Performance Reports (`perf-report`) on Crane is shown below:
{{% panel theme="info" header="blastn_perf_report.submit" %}}
{{< highlight batch >}}
#!/bin/sh
!!! note "blastn_perf_report.submit"
```bat
#!/bin/bash
#SBATCH --job-name=BlastN
#SBATCH --nodes=1
#SBATCH --ntasks=16
......@@ -30,8 +30,8 @@ perf-report --openmp-threads=$SLURM_NTASKS_PER_NODE --nompi `which blastn` \
blastn_output.alignments -num_threads $SLURM_NTASKS_PER_NODE
cp blastn\_output.alignments .
{{< /highlight >}}
{{% /panel %}}
```
BLAST uses OpenMP and therefore the Allinea Performance Reports options
`--openmp-threads` and `--nompi` are used. The perf-report
......@@ -39,20 +39,20 @@ part, `perf-report --openmp-threads=$SLURM_NTASKS_PER_NODE --nompi`,
is placed in front of the actual `blastn` command we want
to analyze.
{{% notice info %}}
!!! note
If you see the error "**Allinea Performance Reports - target file
'application' does not exist on this machine... exiting**", this means
that instead of just using the executable '*application*', the full path
to that application is required. This is the reason why in the script
above, instead of using "*blastn*", we use *\`which blastn\`* which
gives the full path of the *blastn* executable.
{{% /notice %}}
When the application finishes, the performance report is generated in
the working directory.
For the executed application, this is how the report looks like:
{{< figure src="/images/11635296.png" width="850" >}}
<img src="/images/11635296.png" width="850">
From the report, we can see that **blastn** is Compute-Bound
application. The difference between mean (11.1 GB) and peak (26.3 GB)
......
+++
title = "Allinea Performance Reports"
description = "How to use Allinea Performance Reports to profile application on HCC resources."
+++
---
title: Allinea Performance Reports
summary: "How to use Allinea Performance Reports to profile application on HCC resources."
---
[Allinea Performance Reports](https://www.arm.com/products/development-tools/server-and-hpc/performance-reports)
is a performance evaluation tool that provides a scalable and effective way
......@@ -30,9 +30,9 @@ licenses** that can be used to evaluate applications executed on the clusters.
In order to use Allinea Performance Reports on HCC, the appropriate
module needs to be loaded first. To load the module on, use
{{< highlight bash >}}
```bash
module load allinea/5.0
{{< /highlight >}}
```
Once the module is loaded, Allinea Performance Reports runs by adding
the `perf-report` command in front of the standard application
......@@ -42,51 +42,51 @@ command.
The basic usage of ``perf-report` is:
{{% panel theme="info" header="perf-report usage" %}}
{{< highlight bash >}}
!!! note "perf-report usage"
```bash
perf-report [OPTION...] PROGRAM [PROGRAM_ARGS]
or
perf-report [OPTION...](mpirun|mpiexec|aprun|...) [MPI_ARGS] PROGRAM [PROGRAM_ARGS]
{{< /highlight >}}
{{% /panel %}}
```
For example, the command below shows how to run `perf-report` with the
application `hello_world`:
{{% panel theme="info" header="perf-report example" %}}
{{< highlight bash >}}
[<username>@login.crane ~]$ perf-report ./hello-world
{{< /highlight >}}
{{% /panel %}}
!!! note "perf-report example"
```bash
[<username>@login.swan ~]$ perf-report ./hello-world
```
Stdin redirection
{{% notice info %}}
!!! note
If your program normally uses the '`<`' syntax to redirect standard in
to read from a file, you must use the `--input` option to the
`perf-report `command instead.
{{% /notice %}}
{{% panel theme="info" header="perf-report stdin redirection" %}}
{{< highlight bash >}}
[<username>@login.crane ~]$ perf-report --input=my_input.txt ./hello-world
{{< /highlight >}}
{{% /panel %}}
!!! note "perf-report stdin redirection"
```bash
[<username>@login.swan ~]$ perf-report --input=my_input.txt ./hello-world
```
### Allinea Performance Reports Options
More **perf-report** options can be seen by using:
{{% panel theme="info" header="perf-report options" %}}
{{< highlight bash >}}
[<username>@login.crane ~]$ perf-report --help
{{< /highlight >}}
{{% /panel %}}
!!! note "perf-report options"
```bash
[<username>@login.swan ~]$ perf-report --help
```
Some of the most useful options are:
{{% panel theme="info" header="perf-report useful options" %}}
{{< highlight bash >}}
!!! note "perf-report useful options"
```bash
--input=FILE (pass the contents of FILE to the target's stdin)
--nompi, --no-mpi (run without MPI support)
--mpiargs=ARGUMENTS (command line arguments to pass to mpirun)
......@@ -94,21 +94,21 @@ Some of the most useful options are:
--openmp-threads=NUMTHREADS (configure the number of OpenMP threads for the target)
-n, --np, --processes=NUMPROCS (specify the number of MPI processes)
--procs-per-node=PROCS (configure the number of processes per node for MPI jobs)
{{< /highlight >}}
{{% /panel %}}
```
The following pages
- [Blast with Allinea Performance Reports]({{< relref "blast_with_allinea_performance_reports" >}})
- [Ray with Allinea Performance Reports]({{< relref "ray_with_allinea_performance_reports" >}})
- [LAMMPS with Allinea Performance Reports]({{< relref "lammps_with_allinea_performance_reports" >}})
- [Blast with Allinea Performance Reports](blast_with_allinea_performance_reports)
- [Ray with Allinea Performance Reports](ray_with_allinea_performance_reports)
- [LAMMPS with Allinea Performance Reports](lammps_with_allinea_performance_reports)
show how to run Allinea Performance Reports with applications using OpenMP, MPI and standard input/output
respectively.
{{% notice tip %}}
!!! tip
Currently, Allinea Performance Reports works best with compiled
binaries of an application (for some perl/python files perf-report needs
to be added in the actual file).
{{% /notice %}}
+++
title = "LAMMPS with Allinea Performance Reports"
description = "Example of how to profile LAMMPS using Allinea Performance Reports."
+++
---
title: LAMMPS with Allinea Performance Reports
summary: "Example of how to profile LAMMPS using Allinea Performance Reports."
---
Simple example of using [LAAMPS](http://lammps.sandia.gov)
with Allinea Performance Reports (`perf-report`) on Crane is shown
below:
{{% panel theme="info" header="lammps_perf_report.submit" %}}
{{< highlight batch >}}
#!/bin/sh
!!! note "lammps_perf_report.submit"
```bat
#!/bin/bash
#SBATCH --job-name=LAMMPS
#SBATCH --ntasks=64
#SBATCH --time=12:00:00
......@@ -22,9 +22,9 @@ module load allinea
module load compiler/gcc/4.8 openmpi/1.8
module load lammps/30Oct2014
perf-report --input=in.copper --np=64 `which lmp_ompi_g++`
{{< /highlight >}}
{{% /panel %}}
perf-report --input=in.copper --np=64 `which lmp_ompi_g---`
```
LAMMPS runs on a single processor or in parallel using message-passing
interface, and therefore additional Allinea Performance Reports options
......@@ -35,20 +35,20 @@ defined. Therefore, in the perf-report part we have `perf-report
--input=in.copper`, where `in.copper` is the input file used for
LAMMPS.
{{% notice info %}}
!!! note
If you see the error "**Allinea Performance Reports - target file
'application' does not exist on this machine... exiting**", this means
that instead of just using the executable '*application*', the full path
to that application is required. This is the reason why in the script
above, instead of using "*lmp\_ompi\_g++*", we use *\`which
lmp\_ompi\_g++\`* which gives the full path of the *LAMMPS* executable.
{{% /notice %}}
above, instead of using "*lmp\_ompi\_g---*", we use *\`which
lmp\_ompi\_g---\`* which gives the full path of the *LAMMPS* executable.
When the application finishes, the performance report is generated in
the working directory.
For the executed application, this is how the report looks like:
{{< figure src="/images/11635309.png" width="850" >}}
<img src="/images/11635309.png" width="850">
From the report, we can see that **LAMMPS **is Compute-Bound
application. Most of the MPI communication is spent in collective calls
......
+++
title = "Ray with Allinea Performance Reports"
description = "Example of how to profile Ray using Allinea Performance Reports"
+++
---
title: Ray with Allinea Performance Reports
summary: "Example of how to profile Ray using Allinea Performance Reports"
---
Simple example of using [Ray]({{< relref "/guides/running_applications/bioinformatics_tools/de_novo_assembly_tools/ray" >}})
with Allinea PerformanceReports (`perf-report`) on Tusker is shown below:
Simple example of using [Ray](/applications/app_specific/bioinformatics_tools/de_novo_assembly_tools/ray)
with Allinea PerformanceReports (`perf-report`) is shown below:
{{% panel theme="info" header="ray_perf_report.submit" %}}
{{< highlight batch >}}
#!/bin/sh
!!! note "ray_perf_report.submit"
```bat
#!/bin/bash
#SBATCH --job-name=Ray
#SBATCH --ntasks-per-node=16
#SBATCH --time=10:00:00
......@@ -20,8 +20,8 @@ module load allinea
module load compiler/gcc/4.7 openmpi/1.6 ray/2.3
perf-report mpiexec -n 16 Ray -k 31 -p -p input_reads_pair_1.fasta input_reads\_pair_2.fasta -o output_directory
{{< /highlight >}}
{{% /panel %}}
```
Ray is MPI and therefore additional Allinea Performance Reports options
are not required. The `perf-report` command is placed in front of the
......@@ -31,7 +31,7 @@ When the application finishes, the performance report is generated in
the working directory.
For the executed application, this is how the report looks like:
{{< figure src="/images/11635303.png" width="850" >}}
<img src="/images/11635303.png" width="850">
From the report, we can see that **Ray **is Compute-Bound application.
Most of the running time is spent in point-to-point calls with a low
......
---
title: Allinea Profiling & Debugging Tools
summary: "How to use the Allinea suite of tools for profiling and debugging."
---
HCC provides both the Allinea Forge suite and Performance Reports to
assist with debugging and profiling C/C---/Fortran code. These tools
support single-threaded, multi-threaded (pthreads/OpenMP), MPI, and CUDA
code. The Allinea Forge suite consists of two programs: DDT for
debugging and MAP for profiling. The Performance Reports software
provides a convenient way to profile HPC applications. It generates an
easy-to-read single-page HTML report.
For information on using each tool, see the following pages.
[Using Allinea Forge via Reverse Connect](using_allinea_forge_via_reverse_connect)
[Allinea Performance Reports](allinea_performance_reports)
#### External links
The following external links may also be useful:
[Using Arm Forge when working remotely](https://community.arm.com/arm-community-blogs/b/high-performance-computing-blog/posts/debugging-profiling-hpc-applications-working-remotely)
[Getting started videos](https://developer.arm.com/tools-and-software/server-and-hpc/arm-architecture-tools/arm-forge/videos)
[Offline debugging](https://community.arm.com/developer/tools-software/hpc/b/hpc-blog/posts/debugging-while-you-sleep)
[More offine debugging](https://community.arm.com/developer/tools-software/hpc/b/hpc-blog/posts/more-debugging-while-you-sleep-with-ddt)
[Python debugging](https://developer.arm.com/documentation/101136/2102/DDT/Get-started-with-DDT/Python-debugging)
[Python profiling](https://developer.arm.com/documentation/101136/2102/MAP/Python-profiling)
+++
title = "Using Allinea Forge via Reverse Connect"
description = "How to use the Reverse Connect feature of Allinea Forge."
+++
---
title: Using Allinea Forge via Reverse Connect
summary: "How to use the Reverse Connect feature of Allinea Forge."
---
### Setup the Allinea client software to use Reverse Connect
......@@ -16,93 +16,81 @@ these instructions to setup the Allinea client on your laptop.
First, download and install the remote client software for either
Windows or OS X from
[this page](https://developer.arm.com/products/software-development-tools/hpc/downloads/download-arm-forge#remote-client).
[this page](https://developer.arm.com/downloads/-/arm-forge).
Alternatively, download the software directly for your OS:
[[OS X direct link]](http://content.allinea.com/downloads/arm-forge-client-latest-MacOSX-10.7.5-x86_64.dmg)
[[Windows 64-bit direct link]](http://content.allinea.com/downloads/arm-forge-client-latest-Windows-10.0-x64.exe)
[[OS X direct link]](https://content.allinea.com/downloads/arm-forge-client-22.1-macos-x86_64.dmg)
[[Windows 64-bit direct link]](https://content.allinea.com/downloads/arm-forge-client-22.1-windows-x86_64.exe)
Start the Allinea software, and choose *Configure...* from the *Remote
Launch* dropdown menu.
{{< figure src="/images/16516460.png" width="300" >}}
<img src="/images/16516460.png" width="300">
Click the *Add* button on the new window.
{{< figure src="/images/16516459.png" width="400" >}}
<img src="/images/16516459.png" width="400">
To setup a connection to Crane, fill in the fields as follows:
*Connection Name:* Crane
*Host Name:* \<username\>@crane.unl.edu
*Remote Installation Directory:* /util/opt/allinea/18.2
To setup a connection to Swan, fill in the fields as follows:
```
Connection Name: Swan
Host Name: <username>@swan.unl.edu
Remote Installation Directory: /util/opt/allinea/22.0
```
It should appear similar to this:
{{< figure src="/images/16519633.png" width="500" >}}
<img src="/images/16519633.png" width="500">
Be sure to replace *demo02* with your HCC username.
Be sure to replace `<username>` with your HCC username.
Click *OK* to close this dialog, and then *Close* on *Configure Remote
Connections* to return back to the main Allinea window.
Next, log in to Crane. The Allinea software uses a `.allinea` directory
in your home directory to store configuration information. Since `/home`
is read-only from the nodes in the cluster, the directory will be
created in `/work` and symlink'd. To do so, run the following commands:
{{% panel theme="info" header="Create and symlink .allinea directory" %}}
{{< highlight bash >}}
rm -rf $HOME/.allinea
mkdir -p $WORK/.allinea
ln -s $WORK/.allinea $HOME/.allinea
{{< /highlight >}}
{{% /panel %}}
### Test the Reverse Connect feature
To test the connection, choose *Crane* from the *Remote Launch* menu.
To test the connection, choose *Swan* from the *Remote Launch* menu.
{{< figure src="/images/16516457.png" width="300" >}}
<img src="/images/16516457.png" width="300">
A *Connect to Remote Host* dialog will appear and prompt for a password.
{{< figure src="/images/16516456.png" width="500" >}}
<img src="/images/16516456.png" width="500">
The login procedure is the same as for PuTTY or any other SSH program.
Enter your HCC password followed by the Duo login.
If the login was successful, you should see
*Connected to: \<username\>@crane.unl.edu* in the lower right corner of
*Connected to: \<username\>@swan.unl.edu* in the lower right corner of
the Allinea window.
The next step is to run a sample interactive job and test the Reverse
Connect connection. Start an interactive job by running the following
command.
{{% panel theme="info" header="Start an interactive job" %}}
{{< highlight bash >}}
!!! note "Start an interactive job"
```bash
srun --pty --qos=short bash
{{< /highlight >}}
{{% /panel %}}
```
Once the job has started, load the allinea module and start DDT using
the `--connect` option.
{{% panel theme="info" header="Start DDT" %}}
{{< highlight bash >}}
!!! note "Start DDT"
```bash
module load allinea
ddt --connect
{{< /highlight >}}
{{% /panel %}}
```
On your local machine, a pop-up box should appear prompting you to
accept the Reverse Connect request.
{{< figure src="/images/16516453.png" width="450" >}}
<img src="/images/16516453.png" width="450">
Choose *Accept.* The *Remote Launch* section should change to indicate
you are connected via tunnel, similar to:
{{< figure src="/images/16516455.png" width="250" >}}
<img src="/images/16516455.png" width="250">
Once that happens, Reverse Connect is working successfully and a
debugging or profiling session can be started.
......@@ -113,26 +101,26 @@ The `srun `syntax is slightly different depending on which type of code
you are debugging. Use the following commands to start interactive jobs
for each type.
{{% panel theme="info" header="Serial code" %}}
{{< highlight bash >}}
!!! note "Serial code"
```bash
srun --pty --time=2:00:00 bash
{{< /highlight >}}
{{% /panel %}}
```
{{% panel theme="info" header="OpenMP/pthreads code" %}}
{{< highlight bash >}}
!!! note "OpenMP/pthreads code"
```bash
srun --pty --time=2:00:00 --nodes=1 --ntasks-per-node=4 bash
{{< /highlight >}}
{{% /panel %}}
```
{{% panel theme="info" header="MPI code" %}}
{{< highlight bash >}}
!!! note "MPI code"
```bash
srun --pty --time=2:00:00 --ntasks=4 bash
{{< /highlight >}}
{{% /panel %}}
```
{{% panel theme="info" header="CUDA code" %}}
{{< highlight bash >}}
!!! note "CUDA code"
```bash
srun --pty --time=2:00:00 --partition=gpu --gres=gpu bash
{{< /highlight >}}
{{% /panel %}}
```
+++
title = "Create Local BLAST Database"
description = "How to create local BLAST database on HCC resources"
weight = "10"
+++
---
title: Create Local BLAST Database
summary: "How to create local BLAST database on HCC resources"
---
The basic way to create a local BLAST database is to use the **makeblastdb** command:
{{< highlight bash >}}
```bash
$ makeblastdb -in input_reads.fasta -dbtype [nucl|prot] -out input_reads_db
{{< /highlight >}}
```
where **input_reads.fasta** is the input file containing all sequences that need to be made into a database, and **dbtype** can be either `nucl` or `prot` depending on the type of the input file.
Simple example of how **makeblastdb** can be run on Crane using SLURM script and nucleotide database is shown below:
{{% panel header="`blast_db.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
Simple example of how **makeblastdb** can be run on Swan using SLURM script and nucleotide database is shown below:
!!! note ""
```bash
#!/bin/bash
#SBATCH --job-name=Blast_DB
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
......@@ -24,14 +23,14 @@ Simple example of how **makeblastdb** can be run on Crane using SLURM script and
#SBATCH --output=BlastDB.%J.out
#SBATCH --error=BlastDB.%J.err
module load blast/2.7
module load blast/2.10
makeblastdb -in input_reads.fasta -dbtype nucl -out input_reads_db
{{< /highlight >}}
{{% /panel %}}
```
More parameters used with **makeblastdb** can be seen by typing:
{{< highlight bash >}}
```bash
$ makeblastdb -help
{{< /highlight >}}
```
---
title: BLAST
summary: "How to use BLAST on HCC machines"
---
[BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi) is a local alignment tool that finds similarity between sequences. This tool compares nucleotide or protein sequences to sequence databases, and calculates significance of matches. Sometimes these input sequences are large and using the command-line BLAST is required.
The following pages, [Create Local BLAST Database](create_local_blast_database) and [Running BLAST Alignment](running_blast_alignment) describe how to run some of the most common BLAST executables as a single job using the SLURM scheduler on HCC.
### Useful Information
In order to test the BLAST (blast/2.2) performance on Swan, we aligned three nucleotide query datasets, `small.fasta`, `medium.fasta` and `large.fasta`, against the non-redundant nucleotide **nt.fasta** database from NCBI. Some statistics about the query datasets and the time and memory resources used for the alignment are shown on the table below:
{%
include "../../../../../static/html/blast.html"
%}
+++
title = " Running BLAST Alignment"
description = "How to run BLAST alignment on HCC resources"
weight = "10"
+++
---
title: Running BLAST Alignment
summary: "How to run BLAST alignment on HCC resources"
---
Basic BLAST has the following commands:
......@@ -15,53 +14,43 @@ Basic BLAST has the following commands:
The basic usage of **blastn** is:
{{< highlight bash >}}
```bash
$ blastn -query input_reads.fasta -db input_reads_db -out blastn_output.alignments [options]
{{< /highlight >}}
```
where **input_reads.fasta** is an input file of sequence data in fasta format, **input_reads_db** is the generated BLAST database, and **blastn_output.alignments** is the output file where the alignments are stored.
Additional parameters can be found in the [BLAST manual](https://www.ncbi.nlm.nih.gov/books/NBK279690/), or by typing:
{{< highlight bash >}}
```bash
$ blastn -help
{{< /highlight >}}
```
These BLAST alignment commands are multi-threaded, and therefore using the BLAST option **-num_threads <number_of_CPUs>** is recommended.
HCC hosts multiple BLAST databases and indices on Crane. In order to use these resources, the ["biodata" module] ({{<relref "/guides/running_applications/bioinformatics_tools/biodata_module">}}) needs to be loaded first. The **$BLAST** variable contains the following currently available databases:
HCC hosts multiple BLAST databases and indices on Swan. In order to use these resources, the ["biodata" module](/applications/app_specific/bioinformatics_tools/biodata_module/) needs to be loaded first. The **$BLAST** variable contains the following currently available databases:
- **16SMicrobial**
- **env_nt**
- **est**
- **est_human**
- **est_mouse**
- **est_others**
- **gss**
- **human_genomic**
- **human_genomic_transcript**
- **mouse_genomic_transcript**
- **nr**
- **nt**
- **other_genomic**
- **refseq_genomic**
- **refseq_rna**
- **sts**
- **swissprot**
- **tsa_nr**
- **tsa_nt**
If you want to create and use a BLAST database that is not mentioned above, check [Create Local BLAST Database]({{<relref "create_local_blast_database" >}}).
If you want to create and use a BLAST database that is not mentioned above, check [Create Local BLAST Database](../create_local_blast_database/). If you want a database to be added to the ["biodata" module](/applications/app_specific/bioinformatics_tools/biodata_module/), please send a request to bcrf-support@unl.edu.
!!! note
**To access the older format of BLAST databases that work with BLAST+ 2.9 and lower, please use the variable BLAST_V4.**
**The variable BLAST points to the directory with the new version 5 of the nucleotide and protein databases required for BLAST+ 2.10 and higher.**
Basic SLURM example of nucleotide BLAST run against the non-redundant **nt** BLAST database with `8 CPUs` is provided below. When running BLAST alignment, it is recommended to first copy the query and database files to the **/scratch/** directory of the worker node. Moreover, the BLAST output is also saved in this directory (**/scratch/blastn_output.alignments**). After BLAST finishes, the output file is copied from the worker node to your current work directory.
{{% notice info %}}
**Please note that the worker nodes can not write to the */home/* directories and therefore you need to run your job from your */work/* directory.**
**This example will first copy your database to faster local storage called “scratch”. This can greatly improve performance!**
{{% /notice %}}
{{% panel header="`blastn_alignment.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
!!! note
**This example will first copy the database and your input file to faster local storage called "scratch", assuming that the input file exists in your current directory. This can greatly improve performance!**
!!! note ""
```bash
#!/bin/bash
#SBATCH --job-name=BlastN
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
......@@ -70,38 +59,37 @@ Basic SLURM example of nucleotide BLAST run against the non-redundant **nt** BL
#SBATCH --output=BlastN.%J.out
#SBATCH --error=BlastN.%J.err
module load blast/2.7
module load blast/2.10
module load biodata/1.0
cd $WORK/<project_folder>
# Be sure to use a directory under $WORK for your job
cp $BLAST/nt.* /scratch/
cp input_reads.fasta /scratch/
blastn -query /scratch/input_reads.fasta -db /scratch/nt -out /scratch/blastn_output.alignments -num_threads $SLURM_NTASKS_PER_NODE
cp /scratch/blastn_output.alignments $WORK/<project_folder>
{{< /highlight >}}
{{% /panel %}}
cp /scratch/blastn_output.alignments .
```
One important BLAST parameter is the **e-value threshold** that changes the number of hits returned by showing only those with value lower than the given. To show the hits with **e-value** lower than 1e-10, modify the given script as follows:
{{< highlight bash >}}
```bash
$ blastn -query input_reads.fasta -db input_reads_db -out blastn_output.alignments -num_threads $SLURM_NTASKS_PER_NODE -evalue 1e-10
{{< /highlight >}}
```
The default BLAST output is in pairwise format. However, BLAST’s parameter **-outfmt** supports output in [different formats](https://www.ncbi.nlm.nih.gov/books/NBK279684/) that are easier for parsing.
Basic SLURM example of protein BLAST run against the non-redundant **nr **BLAST database with tabular output format and `8 CPUs` is shown below. Similarly as before, the query and database files are copied to the **/scratch/** directory. The BLAST output is also saved in this directory (**/scratch/blastx_output.alignments**). After BLAST finishes, the output file is copied from the worker node to your current work directory.
{{% notice info %}}
**Please note that the worker nodes can not write to the */home/* directories and therefore you need to run your job from your */work/* directory.**
**This example will first copy your database to faster local storage called “scratch”. This can greatly improve performance!**
{{% /notice %}}
{{% panel header="`blastx_alignment.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
!!! note
**This example will first copy the database and your input file to faster local storage called "scratch", assuming that the input file exists in your current directory. This can greatly improve performance!**
!!! note ""
```bash
#!/bin/bash
#SBATCH --job-name=BlastX
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
......@@ -110,15 +98,15 @@ Basic SLURM example of protein BLAST run against the non-redundant **nr **BLAS
#SBATCH --output=BlastX.%J.out
#SBATCH --error=BlastX.%J.err
module load blast/2.7
module load blast/2.10
module load biodata/1.0
cd $WORK/<project_folder>
# Be sure to use a directory under $WORK for your job
cp $BLAST/nr.* /scratch/
cp input_reads.fasta /scratch/
blastx -query /scratch/input_reads.fasta -db /scratch/nr -outfmt 6 -out /scratch/blastx_output.alignments -num_threads $SLURM_NTASKS_PER_NODE
cp /scratch/blastx_output.alignments $WORK/<project_folder>
{{< /highlight >}}
{{% /panel %}}
cp /scratch/blastx_output.alignments .
```
+++
title = "BLAT"
description = "How to run BLAT on HCC resources"
weight = "10"
+++
---
title: BLAT
summary: "How to run BLAT on HCC resources"
---
BLAT is a pairwise alignment tool similar to BLAST. It is more accurate and about 500 times faster than the existing tools for mRNA/DNA alignments and it is about 50 times faster with protein/protein alignments. BLAT accepts short and long query and database sequences as input files.
The basic usage of BLAT is:
{{< highlight bash >}}
```bash
$ blat database query output_alignment.txt [options]
{{< /highlight >}}
```
where **database** is the name of the database used for the alignment, **query** is the name of the input file of sequence data in `fasta/nib/2bit` format, and **output_alignment.txt** is the output alignment file.
Additional parameters for BLAT alignment can be found in the [manual](http://genome.ucsc.edu/FAQ/FAQblat), or by using:
{{< highlight bash >}}
```bash
$ blat
{{< /highlight >}}
```
Running BLAT on Crane with query file `input_reads.fasta` and database `db.fa` is shown below:
{{% panel header="`blat_alignment.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
Running BLAT on Swan with query file `input_reads.fasta` and database `db.fa` is shown below:
!!! note ""
```bash
#!/bin/bash
#SBATCH --job-name=Blat
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
......@@ -36,8 +35,8 @@ Running BLAT on Crane with query file `input_reads.fasta` and database `db.fa` i
module load blat/35x1
blat db.fa input_reads.fasta output_alignment.txt
{{< /highlight >}}
{{% /panel %}}
```
Although BLAT is a single threaded program (`#SBATCH --nodes=1`, `#SBATCH --ntasks-per-node=1`) it is still much faster than the other alignment tools.
......
+++
title = "Bowtie"
description = "How to run Bowtie on HCC resources"
weight = "10"
+++
---
title: Bowtie
summary: "How to run Bowtie on HCC resources"
---
[Bowtie](http://bowtie-bio.sourceforge.net/index.shtml) is an ultrafast and memory-efficient aligner for large sets of sequencing reads to a reference genome. Bowtie indexes the genome with a Burrows-Wheeler index to keep its memory footprint small. Bowtie also supports usage of multiple processors to achieve greater alignment speed.
The first and basic step of running Bowtie is to build and format an index from the reference genome. The basic usage of this command, **bowtie-build** is:
{{< highlight bash >}}
```bash
$ bowtie-build input_reference.fasta index_prefix
{{< /highlight >}}
```
where **input_reference.fasta** is an input file of sequence reads in fasta format, and **index_prefix** is the prefix of the generated index files.
After the index of the reference genome is generated, the next step is to align the reads. The basic usage of bowtie is:
{{< highlight bash >}}
```bash
$ bowtie [-q|-f|-r|-c] index_prefix [-1 input_reads_pair_1.[fasta|fastq] -2 input_reads_pair_2.[fasta|fastq] | input_reads.[fasta|fastq]] [options]
{{< /highlight >}}
```
where **index_prefix** is the generated index using the **bowtie-build** command, and **options** are optional parameters that can be found in the [Bowtie
manual](http://bowtie-bio.sourceforge.net/manual.shtml).
......@@ -25,10 +24,10 @@ manual] (http://bowtie-bio.sourceforge.net/manual.shtml).
Bowtie supports both single-end (`input_reads.[fasta|fastq]`) and paired-end (`input_reads_pair_1.[fasta|fastq]`, `input_reads_pair_2.[fasta|fastq]`) files in fasta or fastq format. The format of the input files also needs to be specified by using the following flags: **-q** (fastq files), **-f** (fasta files), **-r** (raw one-sequence per line), or **-c** (sequences given on command line).
An example of how to run Bowtie alignment on Crane with single-end fastq file and `8 CPUs` is shown below:
{{% panel header="`bowtie_alignment.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
An example of how to run Bowtie alignment on Swan with single-end fastq file and `8 CPUs` is shown below:
!!! note ""
```bash
#!/bin/bash
#SBATCH --job-name=Bowtie
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
......@@ -40,8 +39,8 @@ An example of how to run Bowtie alignment on Crane with single-end fastq file an
module load bowtie/1.1
bowtie -q index_prefix input_reads.fastq -p $SLURM_NTASKS_PER_NODE > bowtie_alignments.sam
{{< /highlight >}}
{{% /panel %}}
```
### Bowtie Output
......
+++
title = "Bowtie2"
description = "How to run Bowtie2 on HCC resources"
weight = "10"
+++
---
title: Bowtie2
summary: "How to run Bowtie2 on HCC resources"
---
[Bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml) is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences. Although Bowtie and Bowtie2 are both fast read aligners, there are few main differences between them:
......@@ -18,23 +17,23 @@ weight = "10"
Same as Bowtie, the first and basic step of running Bowtie2 is to build Bowtie2 index from a reference genome sequence. The basic usage of the
command **bowtie2-build** is:
{{< highlight bash >}}
```bash
$ bowtie2-build -f input_reference.fasta index_prefix
{{< /highlight >}}
```
where **input_reference.fasta** is an input file of sequence reads in fasta format, and **index_prefix** is the prefix of the generated index files. Beside the option **-f** that is used when the reference input file is a fasta file, the option **-c** can be used when the reference sequences are given on the command line.
The command **bowtie2** takes a Bowtie2 index and set of sequencing read files and outputs set of alignments in SAM format. The general **bowtie2** usage is:
{{< highlight bash >}}
```bash
$ bowtie2 -x index_prefix [-q|--qseq|-f|-r|-c] [-1 input_reads_pair_1.[fasta|fastq] -2 input_reads_pair_2.[fasta|fastq] | -U input_reads.[fasta|fastq]] -S bowtie2_alignments.sam [options]
{{< /highlight >}}
```
where **index_prefix** is the generated index using the **bowtie2-build** command, and **options** are optional parameters that can be found in the [Bowtie2 manual](http://bowtie-bio.sourceforge.net/bowtie2/manual.shtml). Bowtie2 supports both single-end (`input_reads.[fasta|fastq]`) and paired-end (`input_reads_pair_1.[fasta|fastq]`, `input_reads_pair_2.[fasta|fastq]`) files in fasta or fastq format. The format of the input files also needs to be specified by using one of the following flags: **-q** (fastq files), **--qseq** (Illumina's qseq format), **-f** (fasta files), **-r** (raw one sequence per line), or **-c** (sequences given on command line).
An example of how to run Bowtie2 local alignment on Crane with paired-end fasta files and `8 CPUs` is shown below:
{{% panel header="`bowtie2_alignment.submit`"%}}
{{< highlight bash >}}
#!/bin/sh
An example of how to run Bowtie2 local alignment on Swan with paired-end fasta files and `8 CPUs` is shown below:
!!! note ""
```bash
#!/bin/bash
#SBATCH --job-name=Bowtie2
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
......@@ -46,8 +45,8 @@ An example of how to run Bowtie2 local alignment on Crane with paired-end fasta
module load bowtie/2.3
bowtie2 -x index_prefix -f -1 input_reads_pair_1.fasta -2 input_reads_pair_2.fasta -S bowtie2_alignments.sam --local -p $SLURM_NTASKS_PER_NODE
{{< /highlight >}}
{{% /panel %}}
```
### Bowtie2 Output
......
+++
title = "BWA"
description = "How to use BWA on HCC machines"
weight = "52"
+++
---
title: BWA
summary: "How to use BWA on HCC machines"
---
BWA (Burrows-Wheeler Aligner) is a software package for mapping relatively short nucleotide sequences against a long reference sequence. BWA is slower than Bowtie, but allows indels in the alignment.
The basic usage of BWA is:
{{< highlight bash >}}
```bash
$ bwa COMMAND [options]
{{< /highlight >}}
```
where **COMMAND** is one of the available BWA commands:
- **index**: index sequences in the FASTA format
......@@ -30,10 +29,10 @@ where **COMMAND** is one of the available BWA commands:
BWA supports three alignment algorithms, **mem**, **bwasw**, and **aln**/**samse**/**sampe**. **bwa mem** is the latest algorithm, and is faster, more accurate and has better performance than **bwa bwasw** and **bwa aln**/**samse**/**sampe**. Therefore, if there are not any specific reasons, **bwa mem** is recommended for first-time users.
For detailed description and more information on a specific command, just type:
{{< highlight bash >}}
```bash
$ bwa COMMAND
{{< /highlight >}}
```
or check the [BWA manual](http://bio-bwa.sourceforge.net/bwa.shtml).
The page [Running BWA Commands]({{<relref "running_bwa_commands" >}}) shows how to run BWA on HCC.
The page [Running BWA Commands](running_bwa_commands) shows how to run BWA on HCC.