| **Crane** | 548 node Production-mode LINUX cluster | 452 Intel Xeon E5-2670 2.60GHz 2 CPU/16 cores per node<br> <br>116 Intel Xeon E5-2697 v4 2.3GHz, 2 CPU/36 cores per node<br><br>("CraneOPA") | 452 nodes @ \*64GB<br><br>79 nodes @ \*\*256GB<br><br>37 nodes @ \*\*\*512GB | QDR Infiniband<br><br>EDR Omni-Path Architecture | ~1.8 TB local scratch per node<br><br>~4 TB local scratch per node<br><br>~1452 TB shared Lustre storage
| **Tusker** | 82 node Production-mode LINUX cluster | Opteron 6272 2.1GHz, 4 CPU/64 cores per node | \*\*256 GB RAM per node<br>\*\*\*2 Nodes with 512GB per node<br>\*\*\*\*2 Nodes with 1024GB per node | QDR Infiniband | ~500 TB shared Lustre storage<br>~500GB local scratch |
| **Red** | 344 node Production-mode LINUX cluster | Various Xeon and Opteron processors 7,280 cores maximum, actual number of job slots depends on RAM usage | 1.5-4GB RAM per job slot | 1Gb, 10Gb, and 40Gb Ethernet | ~6.67PB of raw storage space |
| **Anvil** | 76 Compute nodes (Partially used for cloud, the rest used for general computing), 12 Storage nodes, 2 Network nodes Openstack cloud | 76 Intel Xeon E5-2650 v3 2.30GHz 2 CPU/20 cores per node | 76 nodes @ 256GB | 10Gb Ethernet | 528 TB Ceph shared storage (349TB available now) |
@@ -6,11 +6,13 @@ This document details the equipment resident in the Holland Computing Center (HC
HCC has two primary locations directly interconnected by a pair of 10 Gbps fiber optic links (20 Gbps total). The 1800 sq. ft. HCC machine room at the Peter Kiewit Institute (PKI) in Omaha can provide up to 500 kVA in UPS and genset protected power, and 160 ton cooling. A 2200 sq. ft. second machine room in the Schorr Center at the University of Nebraska-Lincoln (UNL) can currently provide up to 100 ton cooling with up to 400 kVA of power. One Brocade MLXe router and two Dell Z9264F-ON core switches in each location provide both high WAN bandwidth and Software Defined Networking (SDN) capability. The Schorr machine room connects to campus and Internet2/ESnet at 100 Gbps while the PKI machine room connects at 10 Gbps. HCC uses multiple data transfer nodes as well as a FIONA (flash IO network appliance) to facilitate end-to-end performance for data intensive workflows.
HCC's resources at UNL include two distinct offerings: Sandhills and Red. Sandhills is a linux cluster dedicated to general campus usage with 5,472 compute cores interconnected by low-latency InfiniBand networking. 175 TB of Lustre storage is complemented by 50 TB of NFS storage and 3 TB of local scratch per node.
HCC's resources at UNL include two distinct offerings: Sandhills and Red. Sandhills is a linux cluster dedicated to general campus usage with 5,472 compute cores interconnected by low-latency InfiniBand networking. 175 TB of Lustre storage is complemented by 50 TB of NFS storage and 3 TB of local scratch per node. Tusker offers 3,712 cores interconnected with Mellanox QDR InfiniBand along with 523TB of Lustre storage. Each compute node is a Dell R815 server with at least 256 GB RAM and 4 Opteron 6272 (2.1 GHz) processors.
The largest machine on the Lincoln campus is Red, with 9,536 job slots interconnected by a mixture of 1, 10, and 40 Gbps ethernet. More importantly, Red serves up over 6.6 PB of storage using the Hadoop Distributed File System (HDFS). Red is integrated with the Open Science Grid (OSG), and serves as a major site for storage and analysis in the international high energy physics project known as CMS (Compact Muon Solenoid).
HCC's resources at PKI (Peter Kiewit Institute) in Omaha include Tusker, Crane, Anvil, Attic, and Common storage. Tusker offers 3,712 cores interconnected with Mellanox QDR InfiniBand along with 523TB of Lustre storage. Each compute node is a Dell R815 server with at least 256 GB RAM and 4 Opteron 6272 (2.1 GHz) processors. Tusker and Sandhills are currently being retired and will be moved to the Walter Scott Engineering Center located in Lincoln consolidated into one, new Tusker cluster.
Tusker and Sandhills are currently decommissioned. These resources will be combined into a new cluster called Rhino which will be available at a future date.
HCC's resources at PKI (Peter Kiewit Institute) in Omaha include Crane, Anvil, Attic, and Common storage.
Crane debuted at 474 on the Top500 list with an HPL benchmark or 121.8 TeraFLOPS. Intel Xeon chips (8-core, 2.6 GHz) provide the processing with 4 GB RAM available per core and a total of 12,236 cores. The cluster shares 1.5 PetaBytes of Lustre storage and contains HCC's GPU resources. We have since expanded the existing cluster: 96 nodes with new Intel Xeon E5-2697 v4 chips and 100GB Intel Omni-Path interconnect were added to Crane. Moreover, Crane has 21 GPU nodes with 57 NVIDIA GPUs in total which enables the most state-of-art research, from drug discovery to deep learning.
...
...
@@ -36,7 +38,7 @@ These resources are detailed further below.
* 175TB shared scratch storage (Lustre) -> /work
* 3TB local scratch
# 1.2 Red
## 1.2 Red
* USCMS Tier-2 resource, available opportunistically via the Open Science Grid
* 60 2-socket Xeon E5530 (2.4GHz) (16 slots per node)
...
...
@@ -57,14 +59,12 @@ These resources are detailed further below.
a fast and robust file transfer service that allows users to quickly
move large amounts of data between computer clusters and even to and
from personal workstations. This service has been made available for
Tusker, Crane, and Attic. HCC users are encouraged to use Globus
Crane, and Attic. HCC users are encouraged to use Globus
Connect for their larger data transfers as an alternative to slower and
more error-prone methods such as scp and winSCP.
...
...
@@ -15,7 +15,7 @@ more error-prone methods such as scp and winSCP.
### Globus Connect Advantages
- Dedicated transfer servers on Tusker, Crane, and Attic allow
- Dedicated transfer servers on Crane, and Attic allow
large amounts of data to be transferred quickly between sites.
- A user can install Globus Connect Personal on his or her workstation
...
...
@@ -38,7 +38,7 @@ the <a href="https://www.globus.org/SignUp" class="external-link">Globus Connec
Accounts are free and grant users access to any Globus endpoint for
which they are authorized. An endpoint is simply a file system to or
from which a user transfers files. All HCC users are authorized to
access their own /home, /work, and /common directories on Tusker and Crane via the Globus endpoints (named: `hcc#tusker` and `hcc#crane`). Those who have purchased Attic storage space can
access their own /home, /work, and /common directories on Crane via the Globus endpoints (named: `hcc#crane`). Those who have purchased Attic storage space can
access their /attic directories via the Globus endpoint hcc\#attic. To
initialize or activate the endpoint, users will be required to enter
their HCC username, password, and Duo credentials for authentication.
@@ -4,13 +4,13 @@ description = "How to activate HCC endpoints on Globus"
weight = 20
+++
You will not be able to transfer files to or from an HCC endpoint using Globus Connect without first activating the endpoint. Endpoints are available for Tusker (`hcc#tusker`), Crane (`hcc#crane`), and Attic (`hcc#attic`). Follow the instructions below to activate any of these endpoints and begin making transfers.
You will not be able to transfer files to or from an HCC endpoint using Globus Connect without first activating the endpoint. Endpoints are available for Crane (`hcc#crane`), and Attic (`hcc#attic`). Follow the instructions below to activate any of these endpoints and begin making transfers.
1.[Sign in](https://www.globus.org/SignIn) to your Globus account using your campus credentials or your Globus ID (if you have one). Then click on 'Endpoints' in the left sidebar.
{{<figuresrc="/images/Glogin.png">}}
{{<figuresrc="/images/endpoints.png">}}
2. Find the endpoint you want by entering '`hcc#tusker`', '`hcc#crane`', or '`hcc#attic`' in the search box and hit 'enter'. Once you have found and selected the endpoint, click the green 'activate' icon. On the following page, click 'continue'.
2. Find the endpoint you want by entering '`hcc#crane`', or '`hcc#attic`' in the search box and hit 'enter'. Once you have found and selected the endpoint, click the green 'activate' icon. On the following page, click 'continue'.
@@ -12,5 +12,5 @@ The following pages, [Create Local BLAST Database]({{<relref "create_local_blast
### Useful Information
In order to test the BLAST (blast/2.2) performance on Tusker, we aligned three nucleotide query datasets, `small.fasta`, `medium.fasta` and `large.fasta`, against the non-redundant nucleotide **nt.fasta** database from NCBI. Some statistics about the query datasets and the time and memory resources used for the alignment are shown on the table below:
In order to test the BLAST (blast/2.2) performance on Crane, we aligned three nucleotide query datasets, `small.fasta`, `medium.fasta` and `large.fasta`, against the non-redundant nucleotide **nt.fasta** database from NCBI. Some statistics about the query datasets and the time and memory resources used for the alignment are shown on the table below:
where **input_reads.fasta** is the input file containing all sequences that need to be made into a database, and **dbtype** can be either `nucl` or `prot` depending on the type of the input file.
Simple example of how **makeblastdb** can be run on Tusker using SLURM script and nucleotide database is shown below:
Simple example of how **makeblastdb** can be run on Crane using SLURM script and nucleotide database is shown below:
These BLAST alignment commands are multi-threaded, and therefore using the BLAST option **-num_threads <number_of_CPUs>** is recommended.
HCC hosts multiple BLAST databases and indices on both Tusker and Crane. In order to use these resources, the ["biodata" module] ({{<relref"/guides/running_applications/bioinformatics_tools/biodata_module">}}) needs to be loaded first. The **$BLAST** variable contains the following currently available databases:
HCC hosts multiple BLAST databases and indices on Crane. In order to use these resources, the ["biodata" module] ({{<relref"/guides/running_applications/bioinformatics_tools/biodata_module">}}) needs to be loaded first. The **$BLAST** variable contains the following currently available databases:
Bowtie supports both single-end (`input_reads.[fasta|fastq]`) and paired-end (`input_reads_pair_1.[fasta|fastq]`, `input_reads_pair_2.[fasta|fastq]`) files in fasta or fastq format. The format of the input files also needs to be specified by using the following flags: **-q** (fastq files), **-f** (fasta files), **-r** (raw one-sequence per line), or **-c** (sequences given on command line).
An example of how to run Bowtie alignment on Tusker with single-end fastq file and `8 CPUs` is shown below:
An example of how to run Bowtie alignment on Crane with single-end fastq file and `8 CPUs` is shown below:
where **index_prefix** is the generated index using the **bowtie2-build** command, and **options** are optional parameters that can be found in the [Bowtie2 manual] (http://bowtie-bio.sourceforge.net/bowtie2/manual.shtml). Bowtie2 supports both single-end (`input_reads.[fasta|fastq]`) and paired-end (`input_reads_pair_1.[fasta|fastq]`, `input_reads_pair_2.[fasta|fastq]`) files in fasta or fastq format. The format of the input files also needs to be specified by using one of the following flags: **-q** (fastq files), **--qseq** (Illumina's qseq format), **-f** (fasta files), **-r** (raw one sequence per line), or **-c** (sequences given on command line).
An example of how to run Bowtie2 local alignment on Tusker with paired-end fasta files and `8 CPUs` is shown below:
An example of how to run Bowtie2 local alignment on Crane with paired-end fasta files and `8 CPUs` is shown below:
@@ -22,7 +22,7 @@ $ bwa mem index_prefix [input_reads.fastq|input_reads_pair_1.fastq input_reads_p
where **index_prefix** is the index for the reference genome generated from **bwa index**, and **input_reads.fastq**, **input_reads_pair_1.fastq**, **input_reads_pair_2.fastq** are the input files of sequencing data that can be single-end or paired-end respectively. Additional **options** for **bwa mem** can be found in the BWA manual.
Simple SLURM script for running **bwa mem** on Tusker with paired-end fastq input data, `index_prefix` as reference genome index, SAM output file and `8 CPUs` is shown below:
Simple SLURM script for running **bwa mem** on Crane with paired-end fastq input data, `index_prefix` as reference genome index, SAM output file and `8 CPUs` is shown below:
