Commit 03514cfb authored by eharstad's avatar eharstad
Browse files

Reworked data handling sections

parent 29bc2408
......@@ -165,7 +165,7 @@ precious or irreproducible data should not be placed or left on Anvil**.
Transferring files to or from an instance is similar to doing so
with a personal laptop or workstation.  To transfer between an
instance and another HCC resource, both SCP and [Globus
Connect]({{< relref "/Handling_Data/globus_connect" >}}) can be used.  For transferring
Connect]({{< relref "/Data_Transfer/globus_connect" >}}) can be used.  For transferring
between an instance and a laptop/workstation or another instance,
standard file sharing utilities such as Dropbox or Box can be used.
 Globus may also be used, with one stipulation.  In order to
......
+++
title = "Data Storage"
description = "Storing Data on HCC resources."
weight = "50"
+++
{{% children description="true" %}}
+++
title = "Data for UNMC Users Only"
description= "Data storage options for UNMC users"
weight = 50
weight = 60
+++
{{% panel theme="danger" header="Sensitive and Protected Data" %}} HCC currently has no storage that is suitable for HIPAA or other PID
......
+++
title = "Handling Data"
description = "How to work with and transfer data to/from HCC resources."
weight = "50"
title = "Data Storage Overview"
description = "Overview of Data Storage on HCC resources."
weight = "10"
+++
{{% panel theme="danger" header="**Sensitive and Protected Data**" %}}HCC currently has *no storage* that is suitable for **HIPAA** or other **PID** data sets. Users are not permitted to store such data on HCC machines.{{% /panel %}}
All HCC machines have three separate areas for every user to store data,
each intended for a different purpose.   In addition, we have a transfer
service that utilizes [Globus Connect]({{< relref "globus_connect" >}}).
service that utilizes [Globus Connect]({{< relref "/Data_Transfer/globus_connect" >}}).
{{< figure src="/images/35325560.png" height="500" class="img-border">}}
---
......@@ -107,18 +107,18 @@ we will gladly discuss alternatives.
Attic is a near line archive available for purchase at HCC.  Attic
provides reliable large data storage that is designed to be more
reliable then `/work`, and larger than `/home`. Access to Attic is done
through [Globus Connect]({{< relref "globus_connect" >}}).
through [Globus Connect]({{< relref "/Data_Transfer/globus_connect" >}}).
More details on Attic can be found on HCC's
<a href="https://hcc.unl.edu/attic" class="external-link">Attic</a>
website.
---
### [Globus Connect]({{< relref "globus_connect" >}})
### [Globus Connect]({{< relref "/Data_Transfer/globus_connect" >}})
For moving large amounts of data into or out of HCC resources, users are
highly encouraged to consider using [Globus
Connect]({{< relref "globus_connect" >}}).
Connect]({{< relref "/Data_Transfer/globus_connect" >}}).
---
### Using Box
......
+++
title = "Integrating Box with HCC"
description = "How to integrate Box with HCC"
weight = 30
weight = 50
+++
UNL has come to an arrangement
......
+++
title = "Preventing File Loss"
description = "How to prevent file loss on HCC clusters"
weight = 60
weight = 40
+++
Each research group is allocated 50TB of storage in `/work` on HCC
......@@ -26,7 +26,7 @@ directories to your personal computer. To avoid filling up your personal
hard-drives, consider using an external drive that can easily be placed
in a fireproof safe or at an off-site location for an extra level of
protection. To do this, you can either use [Globus
Connect]({{< relref "globus_connect" >}}) or an
Connect]({{< relref "/Data_Transfer/globus_connect" >}}) or an
SCP client, such
as <a href="https://cyberduck.io/" class="external-link">Cyberduck</a> or <a href="https://winscp.net/eng/index.php" class="external-link">WinSCP</a>.
For help setting up an SCP client, check out our [Quick Start
......@@ -45,7 +45,7 @@ Benefits:
- Take advantage of our high speed data transfers (10 Gb/s) when using
Globus Connect or [setup your SCP client to use our dedicated high
speed transfer
servers]({{< relref "high_speed_data_transfers" >}})
servers]({{< relref "/Data_Transfer/high_speed_data_transfers" >}})
Limitations:
......@@ -100,8 +100,8 @@ Benefits:
layer against file loss.
- No limits on individual or total file sizes.
- High speed data transfers between Attic and the clusters when using
[Globus Connect]({{< relref "globus_connect" >}}) and [HCC's high-speed data
servers]({{< relref "high_speed_data_transfers" >}}).
[Globus Connect]({{< relref "/Data_Transfer/globus_connect" >}}) and [HCC's high-speed data
servers]({{< relref "/Data_Transfer/high_speed_data_transfers" >}}).
Limitations:
......
......@@ -9,7 +9,7 @@ provides an economical solution called Attic.  Attic is a reliable
<a href="https://en.wikipedia.org/wiki/Nearline_storage" class="external-link">near-line data archive</a> storage
system. The files in Attic can be accessed and shared from anywhere
using [Globus
Connect]({{< relref "globus_connect" >}}),
Connect]({{< relref "/Data_Transfer/globus_connect" >}}),
with a fast 10Gb/s link.  Also, the data in Attic is backed up between
our Lincoln and Omaha facilities to ensure high availability and
disaster tolerance. The data and user activities on Attic are subject to
......@@ -34,7 +34,7 @@ cost, please see the
The easiest and fastest way to access Attic is via Globus. You can
transfer files between your computer, our clusters ($HOME, $WORK, and $COMMON on
Crane or Rhino), and Attic. Here is a detailed tutorial on
how to set up and use [Globus Connect]({{< relref "globus_connect" >}}). For
how to set up and use [Globus Connect]({{< relref "/Data_Transfer/globus_connect" >}}). For
Attic, use the Globus Endpoint **hcc\#attic**.  Your Attic files are
located at `~, `which is a shortcut
for `/attic/<groupname>/<username>`.
......@@ -79,7 +79,7 @@ rsync -avz <username>@attic.unl.edu:remote/source/path /local/destination/path
{{% /panel %}}
You can also access your data on Attic using our [high-speed
transfer servers]({{< relref "high_speed_data_transfers" >}}) if you prefer.
transfer servers]({{< relref "/Data_Transfer/high_speed_data_transfers" >}}) if you prefer.
Simply use scp or sftp to connect to one of the transfer servers, and
your directory is mounted at `/attic/<groupname>/<username>`.
......@@ -88,7 +88,7 @@ your directory is mounted at `/attic/<groupname>/<username>`.
The usage and quota information for your group and the users in the
group are stored in a file named "disk\_usage.txt" in your group's
directory (`/attic/<groupname>`). You can use either [Globus Connect]({{< relref "globus_connect" >}}) or
directory (`/attic/<groupname>`). You can use either [Globus Connect]({{< relref "/Data_Transfer/globus_connect" >}}) or
scp to download it.  Your usage and expiration is also shown in the web
interface (see below).
......
+++
title = "Using NU's Gitlab instance"
description = "How to use the NU-hosted git.unl.edu"
weight = "40"
weight = "70"
+++
[Git](http://git-scm.com) is a free
......
+++
title = "Using the /common File System"
description = "How to use HCC's /common file system"
weight = 70
weight = 30
+++
### Quick overview: 
......
+++
title = "Data Transfer"
description = "Transferring Data to and from HCC resources."
weight = "55"
+++
{{% children description="true" %}}
+++
title = "Connecting to CB3 iRODS"
description= "How to connect to the CB3 iRODS instance."
weight = 80
weight = 40
+++
{{% panel theme="danger" header="Sensitive and Protected Data" %}}
......
+++
title = "Globus Connect"
description = "Globus Connect overview"
weight = 20
+++
<a href="https://www.globus.org/globus-connect" class="external-link">Globus Connect</a> is
......
......@@ -32,7 +32,7 @@ and will not have to be activated again for the next 7 days.
To transfer files between any two HCC clusters, you will need to
activate both endpoints individually.
Next, learn how to [make file transfers between HCC endpoints]({{< relref "file_transfers_between_endpoints" >}}) or how to [transfer between HCC endpoints and a personal computer]({{< relref "file_transfers_to_and_from_personal_workstations" >}}).
Next, learn how to [make file transfers between HCC endpoints]({{< relref "/Data_Transfer/globus_connect/file_transfers_between_endpoints" >}}) or how to [transfer between HCC endpoints and a personal computer]({{< relref "/Data_Transfer/globus_connect/file_transfers_to_and_from_personal_workstations" >}}).
---
......
......@@ -5,16 +5,16 @@ weight = 30
+++
To transfer files between HCC clusters, you will first need to
[activate]({{< relref "activating_hcc_cluster_endpoints" >}}) the
[activate]({{< relref "/Data_Transfer/globus_connect/activating_hcc_cluster_endpoints" >}}) the
two endpoints you would like to use (the available endpoints
are: `hcc#crane` `hcc#rhino`, and `hcc#attic`).  Once
that has been completed, follow the steps below to begin transferring
files.  (Note: You can also transfer files between an HCC endpoint and
any other Globus endpoint for which you have authorized access.  That
may include a [personal
endpoint]({{< relref "file_transfers_to_and_from_personal_workstations" >}}),
endpoint]({{< relref "/Data_Transfer/globus_connect/file_transfers_to_and_from_personal_workstations" >}}),
a [shared
endpoint]({{< relref "file_sharing" >}}),
endpoint]({{< relref "/Data_Transfer/globus_connect/file_sharing" >}}),
or an endpoint on another computing resource or cluster.  Once the
endpoints have been activated, the file transfer process is generally
the same regardless of the type of endpoints you use.  For demonstration
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment