Skip to content
Snippets Groups Projects

Update handling data articles

Merged Update handling data articles
fix_handling_data into master
Merged eharstad requested to merge fix_handling_data into master
Compare
and
9 files
+ 259
− 496
Compare changes
  • Side-by-side
  • Inline
Files
9
content/guides/handling_data/_index.md
+ 35
− 76
@@ -4,33 +4,20 @@ description = "How to work with and transfer data to/from HCC resources."
weight = "30"
+++
<span id="title-text"> HCC-DOCS : Handling Data </span>
=======================================================
Created by <span class="author"> Derek Weitzel</span>, last modified by
<span class="editor"> Carrie Brown</span> on Sep 18, 2018
<span
class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span>
HCC currently has no storage that is suitable for HIPAA or other PID
data sets. Users are not permitted to store such data on HCC machines.
{{% panel theme="danger" header="**Sensitive and Protected Data**" %}}HCC currently has *no storage* that is suitable for **HIPAA** or other **PID** data sets. Users are not permitted to store such data on HCC machines.{{% /panel %}}
All HCC machines have three separate areas for every user to store data,
each intended for a different purpose. In addition, we have a transfer
service that utilizes [Globus Connect](Globus-Connect_6357013.html).
<span
class="confluence-embedded-file-wrapper image-center-wrapper confluence-embedded-manual-size"><img src="assets/images/332256/35325560.png" class="confluence-embedded-image image-center" width="1000" /></span>
service that utilizes [Globus Connect]({{< relref "globus_connect" >}}).
{{< figure src="/images/35325560.png" >}}
Home Directory
--------------
<span
class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span>
---
### Home Directory
{{% notice info %}}
You can access your home directory quickly using the $HOME environmental
variable (i.e. '`cd $HOME'`).
{{% /notice %}}
Your home directory (i.e. `/home/[group]/[username]`) is meant for items
that take up relatively small amounts of space. For example: source
@@ -40,28 +27,25 @@ for the purposes of best-effort disaster recovery.  This space is not
intended as an area for I/O to active jobs. **/home** is mounted
**read-only** on cluster worker nodes to enforce this policy.
Common Directory
----------------
<span
class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span>
---
### Common Directory
{{% notice info %}}
You can access your common directory quickly using the $COMMON
environmental variable (i.e. '`cd $COMMON`')
{{% /notice %}}
The common directory operates similarly to work and is mounted with
**read and write capability to worker nodes all HCC Clusters**. This
means that any files stored in common can be accessed from Crane, Tusker
and Sandhills making this directory ideal for items that need to be
means that any files stored in common can be accessed from Crane and Tusker, making this directory ideal for items that need to be
accessed from multiple clusters such as reference databases and shared
data files.
<span
class="aui-icon aui-icon-small aui-iconfont-warning confluence-information-macro-icon"></span>
{{% notice warning %}}
Common is not designed for heavy I/O usage. Please continue to use your
work directory for active job output to ensure the best performance of
your jobs.
{{% /notice %}}
Quotas for common are **30 TB per group**, with larger quotas available
for purchase if needed. However, files stored here will **not be backed
@@ -69,23 +53,17 @@ up** and are **not subject to purge** at this time. Please continue to
backup your files to prevent irreparable data loss.
Additional information on using the common directories can be found in
the documentation on [Using the /common File System](30444241.html)
the documentation on [Using the /common File System]({{< relref "using_the_common_file_system" >}})
High Performance Work Directory
-------------------------------
<span
class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span>
---
### High Performance Work Directory
{{% notice info %}}
You can access your work directory quickly using the $WORK environmental
variable (i.e. '`cd $WORK'`).
{{% /notice %}}
<span
class="aui-icon aui-icon-small aui-iconfont-error confluence-information-macro-icon"></span>
The `/work` directories are **not backed up**. Irreparable data loss is
possible with a mis-typed command. See [Preventing File
Loss](Preventing-File-Loss_29065313.html) for strategies to avoid this.
{{% panel theme="danger" header="**File Loss**" %}}The `/work` directories are **not backed up**. Irreparable data loss is possible with a mis-typed command. See [Preventing File Loss]({{< relref "preventing_file_loss" >}}) for strategies to avoid this.{{% /panel %}}
Every user has a corresponding directory under /work using the same
naming convention as `/home` (i.e. `/work/[group]/[username]`). We
@@ -93,11 +71,11 @@ encourage all users to use this space for I/O to running jobs.  This
directory can also be used when larger amounts of space are temporarily
needed. There is a **50TB per group quota**; space in /work is shared
among all users. It should be treated as short-term scratch space, and
**is not backed up**. <span style="color: rgb(255,0,0);"><span
style="color: rgb(0,0,0);">Please use the `hcc-du` command to check your
**is not backed up**. **Please use the `hcc-du` command to check your
own and your group's usage, and back up and clean up your files at
reasonable intervals in $WORK.</span></span>
reasonable intervals in $WORK.**
---
### Purge Policy
HCC has a **purge policy on /work** for files that become dormant.
@@ -113,58 +91,39 @@ list the matching files for the user.  The candidate list can also be
accessed at the following path:` /lustre/purge/current/${USER}.list`.
This list is updated twice a week, on Mondays and Thursdays.
<span
class="aui-icon aui-icon-small aui-iconfont-error confluence-information-macro-icon"></span>
/work is intended for recent job output and not long term storage.
Evidence of circumventing the purge policy by users will result in
consequences including account lockout.
{{% notice warning %}}
`/work` is intended for recent job output and not long term storage. Evidence of circumventing the purge policy by users will result in consequences including account lockout.
{{% /notice %}}
If you have space requirements outside what is currently provided,
please
email <a href="mailto:hcc-support@unl.edu" class="external-link">hcc-support@unl.edu</a> and
we will gladly discuss alternatives.
[Attic](Using-Attic_11635580.html)
----------------------------------
---
### [Attic]({{< relref "using_attic" >}})
Attic is a near line archive available for purchase at HCC. Attic
provides reliable large data storage that is designed to be more
reliable then `/work`, and larger than `/home`. Access to Attic is done
through [Globus Connect](Globus-Connect_6357013.html).
through [Globus Connect]({{< relref "globus_connect" >}}).
More details on Attic can be found on HCC's
<a href="https://hcc.unl.edu/attic" class="external-link">Attic</a>
website.
<span style="color: rgb(0,0,0);line-height: 1.4285715;font-size: 20.0px;">[Globus Connect](Globus-Connect_6357013.html)</span>
------------------------------------------------------------------------------------------------------------------------------
---
### [Globus Connect]({{< relref "globus_connect" >}})
For moving large amounts of data into or out of HCC resources, users are
highly encouraged to consider using [Globus
Connect](Globus-Connect_6357013.html).
Connect]({{< relref "globus_connect" >}}).
Using Box
---------
---
### Using Box
You can use your [UNL
Box.com](Integrating-Box-with-HCC_8192521.html) account to download and
Box.com]({{< relref "integrating_box_with_hcc" >}}) account to download and
upload files from any of the HCC clusters.
Attachments:
------------
<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" />
[HCCStorageOptions\_cb\_edits.pdf](attachments/332256/30444364.pdf)
(application/pdf)
<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" />
[HCCStorageOptions\_cb\_edits.png](attachments/332256/30444365.png)
(image/png)
<img src="assets/images/icons/bullet_blue.gif" width="8" height="8" />
[StorageOptions.png](attachments/332256/35325560.png) (image/png)