using_attic.md 4.01 KB
Newer Older
eharstad's avatar
eharstad committed
1
2
3
4
5
+++
title = "Using Attic"
description = "How to store data on Attic"
weight = 20
+++
6
7
8
9
10
11

For users who need long-term storage for large amount of data, HCC
provides an economical solution called Attic.  Attic is a reliable
<a href="https://en.wikipedia.org/wiki/Nearline_storage" class="external-link">near-line data archive</a> storage
system. The files in Attic can be accessed and shared from anywhere
using [Globus
eharstad's avatar
eharstad committed
12
Connect]({{< relref "globus_connect" >}}),
13
14
15
16
17
18
with a fast 10Gb/s link.  Also, the data in Attic is backed up between
our Lincoln and Omaha facilities to ensure high availability and
disaster tolerance. The data and user activities on Attic are subject to
our
<a href="http://hcc.unl.edu/hcc-policies" class="external-link">HCC Policies</a>.

eharstad's avatar
eharstad committed
19
20
---
### Accounts and Cost
21
22
23

To use Attic you will first need an
<a href="https://hcc.unl.edu/new-user-request" class="external-link">HCC account</a>, and
eharstad's avatar
eharstad committed
24
25
then you may request an
<a href="http://hcc.unl.edu/attic" class="external-link">Attic allocation</a>.
26
27
28
29
30
We charge a small fee per TB per year, but it is cheaper than most
commercial cloud storage solutions.  For the user application form and
cost, please see the
<a href="http://hcc.unl.edu/attic" class="external-link">HCC Attic page</a>.

eharstad's avatar
eharstad committed
31
32
---
### Transfer Files Using Globus Connect
33
34

The easiest and fastest way to access Attic is via Globus. You can
eharstad's avatar
eharstad committed
35
transfer files between your computer, our clusters ($HOME, $WORK, and $COMMON on
36
Crane), and Attic. Here is a detailed tutorial on
eharstad's avatar
eharstad committed
37
how to set up and use [Globus Connect]({{< relref "globus_connect" >}}). For
38
39
40
41
42
43
44
Attic, use the Globus Endpoint **hcc\#attic**.  Your Attic files are
located at `~, `which is a shortcut
for `/attic/<groupname>/<username>`.  
**Note:** *If you are accessing Attic files from your supplementary
group, you should explicitly set the path to
/attic/&lt;supplementary\_groupname&gt;/. If you don't do that, by
default the endpoint will try to place you in your primary group's Attic
eharstad's avatar
eharstad committed
45
path, to which access will be denied if the primary group doesn't have an Attic allocation.*
46

eharstad's avatar
eharstad committed
47
48
---
### Transfer Files Using SCP/SFTP/RSYNC
49

eharstad's avatar
eharstad committed
50
The transfer server for Attic storage is `attic.unl.edu` (or `attic-xfer.unl.edu`).
51

eharstad's avatar
eharstad committed
52
53
54
55
56
{{% panel theme="info" header="SCP Example" %}}
{{< highlight bash >}}
scp /source/file <username>@attic.unl.edu:~/destination/file
{{< /highlight >}}
{{% /panel %}}
57

eharstad's avatar
eharstad committed
58
59
60
{{% panel theme="info" header="SFTP Example" %}}
{{< highlight bash >}}
sftp <username>@attic.unl.edu
61
62
63
64
65
66
67
Password:
Duo two-factor login for <username>
Connected to attic.unl.edu.
sftp> pwd
Remote working directory: /attic/<groupname>/<username>
sftp> put source/file destination/file
sftp> exit
eharstad's avatar
eharstad committed
68
69
{{< /highlight >}}
{{% /panel %}}
70

eharstad's avatar
eharstad committed
71
72
{{% panel theme="info" header="RSYNC Example" %}}
{{< highlight bash >}}
73
# local to remote rsync command
eharstad's avatar
eharstad committed
74
rsync -avz /local/source/path <username>@attic.unl.edu:remote/destination/path
75
76

# remote to local rsync command
eharstad's avatar
eharstad committed
77
78
79
rsync -avz <username>@attic.unl.edu:remote/source/path /local/destination/path
{{< /highlight >}}
{{% /panel %}}
80

eharstad's avatar
eharstad committed
81
82
You can also access your data on Attic using our [high-speed
transfer servers]({{< relref "high_speed_data_transfers" >}}) if you prefer.
83
84
85
Simply use scp or sftp to connect to one of the transfer servers, and
your directory is mounted at `/attic/<groupname>/<username>`.

eharstad's avatar
eharstad committed
86
87
---
### Check Attic Usage
88
89

The usage and quota information for your group and the users in the
eharstad's avatar
eharstad committed
90
91
group are stored in a file named "disk\_usage.txt" in your group's
directory (`/attic/<groupname>`). You can use either [Globus Connect]({{< relref "globus_connect" >}}) or
92
93
94
scp to download it.  Your usage and expiration is also shown in the web
interface (see below).

eharstad's avatar
eharstad committed
95
96
---
### Use the web interface
97
98
99
100
101
102
103
104
105

For convenience, a web interface is also provided.  Simply go to
<a href="https://attic.unl.edu" class="external-link">https://attic.unl.edu</a>
and login with your HCC credentials.  Using this interface, you can see
your quota usage and expiration, manage files, etc.  **Please note we do
not recommend uploading/downloading large files this way**.  Use one of
the other transfer methods above for large datasets.