Skip to content
Snippets Groups Projects
Commit 4abd07ec authored by Adam Caprez's avatar Adam Caprez
Browse files

Rclone

parent a46629f1
No related branches found
No related tags found
No related merge requests found
......@@ -19,7 +19,7 @@ in good network traffic conditions.  Users can use a tool called lftp to transf
1. Login to your [UNK Box.com](https://unk.account.box.com/), [UNL Box.com](https://unl.account.box.com/), or [UNO Box.com](https://unomaha.account.box.com/) account.
2. Since we are going to be using [webdav](https://en.wikipedia.org/wiki/WebDAV) protocol to access your [Box.com](https://www.box.com/) storage, you need to create an **External Password**. In the Box.com interface, you can create it at **Account Settings > Account > Authentication > Create Password.**
2. Since we are going to be using [ftps](https://en.wikipedia.org/wiki/FTPS) protocol to access your [Box.com](https://www.box.com/) storage, you need to create an **External Password**. In the Box.com interface, you can create it at **Account Settings > Account > Authentication > Create Password.**
{{< figure src="/images/box_create_external_password.png" class="img-border" >}}
3. After logging into the cluster of your choice, load the `lftp` module by entering the command below at the prompt:
......
......@@ -10,9 +10,9 @@ This tool can be used to transfer files between HCC clusters and outside cloud p
---
### Setup RClone
1. You need to create your UNL [Box.com](https://www.box.com/) account [here](https://box.unl.edu/).
1. You must be able to access your [UNL Box.com](https://unl.box.com/), [UNK Box.com](https://unk.account.box.com/), [UNO Box.com](https://unomaha.account.box.com/), or [UNMC Box.com](https://unmc.account.box.com/) before beginning this process. Contact your local campus support if you need help with initial account setup.
2. Due to the clusters being remote machines, Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize box. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
2. Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize Box**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
3. After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
{{% panel theme="info" header="Load the Rclone module" %}}
......@@ -28,7 +28,7 @@ This tool can be used to transfer files between HCC clusters and outside cloud p
{{< /highlight >}}
{{% /panel %}}
5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "UNLBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. When you are prompted for auto config, select `N` and switch to a terminal on your local machine:
5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "MyBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. **When you are prompted for auto config, select `N`**.
{{% panel theme="info" header="Configure box" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone config
......@@ -37,7 +37,7 @@ n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> UNLBox
name> MyBox
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
......@@ -67,8 +67,8 @@ Enter a signed integer. Press Enter for the default ("100").
commit_retries>
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
\* Say Y if not sure
\* Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n
......@@ -80,16 +80,16 @@ result>
{{< /highlight >}}
{{% /panel %}}
6. Run `rclone authorize "box"` on the local machine. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your UNL e-mail address. You will be taken to sign into UNL's Box using your **Canvas** credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
{{% panel theme="info" header="List contents of Box" %}}
6. Now **switch to a terminal on your local machine** and run `rclone authorize "box"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your NU e-mail address. You will be taken to sign into your campus' Box instance using your TrueYou/campus credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
{{% panel theme="info" header="Authorize Box on local machine" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone authorize "box"
[myaccount@local.machine ~]$ rclone authorize "box"
{{< /highlight >}}
{{% /panel %}}
{{< figure src="/images/BoxSSO.png" height="500" class="img-border">}}
{{% panel theme="info" header="Local Config" %}}
{{< highlight bash >}}
[demo2@local.machine ~]$ rclone authorize "box"
{{% panel theme="info" header="Local Authorization" %}}
{{< highlight script >}}
[myaccount@local.machine ~]$ rclone authorize "box"
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize Rclone for access
Waiting for code...
......@@ -100,10 +100,10 @@ For other services, please refer to the [rclone documentation](https://rclone.or
7. Test the connection by running the `ls` command. You should see a listing of your Box files.
7. **Returning to the cluster**, test the connection by running the `ls` command. You should see a listing of your Box files.
{{% panel theme="info" header="List contents of Box" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone ls UNLBox:/
[demo2@login.crane ~]$ rclone ls MyBox:/
{{< /highlight >}}
{{% /panel %}}
......@@ -111,28 +111,28 @@ For other services, please refer to the [rclone documentation](https://rclone.or
8. To upload or download files, use the `clone` command. For example:
{{% panel theme="info" header="Transferring files" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone copy UNLBox:/SomeFile.txt ./
[demo2@login.crane ~]$ rclone copy ./SomeFile.txt UNLBox:/
[demo2@login.crane ~]$ rclone copy MyBox:/SomeFile.txt ./
[demo2@login.crane ~]$ rclone copy ./SomeFile.txt MyBox:/
{{< /highlight >}}
{{% /panel %}}
9. To download directories, use the `clone` command and use directory names over file. This copies the contents of the folders, so you need to specify a destination folder.
{{% panel theme="info" header="Download a directory from Box" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone copy UNLBox:/my_hcc_dir ./my_hcc_dir
[demo2@login.crane ~]$ rclone copy MyBox:/my_hcc_dir ./my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
To upload a directory named `my_hcc_dir` to Box, use `clone`.
{{% panel theme="info" header="Upload a directory to Box" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone copy ./my_hcc_dir UNLBox:/my_hcc_dir
[demo2@login.crane ~]$ rclone copy ./my_hcc_dir MyBox:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
11. Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The exmaple below would sync the files of the local directory to the remote directory on box.
{{% panel theme="info" header="transfer.sh" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone sync ./my_hcc_dir UNLBox:/my_hcc_dir
[demo2@login.crane ~]$ rclone sync ./my_hcc_dir MyBox:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment