Skip to content
Snippets Groups Projects

Rclone Docs

Merged Rclone Docs
rclone into master
Merged Caughlin Bohn requested to merge rclone into master
Compare
and
2 files
+ 139
− 0
Compare changes
  • Side-by-side
  • Inline

Files

content/guides/handling_data/using_rclone_with_hcc.md 0 → 100755
+ 139
− 0
 
+++
 
title = "Using Rclone for File Transfer"
 
description = "How to use Rclone with HCC"
 
weight = 9
 
+++
 
 
Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box, Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task. Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/).
 
 
This tool can be used to transfer files between HCC clusters and outside cloud providers, such as Box.
 
---
 
### Setup RClone
 
 
1. You need to create your UNL [Box.com](https://www.box.com/) account [here](https://box.unl.edu/).
 
 
2. Due to the clusters being remote machines, Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize box. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
 
 
3. After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
 
{{% panel theme="info" header="Load the Rclone module" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ module load rclone
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
4. We will need to start the basic configuration for box. To do this run `rclone config`:
 
{{% panel theme="info" header="Load the rclone config" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone config
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "UNLBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. When you are prompted for auto config, select `N` and switch to a terminal on your local machine:
 
{{% panel theme="info" header="Configure box" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone config
 
No remotes found - make a new one
 
n) New remote
 
s) Set configuration password
 
q) Quit config
 
n/s/q> n
 
name> UNLBox
 
Type of storage to configure.
 
Enter a string value. Press Enter for the default ("").
 
Choose a number from below, or type in your own value
 
 
6 / Box
 
\ "box"
 
 
Storage> 6
 
 
Box App Client Id.
 
Leave blank normally.
 
Enter a string value. Press Enter for the default ("").
 
client_id>
 
Box App Client Secret
 
Leave blank normally.
 
Enter a string value. Press Enter for the default ("").
 
client_secret>
 
Edit advanced config? (y/n)
 
y) Yes
 
n) No
 
y/n> y
 
Cutoff for switching to multipart upload (>= 50MB).
 
Enter a size with suffix k,M,G,T. Press Enter for the default ("50M").
 
upload_cutoff> 15G
 
Max number of times to try committing a multipart file.
 
Enter a signed integer. Press Enter for the default ("100").
 
commit_retries>
 
Remote config
 
Use auto config?
 
* Say Y if not sure
 
* Say N if you are working on a remote or headless machine
 
y) Yes
 
n) No
 
y/n> n
 
For this to work, you will need Rclone available on a machine that has a web browser available.
 
Execute the following on your machine:
 
rclone authorize "box"
 
Then paste the result below:
 
result>
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
6. Run `rclone authorize "box"` on the local machine. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your UNL e-mail address. You will be taken to sign into UNL's Box using your **Canvas** credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
 
{{% panel theme="info" header="List contents of Box" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone authorize "box"
 
{{< /highlight >}}
 
{{% /panel %}}
 
{{< figure src="/images/BoxSSO.png" height="500" class="img-border">}}
 
{{% panel theme="info" header="Local Config" %}}
 
{{< highlight bash >}}
 
[demo2@local.machine ~]$ rclone authorize "box"
 
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
 
Log in and authorize Rclone for access
 
Waiting for code...
 
{{< /highlight >}}
 
{{% /panel %}}
 
For other services, please refer to the [rclone documentation](https://rclone.org/).
 
 
 
 
 
7. Test the connection by running the `ls` command. You should see a listing of your Box files.
 
{{% panel theme="info" header="List contents of Box" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone ls UNLBox:/
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
 
8. To upload or download files, use the `clone` command. For example:
 
{{% panel theme="info" header="Transferring files" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone copy UNLBox:/SomeFile.txt ./
 
[demo2@login.crane ~]$ rclone copy ./SomeFile.txt UNLBox:/
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
9. To download directories, use the `clone` command and use directory names over file. This copies the contents of the folders, so you need to specify a destination folder.
 
{{% panel theme="info" header="Download a directory from Box" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone copy UNLBox:/my_hcc_dir ./my_hcc_dir
 
{{< /highlight >}}
 
{{% /panel %}}
 
To upload a directory named `my_hcc_dir` to Box, use `clone`.
 
{{% panel theme="info" header="Upload a directory to Box" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone copy ./my_hcc_dir UNLBox:/my_hcc_dir
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
11. Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The exmaple below would sync the files of the local directory to the remote directory on box.
 
{{% panel theme="info" header="transfer.sh" %}}
 
{{< highlight bash >}}
 
[demo2@login.crane ~]$ rclone sync ./my_hcc_dir UNLBox:/my_hcc_dir
 
{{< /highlight >}}
 
{{% /panel %}}