Commit c12c2361 authored by Adam Caprez's avatar Adam Caprez
Browse files

Merge branch 'rclone-nextcloud' into 'master'

Add page for rclone+NextCloud.

See merge request !299
parents d9a32056 bc580d03
+++
title = "Using Rclone with NextCloud"
description = "How to setup Rclone for using with HCC's NextCloud"
weight = 100
hidden = true
+++
Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box,
Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task.
Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/).
This tool can be used to transfer files between HCC's NextCloud instance and HCC's clusters, local machines, etc. using the WebDAV protocol.
This guide will show how to create dedicated credentials within NextCloud and configure rclone.
## NextCloud setup
### Copy the server address
Open a text editor or some other convenient way to store the URL and password that will be copied from NextCloud.
Log into HCC's NextCloud instance at https://hcc-nextcloud.unl.edu. You should see a page similar to this.
{{< figure src="/images/nextcloud1.png" height="700" class="img-border">}}
Click the _Settings_ option with the gear icon in the lower left corner of the page.
{{< figure src="/images/nextcloud2.png" height="500" class="img-border">}}
This will open an additional section directly underneath. Highlight the _entire_ string of text under the _WebDAV_ entry
and either using Ctrl-C or right-click to copy it.
{{< figure src="/images/nextcloud3.png" height="400" class="img-border">}}
Paste the text into a conventient editor. This is the server address to be used with rclone.
### Create dedicated rclone password
Next, click the user account circle in the upper right corner and choose _Settings_ from the drop-down menu.
{{< figure src="/images/nextcloud4.png" height="200" class="img-border">}}
This will open a new page with your account settings. Click the _Security_ option on the left-hand menu.
{{< figure src="/images/nextcloud5.png" height="400" class="img-border">}}
This page will allow you to create a dedicated password to use for rclone transfers to/from NextCloud.
The username used is the same as your HCC username.
Enter a name to use in the box and click _Create new app password_.
{{< figure src="/images/nextcloud6.png" height="400" class="img-border">}}
Once generated, click the clipboard icon to copy the password and paste it into your text editor.
_This is the only time the password will be shown._
{{< figure src="/images/nextcloud7.png" height="400" class="img-border">}}
Finally, click the _Done_ button when finished.
## Rclone setup
{{% notice info %}}
The instructions here are written assuming rclone usage on an HCC cluster. The same procedure (starting with
the `rclone config` step) however can be used on any machine (i.e. local desktop) that rclone is installed on.
See [this link](https://rclone.org/install/) for details on installing rclone locally.
{{% /notice %}}
### Create new remote
After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
{{% panel theme="info" header="Load the Rclone module" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ module load rclone
{{< /highlight >}}
{{% /panel %}}
To add a new remote in Rclone, run `rclone config`:
{{% panel theme="info" header="Load the rclone config" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone config
{{< /highlight >}}
{{% /panel %}}
Choose the new remote option and enter a name (here "HCCNC" is used).
{{% panel theme="info" header="Create new remote" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone config
2021/11/11 22:38:42 NOTICE: Config file "/work/demo/appstest/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> HCCNC
{{< /highlight >}}
{{% /panel %}}
Rclone will print a list of supported storage systems and prompt for a choice. Type in `webdav` and hit enter.
{{% panel theme="info" header="Choose storage type" %}}
{{< highlight bash >}}
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
33 / Webdav
\ "webdav"
...
Storage> webdav
{{< /highlight >}}
{{% /panel %}}
Next, rclone will prompt for the server address. Paste in the address copied from NextCloud in the previous section and hit enter.
Note that here `appstest` is used as the username, so be sure to copy value from your account.
{{% panel theme="info" header="Enter server address" %}}
{{< highlight bash >}}
URL of http host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Connect to example.com
\ "https://example.com"
url> https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/
{{< /highlight >}}
{{% /panel %}}
Rclone will then ask for the specific type of system. Choose the `nextcloud` option and hit return.
{{% panel theme="info" header="Enter storage service" %}}
{{< highlight bash >}}
Name of the Webdav site/service/software you are using
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Nextcloud
\ "nextcloud"
2 / Owncloud
\ "owncloud"
3 / Sharepoint
\ "sharepoint"
4 / Other site/service or software
\ "other"
vendor> 1
{{< /highlight >}}
{{% /panel %}}
Next, enter your HCC username and hit enter.
{{% panel theme="info" header="Enter username" %}}
{{< highlight bash >}}
User name
Enter a string value. Press Enter for the default ("").
user> appstest
{{< /highlight >}}
{{% /panel %}}
Rclone will prompt for whether or not to use a password; select `y` for yes.
Use the password copied from NextCloud in the previous section and paste it into the terminal.
Note that the password _will not show up after being pasted in_,so be careful to hit paste only once and then enter.
You will be prompted to enter the password twice.
{{% panel theme="info" header="Enter password" %}}
{{< highlight bash >}}
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:
{{< /highlight >}}
{{% /panel %}}
The default choices are used for the next two sections, so simply hit enter twice to proceed to the final
confirmation.
{{% panel theme="info" header="Use defaults" %}}
{{< highlight bash >}}
Bearer token instead of user/pass (eg a Macaroon)
Enter a string value. Press Enter for the default ("").
bearer_token>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
{{< /highlight >}}
{{% /panel %}}
Choose `y` to confirm the information is correct, then `q` to exit the config process and return to the terminal.
{{% panel theme="info" header="Final confirmation" %}}
{{< highlight bash >}}
[HCCNC]
type = webdav
url = https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/
vendor = nextcloud
user = appstest
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
HCCNC webdav
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[appstest@login.crane ~]$
{{< /highlight >}}
{{% /panel %}}
To verify things are correct, use the `rclone lsf` command to list the contents of your main folder.
{{% panel theme="info" header="Test Connection" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone lsf HCCNC:
Documents/
Nextcloud Manual.pdf
Photos/
Readme.md
Templates/
{{< /highlight >}}
{{% /panel %}}
### Transferring files
To upload or download files, use the `rclone copy` command. For example:
{{% panel theme="info" header="Transferring files" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone copy HCCNC:/SomeFile.txt ./
[appstest@login.crane ~]$ rclone copy ./SomeFile.txt HCCNC:/
{{< /highlight >}}
{{% /panel %}}
To download directories, use the `rclone copy` command and use directory names over file. This copies the contents of the folders, so you need to spec
ify a destination folder.
{{% panel theme="info" header="Download a directory from NextCloud" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone copy HCCNC:/my_hcc_dir ./my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
To upload a directory named `my_hcc_dir` to NextCloud, use `rclone copy`.
{{% panel theme="info" header="Upload a directory to NextCloud" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone copy ./my_hcc_dir HCCNC:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are
updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on NextCloud.
{{% panel theme="info" header="transfer.sh" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone sync ./my_hcc_dir HCCNC:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
Full details on the various Rclone commands are [here](https://rclone.org/docs/#basic-syntax).
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment