using_rclone_with_nextcloud.md 9.15 KB
Newer Older
Adam Caprez's avatar
Adam Caprez committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
+++
title = "Using Rclone with NextCloud"
description = "How to setup Rclone for using with HCC's NextCloud"
weight = 100
hidden = true
+++

Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box,
Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task.
Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/).

This tool can be used to transfer files between HCC's NextCloud instance and HCC's clusters, local machines, etc. using the WebDAV protocol. 
This guide will show how to create dedicated credentials within NextCloud and configure rclone.

## NextCloud setup

### Copy the server address
Open a text editor or some other convenient way to store the URL and password that will be copied from NextCloud.  
Log into HCC's NextCloud instance at https://hcc-nextcloud.unl.edu. You should see a page similar to this.

{{< figure src="/images/nextcloud1.png" height="700" class="img-border">}}

Click the _Settings_ option with the gear icon in the lower left corner of the page.

{{< figure src="/images/nextcloud2.png" height="500" class="img-border">}}

This will open an additional section directly underneath. Highlight the _entire_ string of text under the _WebDAV_ entry
and either using Ctrl-C or right-click to copy it.

{{< figure src="/images/nextcloud3.png" height="400" class="img-border">}}

Paste the text into a conventient editor. This is the server address to be used with rclone.

### Create dedicated rclone password
Next, click the user account circle in the upper right corner and choose _Settings_ from the drop-down menu.

{{< figure src="/images/nextcloud4.png" height="200" class="img-border">}}

This will open a new page with your account settings. Click the _Security_ option on the left-hand menu.

{{< figure src="/images/nextcloud5.png" height="400" class="img-border">}}

This page will allow you to create a dedicated password to use for rclone transfers to/from NextCloud.
The username used is the same as your HCC username.

Enter a name to use in the box and click _Create new app password_.

{{< figure src="/images/nextcloud6.png" height="400" class="img-border">}}

Once generated, click the clipboard icon to copy the password and paste it into your text editor.
_This is the only time the password will be shown._

{{< figure src="/images/nextcloud7.png" height="400" class="img-border">}}

Finally, click the _Done_ button when finished.

## Rclone setup

{{% notice info %}}
The instructions here are written assuming rclone usage on an HCC cluster.  The same procedure (starting with
the `rclone config` step) however can be used on any machine (i.e. local desktop) that rclone is installed on.
See [this link](https://rclone.org/install/) for details on installing rclone locally.
{{% /notice %}}

### Create new remote
After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:

{{% panel theme="info" header="Load the Rclone module" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ module load rclone
{{< /highlight >}}
{{% /panel %}}

To add a new remote in Rclone, run `rclone config`:

{{% panel theme="info" header="Load the rclone config" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone config
{{< /highlight >}}
{{% /panel %}}

Choose the new remote option and enter a name (here "HCCNC" is used).

{{% panel theme="info" header="Create new remote" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone config
2021/11/11 22:38:42 NOTICE: Config file "/work/demo/appstest/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> HCCNC
{{< /highlight >}}
{{% /panel %}}

Rclone will print a list of supported storage systems and prompt for a choice. Type in `webdav` and hit enter.

{{% panel theme="info" header="Choose storage type" %}}
{{< highlight bash >}}
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
33 / Webdav
   \ "webdav"
...
Storage> webdav
{{< /highlight >}}
{{% /panel %}}

Next, rclone will prompt for the server address. Paste in the address copied from NextCloud in the previous section and hit enter.
Note that here `appstest` is used as the username, so be sure to copy value from your account.

{{% panel theme="info" header="Enter server address" %}}
{{< highlight bash >}}
URL of http host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Connect to example.com
   \ "https://example.com"
url> https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/
{{< /highlight >}}
{{% /panel %}}

Rclone will then ask for the specific type of system. Choose the `nextcloud` option and hit return.

{{% panel theme="info" header="Enter storage service" %}}
{{< highlight bash >}}
Name of the Webdav site/service/software you are using
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Nextcloud
   \ "nextcloud"
 2 / Owncloud
   \ "owncloud"
 3 / Sharepoint
   \ "sharepoint"
 4 / Other site/service or software
   \ "other"
vendor> 1
{{< /highlight >}}
{{% /panel %}}

Next, enter your HCC username and hit enter.

{{% panel theme="info" header="Enter username" %}}
{{< highlight bash >}}
User name
Enter a string value. Press Enter for the default ("").
user> appstest
{{< /highlight >}}
{{% /panel %}}

Rclone will prompt for whether or not to use a password; select `y` for yes. 
Use the password copied from NextCloud in the previous section and paste it into the terminal. 
Note that the password _will not show up after being pasted in_,so be careful to hit paste only once and then enter.
You will be prompted to enter the password twice.

{{% panel theme="info" header="Enter password" %}}
{{< highlight bash >}}
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:
{{< /highlight >}}
{{% /panel %}}

The default choices are used for the next two sections, so simply hit enter twice to proceed to the final
confirmation.

{{% panel theme="info" header="Use defaults" %}}
{{< highlight bash >}}
Bearer token instead of user/pass (eg a Macaroon)
Enter a string value. Press Enter for the default ("").
bearer_token>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
{{< /highlight >}}
{{% /panel %}}

Choose `y` to confirm the information is correct, then `q` to exit the config process and return to the terminal.

{{% panel theme="info" header="Final confirmation" %}}
{{< highlight bash >}}
[HCCNC]
type = webdav
url = https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/
vendor = nextcloud
user = appstest
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
HCCNC                webdav

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[appstest@login.crane ~]$
{{< /highlight >}}
{{% /panel %}}

To verify things are correct, use the `rclone lsf` command to list the contents of your main folder.

{{% panel theme="info" header="Test Connection" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone lsf HCCNC:
Documents/
Nextcloud Manual.pdf
Photos/
Readme.md
Templates/
{{< /highlight >}}
{{% /panel %}}

### Transferring files

To upload or download files, use the `rclone copy` command.  For example:

{{% panel theme="info" header="Transferring files" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone copy HCCNC:/SomeFile.txt ./
[appstest@login.crane ~]$ rclone copy ./SomeFile.txt HCCNC:/
{{< /highlight >}}
{{% /panel %}}

To download directories, use the `rclone copy` command and use directory names over file.  This copies the contents of the folders, so you need to spec
ify a destination folder.

{{% panel theme="info" header="Download a directory from NextCloud" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone copy HCCNC:/my_hcc_dir ./my_hcc_dir
{{< /highlight >}}
{{% /panel %}}

To upload a directory named `my_hcc_dir` to NextCloud, use `rclone copy`.

{{% panel theme="info" header="Upload a directory to NextCloud" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone copy ./my_hcc_dir HCCNC:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}

Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are
updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on NextCloud.

{{% panel theme="info" header="transfer.sh" %}}
{{< highlight bash >}}
[appstest@login.crane ~]$ rclone sync ./my_hcc_dir HCCNC:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}

Full details on the various Rclone commands are [here](https://rclone.org/docs/#basic-syntax).