diff --git a/content/handling_data/data_transfer/using_rclone_with_nextcloud.md b/content/handling_data/data_transfer/using_rclone_with_nextcloud.md new file mode 100644 index 0000000000000000000000000000000000000000..6e34e9074f409860ee5fd088699b12f57e465cab --- /dev/null +++ b/content/handling_data/data_transfer/using_rclone_with_nextcloud.md @@ -0,0 +1,274 @@ ++++ +title = "Using Rclone with NextCloud" +description = "How to setup Rclone for using with HCC's NextCloud" +weight = 100 +hidden = true ++++ + +Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box, +Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task. +Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/). + +This tool can be used to transfer files between HCC's NextCloud instance and HCC's clusters, local machines, etc. using the WebDAV protocol. +This guide will show how to create dedicated credentials within NextCloud and configure rclone. + +## NextCloud setup + +### Copy the server address +Open a text editor or some other convenient way to store the URL and password that will be copied from NextCloud. +Log into HCC's NextCloud instance at https://hcc-nextcloud.unl.edu. You should see a page similar to this. + +{{< figure src="/images/nextcloud1.png" height="700" class="img-border">}} + +Click the _Settings_ option with the gear icon in the lower left corner of the page. + +{{< figure src="/images/nextcloud2.png" height="500" class="img-border">}} + +This will open an additional section directly underneath. Highlight the _entire_ string of text under the _WebDAV_ entry +and either using Ctrl-C or right-click to copy it. + +{{< figure src="/images/nextcloud3.png" height="400" class="img-border">}} + +Paste the text into a conventient editor. This is the server address to be used with rclone. + +### Create dedicated rclone password +Next, click the user account circle in the upper right corner and choose _Settings_ from the drop-down menu. + +{{< figure src="/images/nextcloud4.png" height="200" class="img-border">}} + +This will open a new page with your account settings. Click the _Security_ option on the left-hand menu. + +{{< figure src="/images/nextcloud5.png" height="400" class="img-border">}} + +This page will allow you to create a dedicated password to use for rclone transfers to/from NextCloud. +The username used is the same as your HCC username. + +Enter a name to use in the box and click _Create new app password_. + +{{< figure src="/images/nextcloud6.png" height="400" class="img-border">}} + +Once generated, click the clipboard icon to copy the password and paste it into your text editor. +_This is the only time the password will be shown._ + +{{< figure src="/images/nextcloud7.png" height="400" class="img-border">}} + +Finally, click the _Done_ button when finished. + +## Rclone setup + +{{% notice info %}} +The instructions here are written assuming rclone usage on an HCC cluster. The same procedure (starting with +the `rclone config` step) however can be used on any machine (i.e. local desktop) that rclone is installed on. +See [this link](https://rclone.org/install/) for details on installing rclone locally. +{{% /notice %}} + +### Create new remote +After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt: + +{{% panel theme="info" header="Load the Rclone module" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ module load rclone +{{< /highlight >}} +{{% /panel %}} + +To add a new remote in Rclone, run `rclone config`: + +{{% panel theme="info" header="Load the rclone config" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone config +{{< /highlight >}} +{{% /panel %}} + +Choose the new remote option and enter a name (here "HCCNC" is used). + +{{% panel theme="info" header="Create new remote" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone config +2021/11/11 22:38:42 NOTICE: Config file "/work/demo/appstest/.config/rclone/rclone.conf" not found - using defaults +No remotes found - make a new one +n) New remote +s) Set configuration password +q) Quit config +n/s/q> n +name> HCCNC +{{< /highlight >}} +{{% /panel %}} + +Rclone will print a list of supported storage systems and prompt for a choice. Type in `webdav` and hit enter. + +{{% panel theme="info" header="Choose storage type" %}} +{{< highlight bash >}} +Type of storage to configure. +Enter a string value. Press Enter for the default (""). +Choose a number from below, or type in your own value +... +33 / Webdav + \ "webdav" +... +Storage> webdav +{{< /highlight >}} +{{% /panel %}} + +Next, rclone will prompt for the server address. Paste in the address copied from NextCloud in the previous section and hit enter. +Note that here `appstest` is used as the username, so be sure to copy value from your account. + +{{% panel theme="info" header="Enter server address" %}} +{{< highlight bash >}} +URL of http host to connect to +Enter a string value. Press Enter for the default (""). +Choose a number from below, or type in your own value + 1 / Connect to example.com + \ "https://example.com" +url> https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/ +{{< /highlight >}} +{{% /panel %}} + +Rclone will then ask for the specific type of system. Choose the `nextcloud` option and hit return. + +{{% panel theme="info" header="Enter storage service" %}} +{{< highlight bash >}} +Name of the Webdav site/service/software you are using +Enter a string value. Press Enter for the default (""). +Choose a number from below, or type in your own value + 1 / Nextcloud + \ "nextcloud" + 2 / Owncloud + \ "owncloud" + 3 / Sharepoint + \ "sharepoint" + 4 / Other site/service or software + \ "other" +vendor> 1 +{{< /highlight >}} +{{% /panel %}} + +Next, enter your HCC username and hit enter. + +{{% panel theme="info" header="Enter username" %}} +{{< highlight bash >}} +User name +Enter a string value. Press Enter for the default (""). +user> appstest +{{< /highlight >}} +{{% /panel %}} + +Rclone will prompt for whether or not to use a password; select `y` for yes. +Use the password copied from NextCloud in the previous section and paste it into the terminal. +Note that the password _will not show up after being pasted in_,so be careful to hit paste only once and then enter. +You will be prompted to enter the password twice. + +{{% panel theme="info" header="Enter password" %}} +{{< highlight bash >}} +Password. +y) Yes type in my own password +g) Generate random password +n) No leave this optional password blank (default) +y/g/n> y +Enter the password: +password: +Confirm the password: +password: +{{< /highlight >}} +{{% /panel %}} + +The default choices are used for the next two sections, so simply hit enter twice to proceed to the final +confirmation. + +{{% panel theme="info" header="Use defaults" %}} +{{< highlight bash >}} +Bearer token instead of user/pass (eg a Macaroon) +Enter a string value. Press Enter for the default (""). +bearer_token> +Edit advanced config? (y/n) +y) Yes +n) No (default) +y/n> n +Remote config +-------------------- +{{< /highlight >}} +{{% /panel %}} + +Choose `y` to confirm the information is correct, then `q` to exit the config process and return to the terminal. + +{{% panel theme="info" header="Final confirmation" %}} +{{< highlight bash >}} +[HCCNC] +type = webdav +url = https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/ +vendor = nextcloud +user = appstest +pass = *** ENCRYPTED *** +-------------------- +y) Yes this is OK (default) +e) Edit this remote +d) Delete this remote +y/e/d> y +Current remotes: + +Name Type +==== ==== +HCCNC webdav + +e) Edit existing remote +n) New remote +d) Delete remote +r) Rename remote +c) Copy remote +s) Set configuration password +q) Quit config +e/n/d/r/c/s/q> q +[appstest@login.crane ~]$ +{{< /highlight >}} +{{% /panel %}} + +To verify things are correct, use the `rclone lsf` command to list the contents of your main folder. + +{{% panel theme="info" header="Test Connection" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone lsf HCCNC: +Documents/ +Nextcloud Manual.pdf +Photos/ +Readme.md +Templates/ +{{< /highlight >}} +{{% /panel %}} + +### Transferring files + +To upload or download files, use the `rclone copy` command. For example: + +{{% panel theme="info" header="Transferring files" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone copy HCCNC:/SomeFile.txt ./ +[appstest@login.crane ~]$ rclone copy ./SomeFile.txt HCCNC:/ +{{< /highlight >}} +{{% /panel %}} + +To download directories, use the `rclone copy` command and use directory names over file. This copies the contents of the folders, so you need to spec +ify a destination folder. + +{{% panel theme="info" header="Download a directory from NextCloud" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone copy HCCNC:/my_hcc_dir ./my_hcc_dir +{{< /highlight >}} +{{% /panel %}} + +To upload a directory named `my_hcc_dir` to NextCloud, use `rclone copy`. + +{{% panel theme="info" header="Upload a directory to NextCloud" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone copy ./my_hcc_dir HCCNC:/my_hcc_dir +{{< /highlight >}} +{{% /panel %}} + +Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are +updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on NextCloud. + +{{% panel theme="info" header="transfer.sh" %}} +{{< highlight bash >}} +[appstest@login.crane ~]$ rclone sync ./my_hcc_dir HCCNC:/my_hcc_dir +{{< /highlight >}} +{{% /panel %}} + +Full details on the various Rclone commands are [here](https://rclone.org/docs/#basic-syntax). diff --git a/static/images/nextcloud1.png b/static/images/nextcloud1.png new file mode 100644 index 0000000000000000000000000000000000000000..024742641ce5970856741b54d6b3206694b06473 Binary files /dev/null and b/static/images/nextcloud1.png differ diff --git a/static/images/nextcloud2.png b/static/images/nextcloud2.png new file mode 100644 index 0000000000000000000000000000000000000000..12c2223bbec4812b6632d81e923fe10adf8966ad Binary files /dev/null and b/static/images/nextcloud2.png differ diff --git a/static/images/nextcloud3.png b/static/images/nextcloud3.png new file mode 100644 index 0000000000000000000000000000000000000000..705da716af65e2ad65857e41d7e3b720c8ec945d Binary files /dev/null and b/static/images/nextcloud3.png differ diff --git a/static/images/nextcloud4.png b/static/images/nextcloud4.png new file mode 100644 index 0000000000000000000000000000000000000000..b0f779a6f5cbb4fc3598e11223b38b330752d28c Binary files /dev/null and b/static/images/nextcloud4.png differ diff --git a/static/images/nextcloud5.png b/static/images/nextcloud5.png new file mode 100644 index 0000000000000000000000000000000000000000..5992d8a4cce6ccace1874d730e9e296873c32253 Binary files /dev/null and b/static/images/nextcloud5.png differ diff --git a/static/images/nextcloud6.png b/static/images/nextcloud6.png new file mode 100644 index 0000000000000000000000000000000000000000..54152e9e9dec0d5d6aabf0a3dbdf2a8d14f7569c Binary files /dev/null and b/static/images/nextcloud6.png differ diff --git a/static/images/nextcloud7.png b/static/images/nextcloud7.png new file mode 100644 index 0000000000000000000000000000000000000000..15f9e1269d67b01be73c66470fcc96347f9f6eec Binary files /dev/null and b/static/images/nextcloud7.png differ diff --git a/static/images/nextcloud8.png b/static/images/nextcloud8.png new file mode 100644 index 0000000000000000000000000000000000000000..9fa5ea2fafbe5ce47f30d29eb77e846ac99e2b4d Binary files /dev/null and b/static/images/nextcloud8.png differ