Add page for rclone+NextCloud.

content/handling_data/data_transfer/using_rclone_with_nextcloud.md

++++
+title = "Using Rclone with NextCloud"
+description = "How to setup Rclone for using with HCC's NextCloud"
+weight = 100
+hidden = true
++++
+
+Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box,
+Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task.
+Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/).
+
+This tool can be used to transfer files between HCC's NextCloud instance and HCC's clusters, local machines, etc. using the WebDAV protocol. 
+This guide will show how to create dedicated credentials within NextCloud and configure rclone.
+
+## NextCloud setup
+
+### Copy the server address
+Open a text editor or some other convenient way to store the URL and password that will be copied from NextCloud.  
+Log into HCC's NextCloud instance at https://hcc-nextcloud.unl.edu. You should see a page similar to this.
+
+{{< figure src="/images/nextcloud1.png" height="700" class="img-border">}}
+
+Click the _Settings_ option with the gear icon in the lower left corner of the page.
+
+{{< figure src="/images/nextcloud2.png" height="500" class="img-border">}}
+
+This will open an additional section directly underneath. Highlight the _entire_ string of text under the _WebDAV_ entry
+and either using Ctrl-C or right-click to copy it.
+
+{{< figure src="/images/nextcloud3.png" height="400" class="img-border">}}
+
+Paste the text into a conventient editor. This is the server address to be used with rclone.
+
+### Create dedicated rclone password
+Next, click the user account circle in the upper right corner and choose _Settings_ from the drop-down menu.
+
+{{< figure src="/images/nextcloud4.png" height="200" class="img-border">}}
+
+This will open a new page with your account settings. Click the _Security_ option on the left-hand menu.
+
+{{< figure src="/images/nextcloud5.png" height="400" class="img-border">}}
+
+This page will allow you to create a dedicated password to use for rclone transfers to/from NextCloud.
+The username used is the same as your HCC username.
+
+Enter a name to use in the box and click _Create new app password_.
+
+{{< figure src="/images/nextcloud6.png" height="400" class="img-border">}}
+
+Once generated, click the clipboard icon to copy the password and paste it into your text editor.
+_This is the only time the password will be shown._
+
+{{< figure src="/images/nextcloud7.png" height="400" class="img-border">}}
+
+Finally, click the _Done_ button when finished.
+
+## Rclone setup
+
+{{% notice info %}}
+The instructions here are written assuming rclone usage on an HCC cluster.  The same procedure (starting with
+the `rclone config` step) however can be used on any machine (i.e. local desktop) that rclone is installed on.
+See [this link](https://rclone.org/install/) for details on installing rclone locally.
+{{% /notice %}}
+
+### Create new remote
+After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
+
+{{% panel theme="info" header="Load the Rclone module" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ module load rclone
+{{< /highlight >}}
+{{% /panel %}}
+
+To add a new remote in Rclone, run `rclone config`:
+
+{{% panel theme="info" header="Load the rclone config" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone config
+{{< /highlight >}}
+{{% /panel %}}
+
+Choose the new remote option and enter a name (here "HCCNC" is used).
+
+{{% panel theme="info" header="Create new remote" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone config
+2021/11/11 22:38:42 NOTICE: Config file "/work/demo/appstest/.config/rclone/rclone.conf" not found - using defaults
+No remotes found - make a new one
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> HCCNC
+{{< /highlight >}}
+{{% /panel %}}
+
+Rclone will print a list of supported storage systems and prompt for a choice. Type in `webdav` and hit enter.
+
+{{% panel theme="info" header="Choose storage type" %}}
+{{< highlight bash >}}
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+...
+33 / Webdav
+   \ "webdav"
+...
+Storage> webdav
+{{< /highlight >}}
+{{% /panel %}}
+
+Next, rclone will prompt for the server address. Paste in the address copied from NextCloud in the previous section and hit enter.
+Note that here `appstest` is used as the username, so be sure to copy value from your account.
+
+{{% panel theme="info" header="Enter server address" %}}
+{{< highlight bash >}}
+URL of http host to connect to
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Connect to example.com
+   \ "https://example.com"
+url> https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/
+{{< /highlight >}}
+{{% /panel %}}
+
+Rclone will then ask for the specific type of system. Choose the `nextcloud` option and hit return.
+
+{{% panel theme="info" header="Enter storage service" %}}
+{{< highlight bash >}}
+Name of the Webdav site/service/software you are using
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Nextcloud
+   \ "nextcloud"
+ 2 / Owncloud
+   \ "owncloud"
+ 3 / Sharepoint
+   \ "sharepoint"
+ 4 / Other site/service or software
+   \ "other"
+vendor> 1
+{{< /highlight >}}
+{{% /panel %}}
+
+Next, enter your HCC username and hit enter.
+
+{{% panel theme="info" header="Enter username" %}}
+{{< highlight bash >}}
+User name
+Enter a string value. Press Enter for the default ("").
+user> appstest
+{{< /highlight >}}
+{{% /panel %}}
+
+Rclone will prompt for whether or not to use a password; select `y` for yes. 
+Use the password copied from NextCloud in the previous section and paste it into the terminal. 
+Note that the password _will not show up after being pasted in_,so be careful to hit paste only once and then enter.
+You will be prompted to enter the password twice.
+
+{{% panel theme="info" header="Enter password" %}}
+{{< highlight bash >}}
+Password.
+y) Yes type in my own password
+g) Generate random password
+n) No leave this optional password blank (default)
+y/g/n> y
+Enter the password:
+password:
+Confirm the password:
+password:
+{{< /highlight >}}
+{{% /panel %}}
+
+The default choices are used for the next two sections, so simply hit enter twice to proceed to the final
+confirmation.
+
+{{% panel theme="info" header="Use defaults" %}}
+{{< highlight bash >}}
+Bearer token instead of user/pass (eg a Macaroon)
+Enter a string value. Press Enter for the default ("").
+bearer_token>
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+{{< /highlight >}}
+{{% /panel %}}
+
+Choose `y` to confirm the information is correct, then `q` to exit the config process and return to the terminal.
+
+{{% panel theme="info" header="Final confirmation" %}}
+{{< highlight bash >}}
+[HCCNC]
+type = webdav
+url = https://hcc-nextcloud.unl.edu/remote.php/dav/files/appstest/
+vendor = nextcloud
+user = appstest
+pass = *** ENCRYPTED ***
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name                 Type
+====                 ====
+HCCNC                webdav
+
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
+[appstest@login.crane ~]$
+{{< /highlight >}}
+{{% /panel %}}
+
+To verify things are correct, use the `rclone lsf` command to list the contents of your main folder.
+
+{{% panel theme="info" header="Test Connection" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone lsf HCCNC:
+Documents/
+Nextcloud Manual.pdf
+Photos/
+Readme.md
+Templates/
+{{< /highlight >}}
+{{% /panel %}}
+
+### Transferring files
+
+To upload or download files, use the `rclone copy` command.  For example:
+
+{{% panel theme="info" header="Transferring files" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone copy HCCNC:/SomeFile.txt ./
+[appstest@login.crane ~]$ rclone copy ./SomeFile.txt HCCNC:/
+{{< /highlight >}}
+{{% /panel %}}
+
+To download directories, use the `rclone copy` command and use directory names over file.  This copies the contents of the folders, so you need to spec
+ify a destination folder.
+
+{{% panel theme="info" header="Download a directory from NextCloud" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone copy HCCNC:/my_hcc_dir ./my_hcc_dir
+{{< /highlight >}}
+{{% /panel %}}
+
+To upload a directory named `my_hcc_dir` to NextCloud, use `rclone copy`.
+
+{{% panel theme="info" header="Upload a directory to NextCloud" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone copy ./my_hcc_dir HCCNC:/my_hcc_dir
+{{< /highlight >}}
+{{% /panel %}}
+
+Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are
+updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on NextCloud.
+
+{{% panel theme="info" header="transfer.sh" %}}
+{{< highlight bash >}}
+[appstest@login.crane ~]$ rclone sync ./my_hcc_dir HCCNC:/my_hcc_dir
+{{< /highlight >}}
+{{% /panel %}}
+
+Full details on the various Rclone commands are [here](https://rclone.org/docs/#basic-syntax).