Merge branch 'rclone-update' into 'master'

Updating rclone instructions to use OOD virtual desktop

See merge request !309

content/handling_data/data_transfer/using_rclone_with_hcc.md

@@ -12,26 +12,31 @@ This tool can be used to transfer files between HCC clusters and outside cloud p
 
 1.  You must be able to access your [NU Office365](http://office.com/) account before beginning this process.  Contact your local campus IT support if you need help with initial account setup.
 
-2.  Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize OneDrive**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
-
-3.  After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
+2.  Open a browser on your local machine and navigate to the [On-Demand portal]({{< relref "/open_ondemand" >}}) for the cluster of your choice.  We use Crane for this example: [https://crane-ood.unl.edu](https://crane-ood.unl.edu).  Select `Desktop` under `Interactive Apps` in the menu at the top of the page to get a virtual desktop on the cluster.  
+{{< figure src="/images/rclone_select_virtual_desktop.png" width="500" class="img-border">}}
+Scroll down to the bottom of the next page, and click on the blue `Launch` button.  When the resource is ready, click on the blue `Launch Desktop` button that appears on the next page.
+{{< figure src="/images/rclone_launch_desktop.png" width="500" class="img-border">}}
+On the virtual desktop, click on the `Terminal Emulator` icon at the bottom of the window to open up a command shell.
+{{< figure src="/images/rclone_open_terminal.png" width="500" class="img-border">}}
+
+3.  At the command prompt in the shell that opens, load the `rclone` module by entering the command below at the prompt:
 {{% panel theme="info" header="Load the Rclone module" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ module load rclone
+[demo03@c0809.crane ~]$ module load rclone
 {{< /highlight >}}
 {{% /panel %}}
 
 4.  We will need to start the basic configuration for OneDrive. To do this run `rclone config`:
 {{% panel theme="info" header="Load the rclone config" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone config
+[demo03@c0809 ~]$ rclone config
 {{< /highlight >}}
 {{% /panel %}}
 
 5.  In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "myOneDrive". Select `Microsoft OneDrive` by entering in the corresponding number, in our case `23`. Hit Enter for the client_id, client_secret, and Edit advanced config. **When you are prompted for auto config, select `n`**.  The terminal will stop at a `result>` prompt.  Proceed to the next step.
 {{% panel theme="info" header="Configure OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone config
+[demo03@c0809 ~]$ rclone config
 No remotes found - make a new one
 n) New remote
 s) Set configuration password
@@ -75,16 +80,13 @@ result>
 {{< /highlight >}}
 {{% /panel %}}
 
-6.  Now **switch to a terminal on your local machine** and run `rclone authorize "onedrive"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next".  You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!"  Return to your local rclone session.  You should see a message instructing you to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
-{{% panel theme="info" header="Authorize OneDrive on local machine" %}}
-{{< highlight bash >}}
-[myaccount@local.machine ~]$ rclone authorize "onedrive"
-{{< /highlight >}}
-{{% /panel %}}
-{{< figure src="/images/MicrosoftSign-In.png" height="500" class="img-border">}}
-{{% panel theme="info" header="Local Authorization after logging in with browser" %}}
+6.  Now **open up another terminal window on the virtual desktop** by clicking again on the `Terminal Emulator` icon at the bottom of the window.
+{{< figure src="/images/rclone_open_second_terminal.png" width="500" class="img-border">}}
+In the new shell that opens, load the `rclone` module like you did in the first shell with `module load rclone`, and then run `rclone authorize "onedrive"` at the command prompt.  You will be prompted to go to a 127.0.0.1 address in a web browser.
+{{% panel theme="info" header="Authorize OneDrive from the terminal" %}}
 {{< highlight script >}}
-[myaccount@local.machine ~]$ rclone authorize "onedrive"
+[demo03@c0809 ~]$ module load rclone
+[demo03@c0809 ~]$ rclone authorize "onedrive"
 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
 Log in and authorize Rclone for access
 Waiting for code...
@@ -94,9 +96,11 @@ Paste the following into your remote machine --->
 <---End paste
 {{< /highlight >}}
 {{% /panel %}}
-For other services, please refer to the [rclone documentation](https://rclone.org/). 
-
-7.  **Returning to the cluster**, now paste the code into the waiting `result` prompt.  Then select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`.  Press `y` at the next two prompts to confirm and `q` to exit.
+Right-click on the link and select `open link` from the menu.  A browser window will open on the virtual desktop.  On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next".  You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!" 
+{{< figure src="/images/rclone_authorize_success.png" width="500" class="img-border">}}
+Return to the terminal window where you ran the authorize command.  You should see a message instructing you to paste a line of code into your "remote machine" which is the first terminal you opened to run `rclone config`.
+{{< figure src="/images/rclone_authorize_code.png" width="500" class="img-border">}}
+Copy the text by highlighting it, right-clicking, and selecting `copy`.  Then, return to the first terminal with the waiting `result` prompt and paste the text by right-clicking and selecting `paste`, then press `enter`.  Next, select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`.  Press `y` at the next two prompts to confirm and `q` to exit.
 {{% panel theme="info" header="Authorize OnDemand on cluster" %}}
 {{< highlight bash >}}
 result> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
@@ -150,7 +154,7 @@ e/n/d/r/c/s/q> q
 Now test the connection by running the `ls` command.  You should see a listing of your OneDrive files.
 {{% panel theme="info" header="List contents of OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone ls myOneDrive:/
+[demo03@c0809 ~]$ rclone ls myOneDrive:/
 {{< /highlight >}}
 {{% /panel %}}
 
@@ -158,28 +162,28 @@ Now test the connection by running the `ls` command.  You should see a listing o
 8.  To upload or download files, use the `rclone copy` command.  For example:
 {{% panel theme="info" header="Transferring files" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone copy myOneDrive:/SomeFile.txt ./
-[demo2@login.crane ~]$ rclone copy ./SomeFile.txt myOneDrive:/
+[demo03@c0809 ~]$ rclone copy myOneDrive:/SomeFile.txt ./
+[demo03@c0809 ~]$ rclone copy ./SomeFile.txt myOneDrive:/
 {{< /highlight >}}
 {{% /panel %}}
 
 9.  To download directories, use the `rclone copy` command and use directory names over file.  This copies the contents of the folders, so you need to specify a destination folder.
 {{% panel theme="info" header="Download a directory from OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir
+[demo03@c0809 ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir
 {{< /highlight >}}
 {{% /panel %}}
 To upload a directory named `my_hcc_dir` to OneDrive, use `rclone copy`.
 {{% panel theme="info" header="Upload a directory to OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir
+[demo03@c0809 ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir
 {{< /highlight >}}
 {{% /panel %}}
 
 11.  Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on OneDrive.
 {{% panel theme="info" header="transfer.sh" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir
+[demo03@c0809 ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir
 {{< /highlight >}}
 {{% /panel %}}