Commit 6b1d12c6 authored by eharstad's avatar eharstad
Browse files

Merge branch 'rclone-update' into 'master'

Updating rclone instructions to use OOD virtual desktop

See merge request !309
parents 5851b216 0c995db6
......@@ -12,26 +12,31 @@ This tool can be used to transfer files between HCC clusters and outside cloud p
1. You must be able to access your [NU Office365](http://office.com/) account before beginning this process. Contact your local campus IT support if you need help with initial account setup.
2. Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize OneDrive**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
3. After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
2. Open a browser on your local machine and navigate to the [On-Demand portal]({{< relref "/open_ondemand" >}}) for the cluster of your choice. We use Crane for this example: [https://crane-ood.unl.edu](https://crane-ood.unl.edu). Select `Desktop` under `Interactive Apps` in the menu at the top of the page to get a virtual desktop on the cluster.
{{< figure src="/images/rclone_select_virtual_desktop.png" width="500" class="img-border">}}
Scroll down to the bottom of the next page, and click on the blue `Launch` button. When the resource is ready, click on the blue `Launch Desktop` button that appears on the next page.
{{< figure src="/images/rclone_launch_desktop.png" width="500" class="img-border">}}
On the virtual desktop, click on the `Terminal Emulator` icon at the bottom of the window to open up a command shell.
{{< figure src="/images/rclone_open_terminal.png" width="500" class="img-border">}}
3. At the command prompt in the shell that opens, load the `rclone` module by entering the command below at the prompt:
{{% panel theme="info" header="Load the Rclone module" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ module load rclone
[demo03@c0809.crane ~]$ module load rclone
{{< /highlight >}}
{{% /panel %}}
4. We will need to start the basic configuration for OneDrive. To do this run `rclone config`:
{{% panel theme="info" header="Load the rclone config" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone config
[demo03@c0809 ~]$ rclone config
{{< /highlight >}}
{{% /panel %}}
5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "myOneDrive". Select `Microsoft OneDrive` by entering in the corresponding number, in our case `23`. Hit Enter for the client_id, client_secret, and Edit advanced config. **When you are prompted for auto config, select `n`**. The terminal will stop at a `result>` prompt. Proceed to the next step.
{{% panel theme="info" header="Configure OneDrive" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone config
[demo03@c0809 ~]$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
......@@ -75,16 +80,13 @@ result>
{{< /highlight >}}
{{% /panel %}}
6. Now **switch to a terminal on your local machine** and run `rclone authorize "onedrive"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next". You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!" Return to your local rclone session. You should see a message instructing you to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
{{% panel theme="info" header="Authorize OneDrive on local machine" %}}
{{< highlight bash >}}
[myaccount@local.machine ~]$ rclone authorize "onedrive"
{{< /highlight >}}
{{% /panel %}}
{{< figure src="/images/MicrosoftSign-In.png" height="500" class="img-border">}}
{{% panel theme="info" header="Local Authorization after logging in with browser" %}}
6. Now **open up another terminal window on the virtual desktop** by clicking again on the `Terminal Emulator` icon at the bottom of the window.
{{< figure src="/images/rclone_open_second_terminal.png" width="500" class="img-border">}}
In the new shell that opens, load the `rclone` module like you did in the first shell with `module load rclone`, and then run `rclone authorize "onedrive"` at the command prompt. You will be prompted to go to a 127.0.0.1 address in a web browser.
{{% panel theme="info" header="Authorize OneDrive from the terminal" %}}
{{< highlight script >}}
[myaccount@local.machine ~]$ rclone authorize "onedrive"
[demo03@c0809 ~]$ module load rclone
[demo03@c0809 ~]$ rclone authorize "onedrive"
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize Rclone for access
Waiting for code...
......@@ -94,9 +96,11 @@ Paste the following into your remote machine --->
<---End paste
{{< /highlight >}}
{{% /panel %}}
For other services, please refer to the [rclone documentation](https://rclone.org/).
7. **Returning to the cluster**, now paste the code into the waiting `result` prompt. Then select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`. Press `y` at the next two prompts to confirm and `q` to exit.
Right-click on the link and select `open link` from the menu. A browser window will open on the virtual desktop. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next". You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!"
{{< figure src="/images/rclone_authorize_success.png" width="500" class="img-border">}}
Return to the terminal window where you ran the authorize command. You should see a message instructing you to paste a line of code into your "remote machine" which is the first terminal you opened to run `rclone config`.
{{< figure src="/images/rclone_authorize_code.png" width="500" class="img-border">}}
Copy the text by highlighting it, right-clicking, and selecting `copy`. Then, return to the first terminal with the waiting `result` prompt and paste the text by right-clicking and selecting `paste`, then press `enter`. Next, select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`. Press `y` at the next two prompts to confirm and `q` to exit.
{{% panel theme="info" header="Authorize OnDemand on cluster" %}}
{{< highlight bash >}}
result> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
......@@ -150,7 +154,7 @@ e/n/d/r/c/s/q> q
Now test the connection by running the `ls` command. You should see a listing of your OneDrive files.
{{% panel theme="info" header="List contents of OneDrive" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone ls myOneDrive:/
[demo03@c0809 ~]$ rclone ls myOneDrive:/
{{< /highlight >}}
{{% /panel %}}
......@@ -158,28 +162,28 @@ Now test the connection by running the `ls` command. You should see a listing o
8. To upload or download files, use the `rclone copy` command. For example:
{{% panel theme="info" header="Transferring files" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone copy myOneDrive:/SomeFile.txt ./
[demo2@login.crane ~]$ rclone copy ./SomeFile.txt myOneDrive:/
[demo03@c0809 ~]$ rclone copy myOneDrive:/SomeFile.txt ./
[demo03@c0809 ~]$ rclone copy ./SomeFile.txt myOneDrive:/
{{< /highlight >}}
{{% /panel %}}
9. To download directories, use the `rclone copy` command and use directory names over file. This copies the contents of the folders, so you need to specify a destination folder.
{{% panel theme="info" header="Download a directory from OneDrive" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir
[demo03@c0809 ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
To upload a directory named `my_hcc_dir` to OneDrive, use `rclone copy`.
{{% panel theme="info" header="Upload a directory to OneDrive" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir
[demo03@c0809 ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
11. Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on OneDrive.
{{% panel theme="info" header="transfer.sh" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir
[demo03@c0809 ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir
{{< /highlight >}}
{{% /panel %}}
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment