diff --git a/content/handling_data/data_transfer/using_rclone_with_hcc.md b/content/handling_data/data_transfer/using_rclone_with_hcc.md index 2d55b98ce5890c1a111f18af909c97854f0fc9d8..6ee6b99d928db16659ff6a05409448c9fc226b00 100644 --- a/content/handling_data/data_transfer/using_rclone_with_hcc.md +++ b/content/handling_data/data_transfer/using_rclone_with_hcc.md @@ -12,26 +12,31 @@ This tool can be used to transfer files between HCC clusters and outside cloud p 1. You must be able to access your [NU Office365](http://office.com/) account before beginning this process. Contact your local campus IT support if you need help with initial account setup. -2. Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize OneDrive**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine. - -3. After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt: +2. Open a browser on your local machine and navigate to the [On-Demand portal]({{< relref "/open_ondemand" >}}) for the cluster of your choice. We use Crane for this example: [https://crane-ood.unl.edu](https://crane-ood.unl.edu). Select `Desktop` under `Interactive Apps` in the menu at the top of the page to get a virtual desktop on the cluster. +{{< figure src="/images/rclone_select_virtual_desktop.png" width="500" class="img-border">}} +Scroll down to the bottom of the next page, and click on the blue `Launch` button. When the resource is ready, click on the blue `Launch Desktop` button that appears on the next page. +{{< figure src="/images/rclone_launch_desktop.png" width="500" class="img-border">}} +On the virtual desktop, click on the `Terminal Emulator` icon at the bottom of the window to open up a command shell. +{{< figure src="/images/rclone_open_terminal.png" width="500" class="img-border">}} + +3. At the command prompt in the shell that opens, load the `rclone` module by entering the command below at the prompt: {{% panel theme="info" header="Load the Rclone module" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ module load rclone +[demo03@c0809.crane ~]$ module load rclone {{< /highlight >}} {{% /panel %}} 4. We will need to start the basic configuration for OneDrive. To do this run `rclone config`: {{% panel theme="info" header="Load the rclone config" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone config +[demo03@c0809 ~]$ rclone config {{< /highlight >}} {{% /panel %}} 5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "myOneDrive". Select `Microsoft OneDrive` by entering in the corresponding number, in our case `23`. Hit Enter for the client_id, client_secret, and Edit advanced config. **When you are prompted for auto config, select `n`**. The terminal will stop at a `result>` prompt. Proceed to the next step. {{% panel theme="info" header="Configure OneDrive" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone config +[demo03@c0809 ~]$ rclone config No remotes found - make a new one n) New remote s) Set configuration password @@ -75,16 +80,13 @@ result> {{< /highlight >}} {{% /panel %}} -6. Now **switch to a terminal on your local machine** and run `rclone authorize "onedrive"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next". You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!" Return to your local rclone session. You should see a message instructing you to paste a line of code from your local machine to the cluster and then to confirm that the config is correct. -{{% panel theme="info" header="Authorize OneDrive on local machine" %}} -{{< highlight bash >}} -[myaccount@local.machine ~]$ rclone authorize "onedrive" -{{< /highlight >}} -{{% /panel %}} -{{< figure src="/images/MicrosoftSign-In.png" height="500" class="img-border">}} -{{% panel theme="info" header="Local Authorization after logging in with browser" %}} +6. Now **open up another terminal window on the virtual desktop** by clicking again on the `Terminal Emulator` icon at the bottom of the window. +{{< figure src="/images/rclone_open_second_terminal.png" width="500" class="img-border">}} +In the new shell that opens, load the `rclone` module like you did in the first shell with `module load rclone`, and then run `rclone authorize "onedrive"` at the command prompt. You will be prompted to go to a 127.0.0.1 address in a web browser. +{{% panel theme="info" header="Authorize OneDrive from the terminal" %}} {{< highlight script >}} -[myaccount@local.machine ~]$ rclone authorize "onedrive" +[demo03@c0809 ~]$ module load rclone +[demo03@c0809 ~]$ rclone authorize "onedrive" If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize Rclone for access Waiting for code... @@ -94,9 +96,11 @@ Paste the following into your remote machine ---> <---End paste {{< /highlight >}} {{% /panel %}} -For other services, please refer to the [rclone documentation](https://rclone.org/). - -7. **Returning to the cluster**, now paste the code into the waiting `result` prompt. Then select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`. Press `y` at the next two prompts to confirm and `q` to exit. +Right-click on the link and select `open link` from the menu. A browser window will open on the virtual desktop. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next". You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!" +{{< figure src="/images/rclone_authorize_success.png" width="500" class="img-border">}} +Return to the terminal window where you ran the authorize command. You should see a message instructing you to paste a line of code into your "remote machine" which is the first terminal you opened to run `rclone config`. +{{< figure src="/images/rclone_authorize_code.png" width="500" class="img-border">}} +Copy the text by highlighting it, right-clicking, and selecting `copy`. Then, return to the first terminal with the waiting `result` prompt and paste the text by right-clicking and selecting `paste`, then press `enter`. Next, select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`. Press `y` at the next two prompts to confirm and `q` to exit. {{% panel theme="info" header="Authorize OnDemand on cluster" %}} {{< highlight bash >}} result> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"} @@ -150,7 +154,7 @@ e/n/d/r/c/s/q> q Now test the connection by running the `ls` command. You should see a listing of your OneDrive files. {{% panel theme="info" header="List contents of OneDrive" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone ls myOneDrive:/ +[demo03@c0809 ~]$ rclone ls myOneDrive:/ {{< /highlight >}} {{% /panel %}} @@ -158,28 +162,28 @@ Now test the connection by running the `ls` command. You should see a listing o 8. To upload or download files, use the `rclone copy` command. For example: {{% panel theme="info" header="Transferring files" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone copy myOneDrive:/SomeFile.txt ./ -[demo2@login.crane ~]$ rclone copy ./SomeFile.txt myOneDrive:/ +[demo03@c0809 ~]$ rclone copy myOneDrive:/SomeFile.txt ./ +[demo03@c0809 ~]$ rclone copy ./SomeFile.txt myOneDrive:/ {{< /highlight >}} {{% /panel %}} 9. To download directories, use the `rclone copy` command and use directory names over file. This copies the contents of the folders, so you need to specify a destination folder. {{% panel theme="info" header="Download a directory from OneDrive" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir +[demo03@c0809 ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir {{< /highlight >}} {{% /panel %}} To upload a directory named `my_hcc_dir` to OneDrive, use `rclone copy`. {{% panel theme="info" header="Upload a directory to OneDrive" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir +[demo03@c0809 ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir {{< /highlight >}} {{% /panel %}} 11. Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on OneDrive. {{% panel theme="info" header="transfer.sh" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir +[demo03@c0809 ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir {{< /highlight >}} {{% /panel %}} diff --git a/static/images/rclone_authorize_code.png b/static/images/rclone_authorize_code.png new file mode 100644 index 0000000000000000000000000000000000000000..dbe32636df8ffce09fb2f462036f70efc6b9224c Binary files /dev/null and b/static/images/rclone_authorize_code.png differ diff --git a/static/images/rclone_authorize_success.png b/static/images/rclone_authorize_success.png new file mode 100644 index 0000000000000000000000000000000000000000..ee8764a52c1c2e40d825be1a367cb97b8da38012 Binary files /dev/null and b/static/images/rclone_authorize_success.png differ diff --git a/static/images/rclone_launch_desktop.png b/static/images/rclone_launch_desktop.png new file mode 100644 index 0000000000000000000000000000000000000000..a005466ac91f3736cce45e210c5bb2b7973caf58 Binary files /dev/null and b/static/images/rclone_launch_desktop.png differ diff --git a/static/images/rclone_open_second_terminal.png b/static/images/rclone_open_second_terminal.png new file mode 100644 index 0000000000000000000000000000000000000000..15051d44a35ade7f87771a9e1c63eea895d5375a Binary files /dev/null and b/static/images/rclone_open_second_terminal.png differ diff --git a/static/images/rclone_open_terminal.png b/static/images/rclone_open_terminal.png new file mode 100644 index 0000000000000000000000000000000000000000..d1a608d9dc579b9d0411857e97b4862ab1f2ac46 Binary files /dev/null and b/static/images/rclone_open_terminal.png differ diff --git a/static/images/rclone_select_virtual_desktop.png b/static/images/rclone_select_virtual_desktop.png new file mode 100644 index 0000000000000000000000000000000000000000..50068b41450fb52add978ad30da82f118f2a9d9c Binary files /dev/null and b/static/images/rclone_select_virtual_desktop.png differ