diff --git a/content/handling_data/data_storage/integrating_box_with_hcc.md b/content/handling_data/data_storage/integrating_box_with_hcc.md index c047b61a1b0a261068da15234a829073cbfc1556..d3f2d8496a3398d98db7deb20062fd7b9562d202 100644 --- a/content/handling_data/data_storage/integrating_box_with_hcc.md +++ b/content/handling_data/data_storage/integrating_box_with_hcc.md @@ -19,7 +19,7 @@ in good network traffic conditions. Users can use a tool called lftp to transf 1. Login to your [UNK Box.com](https://unk.account.box.com/), [UNL Box.com](https://unl.account.box.com/), or [UNO Box.com](https://unomaha.account.box.com/) account. -2. Since we are going to be using [webdav](https://en.wikipedia.org/wiki/WebDAV) protocol to access your [Box.com](https://www.box.com/) storage, you need to create an **External Password**. In the Box.com interface, you can create it at **Account Settings > Account > Authentication > Create Password.** +2. Since we are going to be using [ftps](https://en.wikipedia.org/wiki/FTPS) protocol to access your [Box.com](https://www.box.com/) storage, you need to create an **External Password**. In the Box.com interface, you can create it at **Account Settings > Account > Authentication > Create Password.** {{< figure src="/images/box_create_external_password.png" class="img-border" >}} 3. After logging into the cluster of your choice, load the `lftp` module by entering the command below at the prompt: diff --git a/content/handling_data/data_transfer/using_rclone_with_hcc.md b/content/handling_data/data_transfer/using_rclone_with_hcc.md index a81a0899f0166eae262fefed978cee24d099d60e..745021ab80dfb58bf3c793479770f6bfaab242d3 100644 --- a/content/handling_data/data_transfer/using_rclone_with_hcc.md +++ b/content/handling_data/data_transfer/using_rclone_with_hcc.md @@ -10,9 +10,9 @@ This tool can be used to transfer files between HCC clusters and outside cloud p --- ### Setup RClone -1. You need to create your UNL [Box.com](https://www.box.com/) account [here](https://box.unl.edu/). +1. You must be able to access your [UNL Box.com](https://unl.box.com/), [UNK Box.com](https://unk.account.box.com/), [UNO Box.com](https://unomaha.account.box.com/), or [UNMC Box.com](https://unmc.account.box.com/) before beginning this process. Contact your local campus support if you need help with initial account setup. -2. Due to the clusters being remote machines, Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize box. Some services, such as Google Drive, do not require Rclone to be installed on your local machine. +2. Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize Box**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine. 3. After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt: {{% panel theme="info" header="Load the Rclone module" %}} @@ -28,7 +28,7 @@ This tool can be used to transfer files between HCC clusters and outside cloud p {{< /highlight >}} {{% /panel %}} -5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "UNLBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. When you are prompted for auto config, select `N` and switch to a terminal on your local machine: +5. In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "MyBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. **When you are prompted for auto config, select `N`**. {{% panel theme="info" header="Configure box" %}} {{< highlight bash >}} [demo2@login.crane ~]$ rclone config @@ -37,7 +37,7 @@ n) New remote s) Set configuration password q) Quit config n/s/q> n -name> UNLBox +name> MyBox Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value @@ -67,8 +67,8 @@ Enter a signed integer. Press Enter for the default ("100"). commit_retries> Remote config Use auto config? - * Say Y if not sure - * Say N if you are working on a remote or headless machine + \* Say Y if not sure + \* Say N if you are working on a remote or headless machine y) Yes n) No y/n> n @@ -80,16 +80,16 @@ result> {{< /highlight >}} {{% /panel %}} -6. Run `rclone authorize "box"` on the local machine. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your UNL e-mail address. You will be taken to sign into UNL's Box using your **Canvas** credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct. -{{% panel theme="info" header="List contents of Box" %}} +6. Now **switch to a terminal on your local machine** and run `rclone authorize "box"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your NU e-mail address. You will be taken to sign into your campus' Box instance using your TrueYou/campus credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct. +{{% panel theme="info" header="Authorize Box on local machine" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone authorize "box" +[myaccount@local.machine ~]$ rclone authorize "box" {{< /highlight >}} {{% /panel %}} {{< figure src="/images/BoxSSO.png" height="500" class="img-border">}} -{{% panel theme="info" header="Local Config" %}} -{{< highlight bash >}} -[demo2@local.machine ~]$ rclone authorize "box" +{{% panel theme="info" header="Local Authorization" %}} +{{< highlight script >}} +[myaccount@local.machine ~]$ rclone authorize "box" If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize Rclone for access Waiting for code... @@ -100,10 +100,10 @@ For other services, please refer to the [rclone documentation](https://rclone.or -7. Test the connection by running the `ls` command. You should see a listing of your Box files. +7. **Returning to the cluster**, test the connection by running the `ls` command. You should see a listing of your Box files. {{% panel theme="info" header="List contents of Box" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone ls UNLBox:/ +[demo2@login.crane ~]$ rclone ls MyBox:/ {{< /highlight >}} {{% /panel %}} @@ -111,28 +111,28 @@ For other services, please refer to the [rclone documentation](https://rclone.or 8. To upload or download files, use the `clone` command. For example: {{% panel theme="info" header="Transferring files" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone copy UNLBox:/SomeFile.txt ./ -[demo2@login.crane ~]$ rclone copy ./SomeFile.txt UNLBox:/ +[demo2@login.crane ~]$ rclone copy MyBox:/SomeFile.txt ./ +[demo2@login.crane ~]$ rclone copy ./SomeFile.txt MyBox:/ {{< /highlight >}} {{% /panel %}} 9. To download directories, use the `clone` command and use directory names over file. This copies the contents of the folders, so you need to specify a destination folder. {{% panel theme="info" header="Download a directory from Box" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone copy UNLBox:/my_hcc_dir ./my_hcc_dir +[demo2@login.crane ~]$ rclone copy MyBox:/my_hcc_dir ./my_hcc_dir {{< /highlight >}} {{% /panel %}} To upload a directory named `my_hcc_dir` to Box, use `clone`. {{% panel theme="info" header="Upload a directory to Box" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone copy ./my_hcc_dir UNLBox:/my_hcc_dir +[demo2@login.crane ~]$ rclone copy ./my_hcc_dir MyBox:/my_hcc_dir {{< /highlight >}} {{% /panel %}} 11. Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The exmaple below would sync the files of the local directory to the remote directory on box. {{% panel theme="info" header="transfer.sh" %}} {{< highlight bash >}} -[demo2@login.crane ~]$ rclone sync ./my_hcc_dir UNLBox:/my_hcc_dir +[demo2@login.crane ~]$ rclone sync ./my_hcc_dir MyBox:/my_hcc_dir {{< /highlight >}} {{% /panel %}}