Merge branch 'onedrive' into 'master'

updating rclone instructions to use onedrive in stead of box

See merge request !258

content/handling_data/data_transfer/using_rclone_with_hcc.md

@@ -6,13 +6,13 @@ weight =60
 
 Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box, Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task. Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/). 
 
-This tool can be used to transfer files between HCC clusters and outside cloud providers, such as Box.
+This tool can be used to transfer files between HCC clusters and outside cloud providers, such as OneDrive.
 ---
 ### Setup RClone
 
-1.  You must be able to access your [UNL Box.com](https://unl.box.com/), [UNK Box.com](https://unk.account.box.com/), [UNO Box.com](https://unomaha.account.box.com/), or [UNMC Box.com](https://unmc.account.box.com/) account before beginning this process.  Contact your local campus support if you need help with initial account setup.
+1.  You must be able to access your [NU Office365 accunt](http://office.com/) account before beginning this process.  Contact your local campus IT support if you need help with initial account setup.
 
-2.  Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize Box**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
+2.  Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize OneDrive**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
 
 3.  After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
 {{% panel theme="info" header="Load the Rclone module" %}}
@@ -21,15 +21,15 @@ This tool can be used to transfer files between HCC clusters and outside cloud p
 {{< /highlight >}}
 {{% /panel %}}
 
-4.  We will need to start the basic configuration for box. To do this run `rclone config`:
+4.  We will need to start the basic configuration for OneDrive. To do this run `rclone config`:
 {{% panel theme="info" header="Load the rclone config" %}}
 {{< highlight bash >}}
 [demo2@login.crane ~]$ rclone config
 {{< /highlight >}}
 {{% /panel %}}
 
-5.  In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "MyBox". Select box by entering in the corresponding number, in our case `6`. Hit Enter for the client_id and client_secret and `y` for Edit advanced config. Due to the file size limit with Box, set the upload_cutoff to `15G` and press enter, also leaving the commit_retries as the default by pressing Enter. **When you are prompted for auto config, select `N`**.  The terminal will stop at a `result>` prompt.  Proceed to the next step.
-{{% panel theme="info" header="Configure box" %}}
+5.  In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "myOneDrive". Select `Microsoft OneDrive` by entering in the corresponding number, in our case `23`. Hit Enter for the client_id, client_secret, and Edit advanced config. **When you are prompted for auto config, select `n`**.  The terminal will stop at a `result>` prompt.  Proceed to the next step.
+{{% panel theme="info" header="Configure OneDrive" %}}
 {{< highlight bash >}}
 [demo2@login.crane ~]$ rclone config
 No remotes found - make a new one
@@ -37,59 +37,54 @@ n) New remote
 s) Set configuration password
 q) Quit config
 n/s/q> n
-name> MyBox
+name> myOneDrive
 Type of storage to configure.
 Enter a string value. Press Enter for the default ("").
 Choose a number from below, or type in your own value
 
- 6 / Box
-   \ "box"
+23 / Microsoft OneDrive
+   \ "onedrive"
 
-Storage> 6
+Storage> 23
+** See help for onedrive backend at: https://rclone.org/onedrive/ **
 
-Box App Client Id.
+Microsoft App Client Id
 Leave blank normally.
 Enter a string value. Press Enter for the default ("").
-client_id>
-Box App Client Secret
+client_id> 
+Microsoft App Client Secret
 Leave blank normally.
 Enter a string value. Press Enter for the default ("").
-client_secret>
+client_secret> 
 Edit advanced config? (y/n)
 y) Yes
-n) No
-y/n> y
-Cutoff for switching to multipart upload (>= 50MB).
-Enter a size with suffix k,M,G,T. Press Enter for the default ("50M").
-upload_cutoff> 15G
-Max number of times to try committing a multipart file.
-Enter a signed integer. Press Enter for the default ("100").
-commit_retries>
+n) No (default)
+y/n> 
 Remote config
 Use auto config?
- \* Say Y if not sure
- \* Say N if you are working on a remote or headless machine
-y) Yes
+ * Say Y if not sure
+ * Say N if you are working on a remote or headless machine
+y) Yes (default)
 n) No
 y/n> n
-For this to work, you will need Rclone available on a machine that has a web browser available.
-Execute the following on your machine:
-        rclone authorize "box"
+For this to work, you will need rclone available on a machine that has a web browser available.
+Execute the following on your machine (same rclone version recommended) :
+	rclone authorize "onedrive"
 Then paste the result below:
-result>
+result> 
 {{< /highlight >}}
 {{% /panel %}}
 
-6.  Now **switch to a terminal on your local machine** and run `rclone authorize "box"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. Select `Use Single Sign On(SSO)` at the bottom and then enter in your NU e-mail address. You will be taken to sign into your campus' Box instance using your TrueYou/campus credentials. Select `Grant access to Box`. You will be told to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
-{{% panel theme="info" header="Authorize Box on local machine" %}}
+6.  Now **switch to a terminal on your local machine** and run `rclone authorize "onedrive"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next".  You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!"  Return to your local rclone session.  You should see a message instructing you to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
+{{% panel theme="info" header="Authorize OneDrive on local machine" %}}
 {{< highlight bash >}}
-[myaccount@local.machine ~]$ rclone authorize "box"
+[myaccount@local.machine ~]$ rclone authorize "onedrive"
 {{< /highlight >}}
 {{% /panel %}}
-{{< figure src="/images/BoxSSO.png" height="500" class="img-border">}}
+{{< figure src="/images/MicrosoftSign-In.png" height="500" class="img-border">}}
 {{% panel theme="info" header="Local Authorization after logging in with browser" %}}
 {{< highlight script >}}
-[myaccount@local.machine ~]$ rclone authorize "box"
+[myaccount@local.machine ~]$ rclone authorize "onedrive"
 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
 Log in and authorize Rclone for access
 Waiting for code...
@@ -101,18 +96,38 @@ Paste the following into your remote machine --->
 {{% /panel %}}
 For other services, please refer to the [rclone documentation](https://rclone.org/). 
 
-7.  **Returning to the cluster**, now paste the code into the waiting `result` prompt, then press `y` to confirm and `q` to exit.
-{{% panel theme="info" header="Authorize Box on cluster" %}}
+7.  **Returning to the cluster**, now paste the code into the waiting `result` prompt.  Then select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`.  Press `y` at the next two prompts to confirm and `q` to exit.
+{{% panel theme="info" header="Authorize OnDemand on cluster" %}}
 {{< highlight bash >}}
 result> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
+Choose a number from below, or type in an existing value
+ 1 / OneDrive Personal or Business
+   \ "onedrive"
+ 2 / Root Sharepoint site
+   \ "sharepoint"
+ 3 / Type in driveID
+   \ "driveid"
+ 4 / Type in SiteID
+   \ "siteid"
+ 5 / Search a Sharepoint site
+   \ "search"
+Your choice> 1
+Found 1 drives, please select the one you want to use:
+0: OneDrive (business) id=b!laCd4ZJ54U-[...]
+Chose drive to use:> 0
+Found drive 'root' of type 'business', URL: https://uofnelincoln-my.sharepoint.com/personal/demo_unl_edu/Documents
+Is that okay?
+y) Yes (default)
+n) No
+y/n> y
 --------------------
-[Box]
-type = box
-box_sub_type = user
-upload_cutoff = 15G
-token = {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
+[myOneDrive]
+type = onedrive
+token = {"access_token": ...}
+drive_id = b!laCd4ZJ54U-[...]
+drive_type = business
 --------------------
-y) Yes this is OK
+y) Yes this is OK (default)
 e) Edit this remote
 d) Delete this remote
 y/e/d> y
@@ -120,7 +135,7 @@ Current remotes:
 
 Name                 Type
 ====                 ====
-Box                  box
+myOneDrive           onedrive
 
 e) Edit existing remote
 n) New remote
@@ -132,39 +147,39 @@ q) Quit config
 e/n/d/r/c/s/q> q
 {{< /highlight >}}
 {{% /panel %}}
-Now test the connection by running the `ls` command.  You should see a listing of your Box files.
-{{% panel theme="info" header="List contents of Box" %}}
+Now test the connection by running the `ls` command.  You should see a listing of your OneDrive files.
+{{% panel theme="info" header="List contents of OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone ls MyBox:/
+[demo2@login.crane ~]$ rclone ls myOneDrive:/
 {{< /highlight >}}
 {{% /panel %}}
 
 
-8.  To upload or download files, use the `clone` command.  For example:
+8.  To upload or download files, use the `rclone copy` command.  For example:
 {{% panel theme="info" header="Transferring files" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone copy MyBox:/SomeFile.txt ./
-[demo2@login.crane ~]$ rclone copy ./SomeFile.txt MyBox:/
+[demo2@login.crane ~]$ rclone copy myOneDrive:/SomeFile.txt ./
+[demo2@login.crane ~]$ rclone copy ./SomeFile.txt myOneDrive:/
 {{< /highlight >}}
 {{% /panel %}}
 
-9.  To download directories, use the `clone` command and use directory names over file.  This copies the contents of the folders, so you need to specify a destination folder.
-{{% panel theme="info" header="Download a directory from Box" %}}
+9.  To download directories, use the `rclone copy` command and use directory names over file.  This copies the contents of the folders, so you need to specify a destination folder.
+{{% panel theme="info" header="Download a directory from OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone copy MyBox:/my_hcc_dir ./my_hcc_dir
+[demo2@login.crane ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir
 {{< /highlight >}}
 {{% /panel %}}
-To upload a directory named `my_hcc_dir` to Box, use `clone`.
-{{% panel theme="info" header="Upload a directory to Box" %}}
+To upload a directory named `my_hcc_dir` to OneDrive, use `rclone copy`.
+{{% panel theme="info" header="Upload a directory to OneDrive" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone copy ./my_hcc_dir MyBox:/my_hcc_dir
+[demo2@login.crane ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir
 {{< /highlight >}}
 {{% /panel %}}
 
-11.  Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The exmaple below would sync the files of the local directory to the remote directory on box.
+11.  Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on OneDrive.
 {{% panel theme="info" header="transfer.sh" %}}
 {{< highlight bash >}}
-[demo2@login.crane ~]$ rclone sync ./my_hcc_dir MyBox:/my_hcc_dir
+[demo2@login.crane ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir
 {{< /highlight >}}
 {{% /panel %}}