using_rclone_with_hcc.md 7.81 KB
Newer Older
Caughlin Bohn's avatar
Caughlin Bohn committed
1
2
3
+++
title = "Using Rclone for File Transfer"
description = "How to use Rclone with HCC"
Adam Caprez's avatar
Adam Caprez committed
4
weight = 60
Caughlin Bohn's avatar
Caughlin Bohn committed
5
6
7
8
+++

Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box, Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task. Guides on how to set up a variety of resources to transfer to and from can be found at [rclone's webpage](https://rclone.org/). 

9
This tool can be used to transfer files between HCC clusters and outside cloud providers, such as OneDrive.
Adam Caprez's avatar
Adam Caprez committed
10

Caughlin Bohn's avatar
Caughlin Bohn committed
11
12
### Setup RClone

13
1.  You must be able to access your [NU Office365](http://office.com/) account before beginning this process.  Contact your local campus IT support if you need help with initial account setup.
Caughlin Bohn's avatar
Caughlin Bohn committed
14

15
2.  Due to the clusters being remote machines, **Rclone will need to be installed on your [local machine](https://rclone.org/downloads/) in order to authorize OneDrive**. Some services, such as Google Drive, do not require Rclone to be installed on your local machine.
Caughlin Bohn's avatar
Caughlin Bohn committed
16
17
18
19
20
21
22
23

3.  After logging into the cluster of your choice, load the `rclone` module by entering the command below at the prompt:
{{% panel theme="info" header="Load the Rclone module" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ module load rclone
{{< /highlight >}}
{{% /panel %}}

24
4.  We will need to start the basic configuration for OneDrive. To do this run `rclone config`:
Caughlin Bohn's avatar
Caughlin Bohn committed
25
26
27
28
29
30
{{% panel theme="info" header="Load the rclone config" %}}
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone config
{{< /highlight >}}
{{% /panel %}}

31
32
5.  In a new configuration, you will see no remotes found. Enter `n` to make a new remote and name it a name you will know. In our example, we will use "myOneDrive". Select `Microsoft OneDrive` by entering in the corresponding number, in our case `23`. Hit Enter for the client_id, client_secret, and Edit advanced config. **When you are prompted for auto config, select `n`**.  The terminal will stop at a `result>` prompt.  Proceed to the next step.
{{% panel theme="info" header="Configure OneDrive" %}}
Caughlin Bohn's avatar
Caughlin Bohn committed
33
34
35
36
37
38
39
{{< highlight bash >}}
[demo2@login.crane ~]$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
40
name> myOneDrive
Caughlin Bohn's avatar
Caughlin Bohn committed
41
42
43
44
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value

45
46
23 / Microsoft OneDrive
   \ "onedrive"
Caughlin Bohn's avatar
Caughlin Bohn committed
47

48
49
Storage> 23
** See help for onedrive backend at: https://rclone.org/onedrive/ **
Caughlin Bohn's avatar
Caughlin Bohn committed
50

51
Microsoft App Client Id
Caughlin Bohn's avatar
Caughlin Bohn committed
52
53
Leave blank normally.
Enter a string value. Press Enter for the default ("").
54
55
client_id> 
Microsoft App Client Secret
Caughlin Bohn's avatar
Caughlin Bohn committed
56
57
Leave blank normally.
Enter a string value. Press Enter for the default ("").
58
client_secret> 
Caughlin Bohn's avatar
Caughlin Bohn committed
59
60
Edit advanced config? (y/n)
y) Yes
61
62
n) No (default)
y/n> 
Caughlin Bohn's avatar
Caughlin Bohn committed
63
64
Remote config
Use auto config?
65
66
67
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
Caughlin Bohn's avatar
Caughlin Bohn committed
68
69
n) No
y/n> n
70
71
72
For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine (same rclone version recommended) :
	rclone authorize "onedrive"
Caughlin Bohn's avatar
Caughlin Bohn committed
73
Then paste the result below:
74
result> 
Caughlin Bohn's avatar
Caughlin Bohn committed
75
76
77
{{< /highlight >}}
{{% /panel %}}

78
79
6.  Now **switch to a terminal on your local machine** and run `rclone authorize "onedrive"`. You will be prompted to go to a 127.0.0.1 address in your web browser if a browser doesn't open automatically. On the Microsoft Office sign-in page that opens enter in your NU e-mail address and click "Next".  You will be taken to your campus' single sign-on page where you can sign in using your TrueYou/campus credentials. If login is successful, you should be redirected to a page that says "Success!"  Return to your local rclone session.  You should see a message instructing you to paste a line of code from your local machine to the cluster and then to confirm that the config is correct.
{{% panel theme="info" header="Authorize OneDrive on local machine" %}}
Caughlin Bohn's avatar
Caughlin Bohn committed
80
{{< highlight bash >}}
81
[myaccount@local.machine ~]$ rclone authorize "onedrive"
Caughlin Bohn's avatar
Caughlin Bohn committed
82
83
{{< /highlight >}}
{{% /panel %}}
84
{{< figure src="/images/MicrosoftSign-In.png" height="500" class="img-border">}}
Adam Caprez's avatar
Adam Caprez committed
85
{{% panel theme="info" header="Local Authorization after logging in with browser" %}}
Adam Caprez's avatar
Rclone    
Adam Caprez committed
86
{{< highlight script >}}
87
[myaccount@local.machine ~]$ rclone authorize "onedrive"
Caughlin Bohn's avatar
Caughlin Bohn committed
88
89
90
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize Rclone for access
Waiting for code...
Adam Caprez's avatar
Adam Caprez committed
91
92
93
94
Got code
Paste the following into your remote machine --->
{"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
<---End paste
Caughlin Bohn's avatar
Caughlin Bohn committed
95
96
97
98
{{< /highlight >}}
{{% /panel %}}
For other services, please refer to the [rclone documentation](https://rclone.org/). 

99
100
7.  **Returning to the cluster**, now paste the code into the waiting `result` prompt.  Then select option 1 for `OneDrive Personal or Business` and then 0 for `OneDrive (business)`.  Press `y` at the next two prompts to confirm and `q` to exit.
{{% panel theme="info" header="Authorize OnDemand on cluster" %}}
Adam Caprez's avatar
Adam Caprez committed
101
102
{{< highlight bash >}}
result> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
Choose a number from below, or type in an existing value
 1 / OneDrive Personal or Business
   \ "onedrive"
 2 / Root Sharepoint site
   \ "sharepoint"
 3 / Type in driveID
   \ "driveid"
 4 / Type in SiteID
   \ "siteid"
 5 / Search a Sharepoint site
   \ "search"
Your choice> 1
Found 1 drives, please select the one you want to use:
0: OneDrive (business) id=b!laCd4ZJ54U-[...]
Chose drive to use:> 0
Found drive 'root' of type 'business', URL: https://uofnelincoln-my.sharepoint.com/personal/demo_unl_edu/Documents
Is that okay?
y) Yes (default)
n) No
y/n> y
Adam Caprez's avatar
Adam Caprez committed
123
--------------------
124
125
126
127
128
[myOneDrive]
type = onedrive
token = {"access_token": ...}
drive_id = b!laCd4ZJ54U-[...]
drive_type = business
Adam Caprez's avatar
Adam Caprez committed
129
--------------------
130
y) Yes this is OK (default)
Adam Caprez's avatar
Adam Caprez committed
131
132
133
134
135
136
137
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
138
myOneDrive           onedrive
Adam Caprez's avatar
Adam Caprez committed
139
140
141
142
143
144
145
146
147
148
149

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
{{< /highlight >}}
{{% /panel %}}
150
151
Now test the connection by running the `ls` command.  You should see a listing of your OneDrive files.
{{% panel theme="info" header="List contents of OneDrive" %}}
Caughlin Bohn's avatar
Caughlin Bohn committed
152
{{< highlight bash >}}
153
[demo2@login.crane ~]$ rclone ls myOneDrive:/
Caughlin Bohn's avatar
Caughlin Bohn committed
154
155
156
157
{{< /highlight >}}
{{% /panel %}}


158
8.  To upload or download files, use the `rclone copy` command.  For example:
Caughlin Bohn's avatar
Caughlin Bohn committed
159
160
{{% panel theme="info" header="Transferring files" %}}
{{< highlight bash >}}
161
162
[demo2@login.crane ~]$ rclone copy myOneDrive:/SomeFile.txt ./
[demo2@login.crane ~]$ rclone copy ./SomeFile.txt myOneDrive:/
Caughlin Bohn's avatar
Caughlin Bohn committed
163
164
165
{{< /highlight >}}
{{% /panel %}}

166
167
9.  To download directories, use the `rclone copy` command and use directory names over file.  This copies the contents of the folders, so you need to specify a destination folder.
{{% panel theme="info" header="Download a directory from OneDrive" %}}
Caughlin Bohn's avatar
Caughlin Bohn committed
168
{{< highlight bash >}}
169
[demo2@login.crane ~]$ rclone copy myOneDrive:/my_hcc_dir ./my_hcc_dir
Caughlin Bohn's avatar
Caughlin Bohn committed
170
171
{{< /highlight >}}
{{% /panel %}}
172
173
To upload a directory named `my_hcc_dir` to OneDrive, use `rclone copy`.
{{% panel theme="info" header="Upload a directory to OneDrive" %}}
Caughlin Bohn's avatar
Caughlin Bohn committed
174
{{< highlight bash >}}
175
[demo2@login.crane ~]$ rclone copy ./my_hcc_dir myOneDrive:/my_hcc_dir
Caughlin Bohn's avatar
Caughlin Bohn committed
176
177
178
{{< /highlight >}}
{{% /panel %}}

179
11.  Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to `rclone copy`. This would only transfer files that are updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on OneDrive.
Caughlin Bohn's avatar
Caughlin Bohn committed
180
181
{{% panel theme="info" header="transfer.sh" %}}
{{< highlight bash >}}
182
[demo2@login.crane ~]$ rclone sync ./my_hcc_dir myOneDrive:/my_hcc_dir
Caughlin Bohn's avatar
Caughlin Bohn committed
183
184
185
186
{{< /highlight >}}
{{% /panel %}}