rclone | NYU Courant

Installing rclone

Please note that the below command is not necesssary on CIMS managed Linux desktops and servers. rclone is already installed on them.

For information on how to install please visit the rclone installation page. Basically, you just need to run:
sudo -v ; curl https://rclone.org/install.sh | sudo bash

Configuring clone

To configure rclone run:
rclone config

There are a number of questions, but taking the defaults is the simplest approach. Though, you will have to choose which cloud service you want to connect to. In this example, the user on our systems is "abc123" and they wish to connect to NYU Box. Also note that we are naming this connection "nyubox" but we could have just as easily named it anything we wanted.

Here's an example of how to configure rclone for NYU Box.

[abc123@access2 ~]$ ./rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> nyubox

Storage> box

Option client_id.
OAuth Client Id.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_id>

Option client_secret.
OAuth Client Secret.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_secret>

Option box_config_file.
Box App config.json location
Leave blank normally.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a value. Press Enter to leave empty.
box_config_file>

Option access_token.
Box App Primary Access Token
Leave blank normally.
Enter a value. Press Enter to leave empty.
access_token>
Option box_sub_type.
Choose a number from below, or type in your own value of type string.
Press Enter for the default (user).
 1 / Rclone should act on behalf of a user.
   \ (user)
 2 / Rclone should act on behalf of a service account.
   \ (enterprise)
box_sub_type> user

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.

y) Yes (default)
n) No
y/n> n

Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
        rclone authorize "box"
Then paste the result.
Enter a value.
config_token> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX","expires_in":XXXX}

Configuration complete.
Options:
- type: box
- token: {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX","expires_in":XXXX}
Keep this "nyubox" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Current remotes:

Name                 Type
====                 ====
nyubox               box

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Please note that at the "Use web browser to automatically authenticate rclone with remote?" step you can and should enter "y" here if at a desktop, even though in the example a "n" was entered.
You can add other cloud services by running "./rclone config" again and choosing one of the others, such as "dropbox" or "drive".

Some Rudimentary Commands

To find where the configuration is stored locally (it's most likely in $HOME/.config/rclone/rclone.conf) run:
rclone config file

To list all the directories at nyubox (the name you gave above):
rclone lsd nyubox:

To This copy all the files in a directory called testdir to a folder on NYU Box called rclone-test:
rclone copy ~/testdir nyubox:rclone-test

To copy all the files in the folder on NYU Box called rclone-test to your present location on the local system:
rclone copy nyubox:rclone-test

For more on how to use rclone see the rclone documentation.

You can also set this up as a cron job as opposed to running it on the command line as shown in the examples above. If you decide to run it as a cronjob, be aware that cron is unaware of your environment variables, so you will need to use full paths. For more information on how to set up a cron job see the man page by executing "man crontab" or even better simply google "how to set up a cron job."

Advanced configuration

It's possble to configure rclone to encrypt the data stored in whatever cloud service you use and to break large files into chunks (most cloud serices have a maximum file size). The following example is based on what's described here. Please note that it assumes you have already configure a remote "nyubox" and that a directory "data" exists at the root of "nyubox".

Configure crypt remote for "nyubox"

[abc123@access2 ~]$ rclone config
Current remotes:

Name                 Type
====                 ====
nyubox               box

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n

Enter name for new remote.
name> nyubox-crypt

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.

Storage> crypt

Option remote.
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a value.
remote> nyubox:data
How to encrypt the filenames.
Choose a number from below, or type in your own value of type string.
Press Enter for the default (standard).
   / Encrypt the filenames.
 1 | See the docs for the details.
   \ (standard)
 2 / Very simple filename obfuscation.
   \ (obfuscate)
   / Don't encrypt the file names.
 3 | Adds a ".bin", or "suffix" extension only.
   \ (off)
filename_encryption> standard

Option directory_name_encryption.
Option to either encrypt directory names or leave them intact.
NB If filename_encryption is "off" then this option will do nothing.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (true).
 1 / Encrypt directory names.
   \ (true)
 2 / Don't encrypt directory names, leave them intact.
   \ (false)
directory_name_encryption> true

Option password.
Password or pass phrase for encryption.
Choose an alternative below.
y) Yes, type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:

Option password2.
Password or pass phrase for salt.
Optional but recommended.
Should be different to the previous password.
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:
- type: crypt
- remote: nyu_box:data
- password: *** ENCRYPTED ***
- password2: *** ENCRYPTED ***
Keep this "n" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Current remotes:

Name                 Type
====                 ====
nyubox-crypt         crypt
nyubox               box

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Now configure filename_encoding parameter for newly created "nyubox-crypt" remote:

[abc123@access2 ~]$ rclone config update nyubox-crypt filename_encoding=base32768

Configure chunked remote for "nyubox-crypt"