How to Use rclone to Copy Files to the Cloud
If you visit the rclone installation page you will quickly realize that the instructions there require sudo (root privileges), which is not an option here.
However, you can take a look at the install file by simply running:
curl https://rclone.org/install.sh
It's obvious from examining the install script that all you need to do is download the right zip file from https://downloads.rclone.org/.
In our case, uname -m in the above install.sh will return x86_64, so the case statement x86_64|amd64 in the install scripts indicates that we need rclone-current-linux-amd64.zip
so downloadrclone-current-linux-amd64.zip
make your rclone directory
mkdir rclone
move the zip file into that directory
mv rclone-current-linux-amd64.zip rclone
enter that directory
cd rclone
unzip the zip file
unzip rclone-current-linux-amd64.zip
enter the directory where the binary is
cd rclone-v1.43.1-linux-amd64
And now configure rclone
./rclone config
There are a number of questions, but taking the defaults is the simplest approach. Though, you will have to choose which cloud service you want to connect to. In this example, the user on our systems is "testuser" and they wish to connect to "google drive". Also note that we are naming this connection "nyudrive" but we could have just as easily named it anything we wanted.
Here's an example of the config session ...
[linserv1] ~/rclone/rclone-v1.43.1-linux-amd64 $ ./rclone config
2018/10/11 16:24:55 NOTICE: Config file "/home/testuser/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> nyudrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Alias for a existing remote
\ "alias"
2 / Amazon Drive
\ "amazon cloud drive"
3 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
\ "s3"
4 / Backblaze B2
\ "b2"
5 / Box
\ "box"
6 / Cache a remote
\ "cache"
7 / Dropbox
\ "dropbox"
8 / Encrypt/Decrypt a remote
\ "crypt"
9 / FTP Connection
\ "ftp"
10 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
11 / Google Drive
\ "drive"
12 / Hubic
\ "hubic"
13 / JottaCloud
\ "jottacloud"
14 / Local Disk
\ "local"
15 / Mega
\ "mega"
16 / Microsoft Azure Blob Storage
\ "azureblob"
17 / Microsoft OneDrive
\ "onedrive"
18 / OpenDrive
\ "opendrive"
19 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
20 / Pcloud
\ "pcloud"
21 / QingCloud Object Storage
\ "qingstor"
22 / SSH/SFTP Connection
\ "sftp"
23 / Webdav
\ "webdav"
24 / Yandex Disk
\ "yandex"
25 / http Connection
\ "http"
Storage> 11
Google Application Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Google Application Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"
scope> 1
ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders. (see docs).
Enter a string value. Press Enter for the default ("").
root_folder_id>
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_file>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine or Y didn't work
y) Yes
n) No
y/n> n
If your browser doesn't open automatically go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id= ... the rest of this line has been deleted
Log in and authorize rclone for access
Enter verification code> 4/dQA-wP ... the rest of this line has been deleted
Configure this as a team drive?
y) Yes
n) No
y/n> n
--------------------
[nyudrive]
type = drive
scope = drive
token = {"access_token":"ya29.Glsz ... the rest of this line has been deleted
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
nyudrive drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
And that's it.
Notes:
- rclone has been tested and works with Dropbox and NYU Drive (the example above). It does not, however, work with Box at NYU, though it will most likely (untested) work with the dot com version of Box.
- in the above config, when you get to the following question:
Use auto config?
You can and should enter "y" here if at a desktop, even though in the example a "n" was entered.
* Say Y if not sure
* Say N if you are working on a remote or headless machine or Y didn't work
y) Yes
n) No
y/n> n - You can add other cloud services by running "./rclone config" again and choosing one of the others, such as 7 (which is Dropbox) or 5 (which is Box)
Some Rudimentary Commands
To find where the configuration is stored locally run rclone config file. It's most likely in /home/testuser/.config/rclone/rclone.conf.
./rclone lsd nyudrive:
This will list all the directories at nyudrive (the name you gave above).
./rclone copy ~/testdir nyudrive:rclone-test
This will copy all the files in a directory called testdir to a folder on goodle drive called rclone-test.
./rclone copy nyudrive:rclone-test .
This will copy all the files in the folder on google drive called rclone-test to your present location on the local system.
For more on how to use rclone see the rclone documentation.
You can also set this up as a cron job as opposed to running it on the command line as shown in the examples above. If you decide to run it as a cronjob, be aware that cron is unaware of your environment variables, so you will need to use full paths. For more information on how to set up a cron job see the man page by executing "man crontab" or even better simply google "how to set up a cron job."