[Howto] S3: transfer files using rclone

Rclone is a command line data transfer tool that is available on Mac, PC and Linux. It can be used to transfer data to nearly any remote data source, such as Google Drive, Box, Dropbox, or to other servers, but in this guide we will use it to transfer data to Research Object Storage, also known as S3 storage.

Prerequisites:

• Rclone installed [Most linux machines will have rclone pre-installed, but ask your IT staff if it is not present]: 

  https://rclone.org/downloads/

• Research Object Storage account and S3 keys. Any PI can request an S3 account here, which grants them 50TB of storage: 

  https://uwmadison.co1.qualtrics.com/jfe/form/SV_5bRln8QCBicRT1A

Setting up rclone:

Rclone uses a configuration to set up 'remotes', which are the services that it can access and transfer data to and from. To create an S3 endpoint:

1. From the command line, run: rclone config
2. Type 'N' to create a new remote
3. Give your remote a descriptive name, like 'PI_NETID-S3'
4. For Storage option, select 4 - Amazon S3 compliant
5. For Provider, select 46 - Any other S3 compatible provider
6. For env_auth, select 1 - enter AWS credentials
7. Copy your Access key, hit enter, and then copy your Secret key on the next step
8. For Region, select 1 - Empty region
9. For Endpoint, use https://campus.s3.wisc.edu
10. Leave Location Constraint blank
11. For ACL, select option 3 - public-read-write
12. When asked for advanced config, select no
13. Review your setup, it should look like this:
    
    Type 'Y' to confirm, and then 'Q' to exit rclone's config

Using rclone to transfer to S3:

Rclone is most useful for transferring directories, but can transfer single files as well. One important note: rclone transfers the contents of folders, not the folders themselves. For instance, if you have a folder like this:

If you tell rclone to copy the 'snap' folder it will copy the directories and files within it to the destination, not the outer 'snap' folder itself.

if you run this command: rclone copy snap /tmp/ it will copy the files and folders inside of 'snap' to /tmp , not create a /tmp/snap/ folder.

To transfer a file to your newly created s3 remote:

1. First, make a folder to put the items you'd like to transfer, such as 'To_transfer'. Place all your files and directories you'd like to transfer within it.
2. Type rclone copy /PATH/TO/To_Transfer REMOTE_NAME:PI_NAME-bucket-01/PATH/TO/DEST -P
The first argument will be the path to the folder. The second argument is the name of the remote you created, a colon, and then the bucket name, which is in the form of PI_NETID-bucket-01 . From there, you can specify folders within the bucket . The -P command adds a progress bar.

There are lots of command line options that can help with large transfers. One good set is: -P --stats 1s --transfers 8 --checkers 2 --max-backlog=-1 --order-by size,mixed

Rclone will only copy files that change, so if you re-run the same command it will only move new or modified files, making it easy to make backups.