Mounting S3 Storage with rclone

Campus S3 storage can be attached to a UWBC-managed server as an ordinary folder, so you can browse it and use it with normal commands and applications instead of transferring files back and forth. This article covers configuring an rclone remote for campus S3, mounting a bucket, and unmounting it when you are finished.

Before you start

• Shell access to a UWBC-managed server. rclone is already installed on these servers, so there is nothing to download.
• Your S3 access key and secret key. If you do not have them, request them from UWBC IT.
• The name of the bucket you intend to mount.
• Somewhere safe to keep the keys. S3 keys are normally issued per lab rather than per person, so anyone holding them has full access to the bucket's contents. Store them in a password manager such as 1Password rather than in a script, a shared document, or a plain text file on the server. Passing them on the command line is also worth avoiding, since commands are recorded in your shell history.

Steps

Configure the remote

Run the configuration tool and answer its prompts:

rclone config

1. Press n to create a new remote.
2. Give the remote a short name, such as s3. The rest of this article refers to whatever you choose here as REMOTENAME.
3. For the storage type, choose Amazon S3 Compliant Storage Providers (option 4 in current versions).
4. For the provider, choose Any other S3 compatible provider (option 53).
5. For authentication, choose Enter AWS credentials in the next step (option 1).
6. Paste your access key at the access_key_id prompt and press Enter.
7. Paste your secret key at the secret_access_key prompt and press Enter.
8. For region, choose option 1.
9. At the endpoint prompt, enter https://campus.s3.wisc.edu
10. Press Enter to skip location_constraint.
11. For the ACL, choose option 1 (private).
12. Press Enter at bucket_object_lock_enabled to accept false.
13. Answer n when asked whether to edit the advanced configuration.
14. Answer y to accept the remote.
15. Press q to quit the configuration tool.

The option numbers above match current versions of rclone, but they shift as providers are added to the list. If a number does not line up with the description given here, pick the entry whose text matches instead of the number.

Mount the bucket

Create an empty directory in your home directory to serve as the mount point. It must already exist and be empty before you mount onto it:

mkdir -p ~/s3

Then mount the bucket, substituting your remote name, your bucket name, and your mount point:

rclone mount REMOTENAME:YOUR-BUCKET-NAME/ ~/s3 --daemon

The --daemon flag puts rclone into the background and returns your prompt. Without it, the command occupies your terminal until you stop it, and you would need a second session to do anything else. You may see & used for this instead; --daemon is preferable, because it is rclone's own background mode rather than a shell job attached to your login session.

Unmount when finished

Release the mount with:

fusermount -u ~/s3

Unmount before logging out rather than relying on the session ending to clean up after itself. Terminating the rclone process directly can leave the mount point in a broken state that has to be cleared before it can be used again.

Verification

To confirm the remote itself is configured correctly, list the buckets available to your keys:

rclone lsd REMOTENAME:

To confirm the mount is working, list the mount point as you would any other directory:

ls ~/s3

Avoid testing with a bare rclone ls REMOTENAME:. That subcommand enumerates every file in the bucket, which produces a great deal of output very quickly on a bucket of any size. If you start it by accident, press Ctrl-C to stop it.

Known issues

• The mount fails with a complaint that the directory is not empty or is already in use: rclone requires an existing, empty directory as its mount point. Either empty the directory or create a new one and mount onto that instead.
• Commands against the mount point return "Transport endpoint is not connected": the rclone process behind the mount has stopped while the mount point is still attached. Run fusermount -u ~/s3 to clear it, then mount again.
• The mount disappears after you log out: a mount backgrounded with & is tied to the login session that started it. Use --daemon, and expect to mount again in your next session; the rclone remote configuration itself persists, so only the mount command needs repeating.
• Access is denied when listing the remote: confirm the endpoint was entered as https://campus.s3.wisc.edu and that both keys were pasted in full. The keys are long, and a truncated paste presents as an authentication failure. Re-run rclone config and edit the existing remote to correct either value.