Table of Contents:

• What is a "bucket"?

• Configuration settings for "default" bucket

• Creating a new bucket

• Access pools (i.e. bucket "gateways")

• "Everyone" Permission (i.e. enabling access without an access key)

• Versioning / backup

What is a "bucket"?

Research Object Storage (S3) utilizes buckets, which are “containers” for objects (i.e. files and folders) stored in your account.  All files/folders must be stored within a bucket. You can create folders within a bucket to help organize your files. 

Buckets have settings that you can configure to achieve different goals, such enabling version history on your files.

Configuration settings for "default" bucket

When your account was created, we provided you with a pre-made "default" bucket that had a set of default configurations applied to it. 

Configuration summary:  The default bucket was created in the Campus-only access pool without "Everyone" permissions set.  Version history is turned on with a 30 day retention history selected. 

Overview of default account / bucket configurations

Creating a new bucket

You have the ability to create an unlimited number of new buckets.  While most users will find that their default bucket meets their needs and is the best choice due to the helpful features and configurations in place on it, some may desire to create additional buckets. 

Default feature settings for user-created buckets

Any bucket you create will not have certain features and/or settings enabled by default. 

Creating a bucket via Cyberduck

1. Within Cyberduck, connect to either the Campus-only or Web access pool.  (Note: if you are connecting to Campus-only, make sure you are physically on campus or are using the campus VPN.
2. Without clicking into an existing bucket, click "File > New Folder" in the main menu (or right-click in an empty section of the bucket browser and select "New Folder").
   
   
   
3. Enter the name for the new bucket and click "Create". Note: The client assumes you are creating the bucket in the AWS public cloud, but our buckets are local and the region setting is ignored. It doesn't matter which option is selected.
   
   

Access pools (i.e. bucket "gateways")

Every bucket (including those that you create) must be created in an "access pool", which refers to the server that provides access to your data.  You might think of access pools as "gateways" to your bucket, allowing or blocking access depending on the pool it is assigned to. 

Access pool options

 There are two access pools:  "Campus-only pool" and "Web pool".  

• • Campus-only pool:  The "campus-only" access pool has firewall rules in place that allow access from UW-Madison networks (including campus VPN) and blocks all off-campus access.

  • Web pool:  The "web" access pool allows access from any network location (ie. the wide internet) and does not block off-campus access. 

The default bucket we provided you (<netid>-bucket-01) is in the campus-only access pool.

Overview of access restrictions by access pool

How to assign your bucket to an access pool

Buckets will automatically be assigned to the access pool that you are connected to when you create the bucket.  In other words, any bucket you create while connected to an access pool will be in that pool.

To learn how to connect to either the Campus-only or Web access pools, see Research Object Storage (S3): Accessing & Transferring Data.

"Everyone" Permission (i.e. enabling access without an access key)

The "Everyone" permission allows people to access the file(s) in your buckets without an access key and secret (ex: via a web link).  You can apply an "Everyone" permission to your entire bucket or individual files within a bucket. The "Everyone" can be tricky, so we recommend using it in specific ways.

"Everyone" permission options

The "Everyone" permission has three permission options:

• Everyone > READ:  This option provides read access without an access key.
• Everyone > WRITE:  This option provides read and write access without an access key. Write access includes deleting files and versions. (not recommended)
• Everyone > FULL CONTROL:  This option provides read, write, and admin/configuration access without an access key. An anonymous user could even delete your bucket. (not recommended)

Which access pool / permissions should I use?

Which access pool and permissions you select for your bucket depends on your use case. 

"Everyone" permission for web sharing

A common use case for Research Object Storage (S3) is hosting static web files. The "Everyone" permission is how you can create and provide links to files that are stored directly in your buckets. You can use these links to allow others to download datasets or link files from your project website. This permission level should only be used for public data, because the "Everyone" permission allows ANYONE to download your files anonymously.

In order to make using the "Everyone" permission simipler and more secure, we recommend creating a new bucket just for this purpose and enabling "Everyone" at the bucket level.  We DO NOT recommend enabling it on your default bucket.

Setting the "Everyone" permission via Cyberduck

1. Within Cyberduck, connect to either the Campus-only or Web access pool.  (Note: if you are connecting to Campus-only, make sure you are on a campus network or are using the campus VPN.
2. Select the bucket you would like to add the permission to.
3. Select the "Get info" button in the top-menu.
   
   
   
   
4. Select the "Permissions" tab.
   
   
   
   
5. Select the drop-down next to the gear icon at the bottom of the screen.
6. Select "Everyone"
   
   
7. Critical step:  In the "Access Control List", locate the "Everyone" grantee and change the corresponding value under "Permission" column to "Read". (DO NOT select any of the other permission options!).  If you fail to complete this step, you may compromise your account's security.
   

   

8. Repeat for every bucket you wish to add this permission to.

Removing the "Everyone" permission via Cyberduck

1. Within Cyberduck, connect to either the Campus-only or Web access pool.  (Note: if you are connecting to Campus-only, make sure you are on a campus network or are using the campus VPN.
2. Select the bucket you would like to remove the permission from.
3. Select the "Get info" button in the top-menu.
4. Select the "Permissions" tab.
5. In the "Access Control List", highlight the "Everyone" grantee that you wish to remove.
   
   
   
   
6. Select the drop-down next to the gear icon at the bottom of the screen.
7. Select "Remove".
   

   

8. Repeat for every bucket you wish to remove this permission for.

Impact of "Everyone" permission by access pool / permission option

Versioning / backup

Unlike with ResearchDrive, data stored on Research Object Storage (S3) is not automatically backed up daily and does not support snapshots. 

Instead, this service offers a feature called "versioning". A "version" is a copy of file that is generated every time the file is uploaded / edited.  This allows you to recover accidentally deleted files, providing you have not deleted the bucket or version files themselves.  Versions are assigned a lifecycle, which is a retention period after which the version is automatically deleted.

Is versioning turned on for my bucket(s) by default?

Yes and no.  Versioning is turned “on” by default for the initial bucket we created for you.  Any additional buckets you create will NOT have versioning turned on by default.  You will need to enable versioning for any buckets you create.

Enabling/disabling versioning

You may enable versioning via an app such as Cyberduck or via the S3 Python API.

Enabling/disabling versioning via Cyberduck

1. Open Cyberduck and connect to the access pool of your choice
2. Select/highlight the bucket you would like to enable versioning on
3. Click the "Get info" button in top menu
   
   
   
   
4. Select "S3" tab
   
   
   
   
5. Check (or uncheck if disabling) the box labeled "Bucket Versioning"
   
   
   
   
6. Note:  The "Lifecycle" options under the S3 tab only work for AWS S3 and will not set a lifecycle on buckets within Research Object Storage (S3).  To set a lifecycle on a bucket, you may use the Python S3 API.