Platform R: Local Storage

For local storage Platform R uses Ceph, a redundant, high-performance filesystem. This storage is for work related to Platform R only. For permanent storage you should use Restricted Research Drive.

Backups

The local Ceph file system is not backed up, so we recommend that you copy results you care about to your restricted research drive reasonably quickly.

Retention

Since the local storage is a shared resource, it is best if you delete data you no longer need promptly. However, there is no time limit on using local storage for active datasets. If the data is being worked on for two years, there's no need to delete that and recopy it to Platform R every time you submit a new set of Slurm jobs.

Pay special attention to temporary and intermediate work files created by your workflows. Sometimes those do not get cleaned up automatically, especially if a program quits due to an error. If you can, write your Slurm submit jobs to clean up after themselves when they do create intermediate work files.

Default Directories

Every user has a home directory and is part of at least one PI group project directory:

/home/wisc/your-netid - your home directory
/mnt/scratch/group/PI-netid - your group project directory

Any files that are shared, software and data, should generally end up in the group project directory. Any data Slurm jobs will use or produce, especially, should be in the group directory.

It is worthwhile for a lab to spend a little time to develop shared conventions for naming and data workflows in the shared project space.

Your home directory is fine for conda and spack environments. If you have a lab workflow that requires everyone to use identical conda setups it is better to set that up in the group project directory.

If you use containers from container repositories (see Platform R: Using Containers in Slurm), Apptainer will cache the images in your home directory, under ~/.apptainer/cache. These can take up a lot of space. The tools Apptainer provides to manage the cache do not offer many options:

apptainer cache list - says how many containers are cached, and how much space they are using; however, it does not tell you what the images are
apptainer cache clean - cleans out the entire cache
apptainer cache clean -D 30 - cleans out cache entries older than 30 days

Quota

Disk use quotas are enforced for home and project directories. This is primarily a measure to prevent accidents where a job unintentionally creates a huge amount of data.

your home directory has a default quota of 250G
your group project directory has a default quota of 10T

When you first log in you are given a usage report for your home directory and group project directory. If you would like to see a report without logging in again, run /usr/local/bin/ceph_quota_motd.py

To request a quota increase, please submit a ticket to the Platform R Navigator.

Preloaded Containers

Platform R offers several containers configured with most of the standard libraries commonly required by Platform R users. These can be found in /mnt/scratch/shared/containers. Let us know if there are other containers that might be broadly useful.