Platform R: Slurm Example Workflows

This document covers a few basic workflows which introduce the core Slurm functionality most people's work will rely on.

If you are new to Slurm, please see Platform R: Slurm Introduction and Platform R: Slurm Glossary first.

Always test a new workflow first, with minimum resource requirements and a smaller subset of your data, to verify it is working as expected. If Slurm jobs put too much resource strain on the cluster a Platform R engineer may have to put a hold on the jobs and contact you to discuss options to minimize impact (Platform R: Slurm Job Suspension Process).

Single Job Using a Single CPU
- Default Options
- Start With a Clean Environment
Single Job Using Multiple CPUs
Parallel Tasks
Array Jobs
Related Documentation

Single Job Using a Single CPU

This is nearly the simplest usable Slurm batch script:

#!/bin/bash

srun hostname

Default Options

With no options the defaults are:

one node (--nodes=1)
one task (--ntasks=1)
one CPU per task (--cpus-per-task=1)
running in the CPU partition (--partition=cpu)
with 4G of memory (--mem=4g)
with a two hour time limit (--time=2:00:00).

In general it is better to be explicit about your job resource requirements.

Start With a Clean Environment

If you use conda or spack environments, or environment modules, it is a good idea to clear out any of those in your batch script before then loading what you need. This is a slightly exaggerated example:

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=1:00
#SBATCH --mem=16G

module purge
conda deactivate

conda activate r-base

srun Rscript --vanilla my_r_program.R

Single Job Using Multiple CPUs

Only software that has been written to use multiple CPUs can take advantage of requesting multiple CPUs. If your software hasn't been written this way, asking for extra CPU cores will not make your program run faster. Consult your software's documentation to verify multi-threaded and multi-process options.

It is important that both Slurm and your software ask for the same number of CPUs to do the work. Different terms might be used to describe the software's multiprocessing capabilities: threads, processes, CPUs, or cores. There will be either environment variables or command line arguments to tell your software how many to use. If your program requests more CPUs than are requested in the Slurm options, your program will likely end up running slower, or the Slurm scheduler might kill it prematurely.

Here is an example using Samtools, which uses -@ threads to specify how many CPUs it will use.

#!/bin/bash
#SBATCH --job-name=samtools-example
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=2:00
#SBATCH --mem=64G

srun samtools sort -@ 8 -o sorted.bam input.bam

Note that the environment variable SLURM_CPUS_PER_TASK is available in the batch script once set in the #SBATCH line, so you could also use this, which will ensure your program always asks for as many CPUs as the batch job has requested:

#!/bin/bash
#SBATCH --job-name=samtools-example
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=2:00
#SBATCH --mem=64G

srun samtools sort -@ $SLURM_CPUS_PER_TASK -o sorted.bam input.bam

Parallel Tasks

If you have a set of programs which go together logically in a single workflow, and do not have dependencies between them, you can bundle the work together in a single batch file. For that you request the total resources you need in the sbatch script, including the --ntasks option set to however many programs there are, and then use srun resource options for each individual job.

#!/bin/bash
#SBATCH --job-name=ntasks-test
#SBATCH --partition=cpu
#SBATCH --time=20:00
#SBATCH --ntasks=3
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --output=%x_%j.out  # %x is the job name, %j the jobID

srun --ntasks=1 part_one.py &
srun --ntasks=1 part_two.py &
srun --ntasks=1 part_three.py & 
wait

Note three things about this batch file in particular:

The srun lines say how many tasks they are using out of the three total requested in the #SBATCH line. If you ask for too few of these, some part of your jobs will run more than once, so that the total used matches what you asked for in the #SBATCH line.
The srun lines have an ampersand at the end, which makes them run in parallel.
The wait command at the end makes the batch script wait until all srun tasks have completed before completing itself.

Array Jobs

A job array is recommended when you are running hundreds or thousands of jobs using the same code with different parameters or datasets, such as when you need to process many different data files with the same analysis.

The maximum number of jobs in a single array on Platform R is 5000.

There are two ways for your program to know which array job they are running as: with an environment variable or through the command line.

Environment Variable

The environment variable SLURM_ARRAY_TASK_ID contains the array task ID. If your batch script has --array=1-5, SLURM_ARRAY_TASK_ID will be set from "1" to "5" across the different array tasks. Here is the way to get an environment variable in common languages used in Platform R:

Python 3: os.environ['SLURM_ARRAY_TASK_ID'] (you must also import os)
R: Sys.getenv("SLURM_ARRAY_TASK_ID")
Matlab: getenv("SLURM_ARRAY_TASK_ID")

Note that all of the above commands return a string, not a numeric type.

Command Line

Your bash submit file will run identically for each array job, which means that SLURM_ARRAY_TASK_ID is set properly in your shell script, too, so you can use it on the command line for your program.

#!/bin/bash
#SBATCH --job-name=array-test
#SBATCH --array=0-3

srun my_program.py ${SLURM_ARRAY_TASK_ID}

Most languages have several libraries to parse command line arguments.

Array File Index

If you have many data files that don't already have sequential numbers in the file names, and it is awkward to rename all of them, you can create an index to make sure each array job runs on a different file.

Here, for example, is a blunt python indexing script. It simply finds all files that end in .fastq in the current directory and then writes a JSON file called array_index.json you can use to match array task IDs to file names.

#!/usr/bin/env python3

import glob, json

# Get name of each fastq file in the current directory.
files = glob.glob("*.fastq")

# Create index dictionary.
index = {idx: files[idx] for idx in range(len(files))}

# Save index dictionary as JSON.
with open("array_index.json", "w") as indexfile:
    json.dump(index, indexfile)

Then in your program you can match an array task ID to a file with a few lines like this:

#!/usr/bin/env python3

import json

# Load the JSON index file.
with open("array_index.json") as file:
  index = json.load(file)

# Match index to file name:
datafile = index[os.environ['SLURM_ARRAY_TASK_ID']]

You can also use the JSON index file on the command line using the JSON processor, jq. Here is a trivial example batch script which assumes there are four data files and you have already created the index file:

#!/bin/bash
#SBATCH --job-name="index-test"
#SBATCH --time=2:00:00
#SBATCH --array=0-3
#SBATCH --cpus-per-task=1
#SBATCH --error=out/job.%a.err
#SBATCH --output=out/job.%a.out

filename=`jq -r ".\"${SLURM_ARRAY_TASK_ID}\"" array_index.json`
echo "Processing ${filename}..."

Another option for using the command line is to create a file with one data file path per line, called array_index.txt and use sed to grab the specific line. This will index from 1. Note that the character p after ${SLURM_ARRAY_TASK_ID}is part of sed's syntax, not a typo.

#!/bin/bash
#SBATCH --job-name="index-test"
#SBATCH --time=2:00:00
#SBATCH --array=0-3
#SBATCH --cpus-per-task=1
#SBATCH --error=out/job.%a.err
#SBATCH --output=out/job.%a.out

filename=`sed -n ${SLURM_ARRAY_TASK_ID}p array_index.txt`
echo "Processing ${filename}..."

Limiting Running Array Jobs

Though the Platform R maximum array size is 5000, there are times when you will not want that many jobs to run. In particular, if all of your jobs are doing many writes to the shared filesystem (see Platform R: Local Storage) the Ceph file server can get quite slow.

To limit the number of array jobs running at the same time, use %LimitNumber at the end of the --array option. For example, to ask for 2000 jobs, but limited to 250 jobs at any one time, use the option --array=0-1999%250