SLURM Job Array
Job arrays are a handy tool for submitting many jobs that differ very little (usually only by the input they are receiving through a file). The ideas is to create one batch script file to submit dozens to hundreds or thousands of jobs instead of creating an individual file for each job that only differs a bit.
Below we illustrate an example where we want 30 jobs to run in our job array and our batch script is called “myslurmarray.slurm”. The program we want to run 30 times is call “myprogram” and it requires and input file. In this example each input filename looks like this (input1.dat, input2.dat, input3.dat … input30.dat).
#SBATCH -J myprog # A single job name for the array
#SBATCH -c 1 # Number of cores
#SBATCH -N 1 # All tasks on one machine
#SBATCH -p community.q# Partition
#SBATCH -t 0-2:00 # 2 hours (D-HH:MM)
#SBATCH -o myprog%A%a.out # Standard output
# %A" is replaced by the job ID and "%a" with the array index
#SBATCH -e myprog%A%a.err # Standard error
To submit this array we user the following command:
>sbatch –array=1-30 myslurmarray.slurm
Apart from SLURM_ARRAY_TASK_ID which is an environment variable unique for each job array job, notice also
%a, which represent the job id and the job array index, respectively. These can be used in the sbatch parameters to generate unique names – such at the output and error files in the above example.