SLURM - Simple Linux Utility for Resource Management

SLURM is an open-source workload manager designed for Linux clusters of all sizes. It provides job scheduling and resource management to optimize cluster utilization.It is a highly scalable cluster management and job scheduling system for large and small Linux clusters. It is used by some of the world’s most powerful supercomputers. It menages resources such as CPU, GPU, Memory, etc…, It allows different users to work simultaneously on the cluster and provides mechanism for distributing the user programs across the cluster.

Please refer to SLURM documentation for more information.

• Open-source and actively developed
• Scalable to tens of thousands of nodes
• Flexible job scheduling options
• Supports job arrays, reservations, and dependencies
• Plugins available for authentication, accounting, and more

• Node – A single computer in the cluster.
• Partition – A group of nodes (like a queue).
• Job – A user-submitted task to be run on the cluster.
• Job Step – A component of a job, such as a single MPI process.
• Scheduler – The component that determines which jobs run when.

SLURM architecture overview ( original picture)

Slurm is based on different components, to menage the cluster resources. Bellow you can find a short summary:

• slurmctld (Controller Daemon)
  1. Runs on the management (head) node.
  2. Handles job scheduling, resource allocation, and overall cluster state.
  3. Usually consists of a primary and a backup controller for failover.

• slurmd (Node Daemon)
  1. Runs on each compute node.
  2. Responsible for launching, monitoring, and cleaning up jobs on the node.
  3. Communicates with the slurmctld to receive instructions.

• slurmdbd (Database Daemon) (optional)
  1. Manages job accounting and usage data.
  2. Works with an external database (e.g., MySQL, MariaDB).
  3. Enables commands like sacct and sreport for usage reporting.

• Client Commands
  1. Tools used by users and admins to interact with Slurm:
     1. sbatch – submit batch jobs
     2. srun – run parallel jobs interactively
     3. scancel – cancel jobs
     4. squeue – view job queues

• Central Database (optional)
  1. Stores job and usage records.
  2. Used in conjunction with slurmdbd for accounting and reporting.
  3. Supports multiple clusters if needed.

Each component communicates over a secure protocol to coordinate resource usage and job execution efficiently.

SchedMD - Slurm Workload Manager

• https://slurm.schedmd.com

Most common commands are:

Create a job script (e.g., `job.sbatch`):

  #!/bin/bash
  #SBATCH --job-name=test_job  # Name of your job. 
  #SBATCH --output=result.out  # Result file, standard output.
  #SBATCH --error=result.err   # Standard error.
  #SBATCH --time=01:00:00      # Work time.
  #SBATCH --partition=standard # Partition to work on.
  #SBATCH --ntasks=1           # Number of parallel tasks.
 
  echo "Hello from SLURM job"