Set up SLURM

Containers can be dispatched to a SLURM cluster. The dispatcher sends work to the cluster using SLURM’s sbatch command, so it works in a variety of SLURM configurations.

In order to run containers, you must run the dispatcher as a user that has permission to set up FUSE mounts and run Docker containers on each compute node. This install guide refers to this user as the crunch user. We recommend you create this user on each compute node with the same UID and GID, and add it to the fuse and docker system groups to grant it the necessary permissions. However, you can run the dispatcher under any account with sufficient permissions across the cluster.

On the API server, install SLURM and munge, and generate a munge key.

On Debian-based systems:

~$ sudo /usr/bin/apt-get install slurm-llnl munge
~$ sudo /usr/sbin/create-munge-key

On Red Hat-based systems:

~$ sudo yum install slurm munge slurm-munge

Now we need to give SLURM a configuration file. On Debian-based systems, this is installed at /etc/slurm-llnl/slurm.conf. On Red Hat-based systems, this is installed at /etc/slurm/slurm.conf. Here’s an example slurm.conf:

ControlMachine=ClusterID.example.com
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/tmp
SlurmdSpoolDir=/tmp/slurmd
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ProctrackType=proctrack/pgid
CacheGroups=0
ReturnToService=2
TaskPlugin=task/affinity
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
#
# SCHEDULING
SchedulerType=sched/backfill
SchedulerPort=7321
SelectType=select/linear
FastSchedule=0
#
# LOGGING
SlurmctldDebug=3
#SlurmctldLogFile=
SlurmdDebug=3
#SlurmdLogFile=
JobCompType=jobcomp/none
#JobCompLoc=
JobAcctGatherType=jobacct_gather/none
#
# COMPUTE NODES
NodeName=DEFAULT
PartitionName=DEFAULT MaxTime=INFINITE State=UP

NodeName=compute[0-255]
PartitionName=compute Nodes=compute[0-255] Default=YES Shared=YES

SLURM configuration essentials

Whenever you change this file, you will need to update the copy on every compute node as well as the controller node, and then run sudo scontrol reconfigure.

ControlMachine should be a DNS name that resolves to the SLURM controller (dispatch/API server). This must resolve correctly on all SLURM worker nodes as well as the controller itself. In general SLURM is very sensitive about all of the nodes being able to communicate with the controller and one another, all using the same DNS names.

SelectType=select/linear is needed on cloud-based installations that update node sizes dynamically, but it can only schedule one container at a time on each node. On a static or homogeneous cluster, use SelectType=select/cons_res with SelectTypeParameters=CR_CPU_Memory instead to enable node sharing.

NodeName=compute[0-255] establishes that the hostnames of the worker nodes will be compute0, compute1, etc. through compute255.

• There are several ways to compress sequences of names, like compute[0-9,80,100-110]. See the “hostlist” discussion in the slurm.conf(5) and scontrol(1) man pages for more information.
• It is not necessary for all of the nodes listed here to be alive in order for SLURM to work, although you should make sure the DNS entries exist. It is easiest to define lots of hostnames up front, assigning them to real nodes and updating your DNS records as the nodes appear. This minimizes the frequency of slurm.conf updates and use of scontrol reconfigure.

Each hostname in slurm.conf must also resolve correctly on all SLURM worker nodes as well as the controller itself. Furthermore, the hostnames used in the configuration file must match the hostnames reported by hostname or hostname -s on the nodes themselves. This applies to the ControlMachine as well as the worker nodes.

For example:

• In slurm.conf on control and worker nodes: ControlMachine=ClusterID.example.com
• In slurm.conf on control and worker nodes: NodeName=compute[0-255]
• In /etc/resolv.conf on control and worker nodes: search ClusterID.example.com
• On the control node: hostname reports ClusterID.example.com
• On worker node 123: hostname reports compute123.ClusterID.example.com

Automatic hostname assignment

The API server will choose an unused hostname from the set given in application.yml, which defaults to compute[0-255].

If it is not feasible to give your compute nodes hostnames like compute0, compute1, etc., you can accommodate other naming schemes with a bit of extra configuration.

If you want Arvados to assign names to your nodes with a different consecutive numeric series like {worker1-0000, worker1-0001, worker1-0002}, add an entry to application.yml; see /var/www/arvados-api/current/config/application.default.yml for details. Example:

• In application.yml: assign_node_hostname: worker1-%<slot_number>04d
• In slurm.conf: NodeName=worker1-[0000-0255]

If your worker hostnames are already assigned by other means, and the full set of names is known in advance, have your worker node bootstrapping script (see Installing a compute node) send its current hostname, rather than expect Arvados to assign one.

• In application.yml: assign_node_hostname: false
• In slurm.conf: NodeName=alice,bob,clay,darlene

If your worker hostnames are already assigned by other means, but the full set of names is not known in advance, you can use the slurm.conf and application.yml settings in the previous example, but you must also update slurm.conf (both on the controller and on all worker nodes) and run sudo scontrol reconfigure whenever a new node comes online.

---

Previous: Set up Docker