Install the Slurm dispatcher

1. Introduction
2. Update config.yml
3. Install crunch-dispatch-slurm
4. Start the service
5. Restart the API server and controller

Introduction

This assumes you already have a Slurm cluster, and have set up all of your compute nodes with Docker or Singularity. Slurm packages are available on all distributions supported by Arvados. Please see your distribution package repositories. For information on installing Slurm from source, see this install guide

The Arvados Slurm dispatcher can run on any node that can submit requests to both the Arvados API server and the Slurm controller (via sbatch). It is not resource-intensive, so you can run it on the API server node.

Update config.yml

Crunch-dispatch-slurm reads the common configuration file at /etc/arvados/config.yml.

Add a DispatchSLURM entry to the Services section, using the hostname where crunch-dispatch-slurm will run, and an available port:

    Services:
      DispatchSLURM:
        InternalURLs:
          "http://hostname.zzzzz.arvadosapi.com:9007": {}

The following configuration parameters are optional.

API.MaxGatewayTunnels

Each Arvados container that runs on your HPC cluster will bring up a long-lived connection to the Arvados controller and keep it open for the entire duration of the container. This connection is used to access real-time container logs from Workbench, and to enable the container shell feature.

Set the MaxGatewayTunnels config entry high enough to accommodate the maximum number of containers you expect to run concurrently on your HPC cluster, plus incoming container shell sessions.

    API:
      MaxGatewayTunnels: 2000

Also, configure Nginx (and any other HTTP proxies or load balancers running between the HPC and Arvados controller) to allow the expected number of connections, i.e., MaxConcurrentRequests + MaxQueuedRequests + MaxGatewayTunnels.

Containers.PollInterval

crunch-dispatch-slurm polls the API server periodically for new containers to run. The PollInterval option controls how often this poll happens. Set this to a string of numbers suffixed with one of the time units ns, us, ms, s, m, or h. For example:

    Containers:
      PollInterval: 3m30s

Containers.ReserveExtraRAM: Extra RAM for jobs

Extra RAM to reserve (in bytes) on each Slurm job submitted by Arvados, which is added to the amount specified in the container’s runtime_constraints. If not provided, the default value is zero. Helpful when using -cgroup-parent-subsystem, where crunch-run and arv-mount share the control group memory limit with the user process. In this situation, at least 256MiB is recommended to accommodate each container’s crunch-run and arv-mount processes.

Supports suffixes KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB (where KB is 10³, KiB is 2¹⁰, MB is 10⁶, MiB is 2²⁰ and so forth).

    Containers:
      ReserveExtraRAM: 256MiB

Containers.MinRetryPeriod: Rate-limit repeated attempts to start containers

If Slurm is unable to run a container, the dispatcher will submit it again after the next PollPeriod. If PollPeriod is very short, this can be excessive. If MinRetryPeriod is set, the dispatcher will avoid submitting the same container to Slurm more than once in the given time span.

    Containers:
      MinRetryPeriod: 30s

Containers.Slurm.SbatchEnvironmentVariables

Some Arvados installations run a local keepstore on each compute node to handle all Keep traffic. To override Keep service discovery and access the local keep server instead of the global servers, set ARVADOS_KEEP_SERVICES in SbatchEnvironmentVariables:

    Containers:
      SLURM:
        SbatchEnvironmentVariables:
          ARVADOS_KEEP_SERVICES: "http://127.0.0.1:25107"

Containers.Slurm.PrioritySpread

crunch-dispatch-slurm adjusts the “nice” values of its Slurm jobs to ensure containers are prioritized correctly relative to one another. This option tunes the adjustment mechanism.

• If non-Arvados jobs run on your Slurm cluster, and your Arvados containers are waiting too long in the Slurm queue because their “nice” values are too high for them to compete with other Slurm jobs, you should use a smaller PrioritySpread value.
• If you have an older Slurm system that limits nice values to 10000, a smaller PrioritySpread can help avoid reaching that limit.
• In other cases, a larger value is beneficial because it reduces the total number of adjustments made by executing scontrol.

The smallest usable value is 1. The default value of 10 is used if this option is zero or negative. Example:

    Containers:
      SLURM:
        PrioritySpread: 1000

Containers.Slurm.SbatchArgumentsList

When crunch-dispatch-slurm invokes sbatch, you can add arguments to the command by specifying SbatchArguments. You can use this to send the jobs to specific cluster partitions or add resource requests. Set SbatchArguments to an array of strings. For example:

    Containers:
      SLURM:
        SbatchArgumentsList:
          - "--partition=PartitionName"

Note: If an argument is supplied multiple times, slurm uses the value of the last occurrence of the argument on the command line. Arguments specified through Arvados are added after the arguments listed in SbatchArguments. This means, for example, an Arvados container with that specifies partitions in scheduling_parameter will override an occurrence of --partition in SbatchArguments. As a result, for container parameters that can be specified through Arvados, SbatchArguments can be used to specify defaults but not enforce specific policy.

Containers.CrunchRunArgumentList: Dispatch to Slurm cgroups

If your Slurm cluster uses the task/cgroup TaskPlugin, you can configure Crunch’s Docker containers to be dispatched inside Slurm’s cgroups. This provides consistent enforcement of resource constraints. To do this, use a crunch-dispatch-slurm configuration like the following:

    Containers:
      CrunchRunArgumentsList:
        - "-cgroup-parent-subsystem=memory"

When using cgroups v1, the choice of subsystem (“memory” in this example) must correspond to one of the resource types enabled in Slurm’s cgroup.conf. The specified subsystem is singled out only to let Crunch determine the name of the cgroup provided by Slurm. Limits for other resource types will also be respected.

When doing this, you should also set ReserveExtraRAM .

Containers.CrunchRunArgumentList: Using host networking for containers

Older Linux kernels (prior to 3.18) have bugs in network namespace handling which can lead to compute node lockups. This by is indicated by blocked kernel tasks in “Workqueue: netns cleanup_net”. If you are experiencing this problem, as a workaround you can disable use of network namespaces by Docker across the cluster. Be aware this reduces container isolation, which may be a security risk.

    Containers:
      CrunchRunArgumentsList:
        - "-container-enable-networking=always"
        - "-container-network-mode=host"

Install crunch-dispatch-slurm

Red Hat, AlmaLinux, and Rocky Linux

# dnf install crunch-dispatch-slurm

Debian and Ubuntu

# apt install crunch-dispatch-slurm

Start the service

# systemctl enable --now crunch-dispatch-slurm
# systemctl status crunch-dispatch-slurm
[...]

If systemctl status indicates it is not running, use journalctl to check logs for errors:

# journalctl --since -5min -u crunch-dispatch-slurm

Restart the API server and controller

Make sure the cluster config file is up to date on the API server host then restart the API server and controller processes to ensure the configuration changes are visible to the whole cluster.

# systemctl restart nginx arvados-controller
# arvados-server check

---

Previous: Set up a compute node with Singularity Next: Configure Slurm