Set up a Slurm compute node

Note:

crunch-dispatch-slurm is only relevant for on premises clusters that will spool jobs to Slurm. Skip this section if you are installing a cloud cluster.

Introduction
Set up Docker
Update fuse.conf
Update docker-cleaner.json
Configure Linux cgroups accounting
Install Docker
Configure the Docker daemon
Install’python-arvados-fuse and crunch-run and arvados-docker-cleaner

Introduction

This page describes how to configure a compute node so that it can be used to run containers dispatched by Arvados, with Slurm on a static cluster. These steps must be performed on every compute node.

Set up Docker

See Set up Docker

Update fuse.conf

FUSE must be configured with the user_allow_other option enabled for Crunch to set up Keep mounts that are readable by containers. Install this file as /etc/fuse.conf:

# Allow non-root users to specify the 'allow_other' or 'allow_root'
# mount options.
user_allow_other

Update docker-cleaner.json

The arvados-docker-cleaner program removes least recently used Docker images as needed to keep disk usage below a configured limit.

Create a file /etc/arvados/docker-cleaner/docker-cleaner.json in an editor, with the following contents.

{
    "Quota": "10G",
    "RemoveStoppedContainers": "always"
}

Choosing a quota: Most deployments will want a quota that’s at least 10G. From there, a larger quota can help reduce compute overhead by preventing reloading the same Docker image repeatedly, but will leave less space for other files on the same storage (usually Docker volumes). Make sure the quota is less than the total space available for Docker images.

Note:

This also removes all containers as soon as they exit, as if they were run with docker run --rm. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with "RemoveStoppedContainers":"never".

Install python-arvados-fuse and crunch-run and arvados-docker-cleaner

Red Hat and Centos

# yum install python-arvados-fuse crunch-run arvados-docker-cleaner

Debian and Ubuntu

# apt-get install python-arvados-fuse crunch-run arvados-docker-cleaner

Start the service

# systemctl enable --now arvados-docker-cleaner
# systemctl status arvados-docker-cleaner
[...]

If systemctl status indicates it is not running, use journalctl to check logs for errors:

# journalctl -n12 --unit arvados-docker-cleaner

Previous: Configure Slurm Next: Test Slurm dispatch