Set up a compute node with Singularity

Note:

This page describes the requirements for a compute node in a Slurm or LSF cluster that will run containers dispatched by crunch-dispatch-slurm or arvados-dispatch-lsf. If you are installing a cloud cluster, refer to Build a cloud compute node image.

Note:

These instructions apply when Containers.RuntimeEngine is set to singularity, refer to Set up a compute node with Docker when running docker.

Introduction
Install python-arvados-fuse and crunch-run and squashfs-tools
Set up Singularity
Singularity mksquashfs configuration

Introduction

Please refer to the Singularity documentation in the Architecture section.

This page describes how to configure a compute node so that it can be used to run containers dispatched by Arvados on a static cluster. These steps must be performed on every compute node.

Install python-arvados-fuse and crunch-run and squashfs-tools

Red Hat and Centos

# yum install python-arvados-fuse crunch-run squashfs-tools

Debian and Ubuntu

# apt-get install python-arvados-fuse crunch-run squashfs-tools

Install NVIDA CUDA Toolkit (optional)

If you want to use NVIDIA GPUs, install the CUDA toolkit.

In addition, you also must install the NVIDIA Container Toolkit:

DIST=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$DIST/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/libnvidia-container.list
sudo apt-get update
apt-get install libnvidia-container1 libnvidia-container-tools nvidia-container-toolkit

Set up Singularity

Follow the Singularity installation instructions. Make sure singularity and mksquashfs are working:

$ singularity version
3.9.9
$ mksquashfs -version
mksquashfs version 4.4 (2019/08/29)
[...]

Then update Containers.RuntimeEngine in your cluster configuration:

      # Container runtime: "docker" (default) or "singularity"
      RuntimeEngine: singularity

Singularity mksquashfs configuration

Docker images are converted on the fly by mksquashfs, which can consume a considerable amount of RAM. The RAM usage of mksquashfs can be restricted in /etc/singularity/singularity.conf with a line like mksquashfs mem = 256M. The amount of memory made available for mksquashfs should be configured lower than the smallest amount of memory requested by a container on the cluster to avoid the conversion being killed for using too much memory. The default memory allocation in CWL is 256M, so that is also a good choice for the mksquashfs mem setting.

Previous: Set up a compute node with Docker Next: Install the Slurm dispatcher