Set up a compute node with Singularity


This page describes the requirements for a compute node in a Slurm or LSF cluster that will run containers dispatched by crunch-dispatch-slurm or arvados-dispatch-lsf. If you are installing a cloud cluster, refer to Build a cloud compute node image.


These instructions apply when Containers.RuntimeEngine is set to singularity, refer to Set up a compute node with Docker when running docker.

  1. Introduction
  2. Install python-arvados-fuse and crunch-run and squashfs-tools
  3. Set up Singularity
  4. Singularity mksquashfs configuration


Please refer to the Singularity documentation in the Architecture section.

This page describes how to configure a compute node so that it can be used to run containers dispatched by Arvados on a static cluster. These steps must be performed on every compute node.

Install python-arvados-fuse and crunch-run and squashfs-tools

Red Hat and Centos

# yum install python-arvados-fuse crunch-run squashfs-tools

Debian and Ubuntu

# apt-get install python-arvados-fuse crunch-run squashfs-tools

Install NVIDA CUDA Toolkit (optional)

If you want to use NVIDIA GPUs, install the CUDA toolkit.

In addition, you also must install the NVIDIA Container Toolkit:

DIST=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L | \
  sudo apt-key add -
curl -s -L$DIST/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/libnvidia-container.list
sudo apt-get update
apt-get install libnvidia-container1 libnvidia-container-tools nvidia-container-toolkit

Set up Singularity

Follow the Singularity installation instructions. Make sure singularity and mksquashfs are working:

$ singularity version
$ mksquashfs -version
mksquashfs version 4.3-git (2014/06/09)

Then update Containers.RuntimeEngine in your cluster configuration:

      # Container runtime: "docker" (default) or "singularity"
      RuntimeEngine: singularity

Singularity mksquashfs configuration

Docker images are converted on the fly by mksquashfs, which can consume a considerable amount of RAM. The RAM usage of mksquashfs can be restricted in /etc/singularity/singularity.conf with a line like mksquashfs mem = 256M. The amount of memory made available for mksquashfs should be configured lower than the smallest amount of memory requested by a container on the cluster to avoid the conversion being killed for using too much memory. The default memory allocation in CWL is 256M, so that is also a good choice for the mksquashfs mem setting.

Singularity loop device errors

With singularity v3.9.1 and earlier, containers may fail intermittently at startup with an error message similar to the following in the container log’s stderr.txt (line breaks added):

FATAL:   container creation failed:
 mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error:
 while mounting image /proc/self/fd/3:
 failed to find loop device:
 could not attach image file to loop device:
 failed to set loop flags on loop device:
 resource temporarily unavailable

This problem is addressed in singularity v3.9.2. For details, please see Arvados issue #18489 and singularity PR #458.

Previous: Set up a compute node with Docker Next: Install the Slurm dispatcher

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.