This page describes the requirements for a compute node in a Slurm or LSF cluster that will run containers dispatched by
arvados-dispatch-lsf. If you are installing a cloud cluster, refer to Build a cloud compute node image.
These instructions apply when Containers.RuntimeEngine is set to
singularity, refer to Set up a compute node with Docker when running
Please refer to the Singularity documentation in the Architecture section.
This page describes how to configure a compute node so that it can be used to run containers dispatched by Arvados on a static cluster. These steps must be performed on every compute node.
# yum install python-arvados-fuse crunch-run squashfs-tools
# apt-get install python-arvados-fuse crunch-run squashfs-tools
If you want to use NVIDIA GPUs, install the CUDA toolkit.
In addition, you also must install the NVIDIA Container Toolkit:
DIST=$(. /etc/os-release; echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | \ sudo apt-key add - curl -s -L https://nvidia.github.io/libnvidia-container/$DIST/libnvidia-container.list | \ sudo tee /etc/apt/sources.list.d/libnvidia-container.list sudo apt-get update apt-get install libnvidia-container1 libnvidia-container-tools nvidia-container-toolkit
Follow the Singularity installation instructions. Make sure
mksquashfs are working:
$ singularity version 3.7.4 $ mksquashfs -version mksquashfs version 4.3-git (2014/06/09) [...]
Containers.RuntimeEngine in your cluster configuration:
# Container runtime: "docker" (default) or "singularity" RuntimeEngine: singularity
Docker images are converted on the fly by
mksquashfs, which can consume a considerable amount of RAM. The RAM usage of mksquashfs can be restricted in
/etc/singularity/singularity.conf with a line like
mksquashfs mem = 256M. The amount of memory made available for mksquashfs should be configured lower than the smallest amount of memory requested by a container on the cluster to avoid the conversion being killed for using too much memory. The default memory allocation in CWL is 256M, so that is also a good choice for the
mksquashfs mem setting.
With singularity v3.9.1 and earlier, containers may fail intermittently at startup with an error message similar to the following in the container log’s
stderr.txt (line breaks added):
FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: failed to set loop flags on loop device: resource temporarily unavailable
The content of this documentation is licensed under the
Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.