Arvados™ Docs
  1. Topics
    1. Arvados components
  2. Storage in Keep
    1. Introduction to Keep
    2. Keep components overview
    3. Keep clients
    4. Data lifecycle
    5. Manifest format
  3. Computation with Crunch
    1. Computing with Crunch
    2. Dispatching containers to cloud VMs
    3. Dispatching containers to HPC
    4. Singularity
  4. Other
    1. Permission model
    2. Federation

Singularity

Arvados can be configured to use Singularity instead of Docker to execute containers on cloud nodes or a Slurm/LSF cluster. Singularity may be preferable due to its simpler installation and lack of long-running daemon process and special system users/groups. For on premises Slurm/LSF clusters, see the Set up a compute node with Singularity page. For cloud compute clusters, see the Build a cloud compute node image page.

Design overview

When Arvados is configured to use Singularity as the runtime engine for Crunch, containers are executed by Singularity. The images specified in workflows and tool definitions must be Docker images uploaded via arv-keepdocker or arvados-cwl-runner. When Singularity is the runtime engine, these images are converted to Singularity format (.sif) at runtime, as needed.

To avoid repeating this conversion work unnecessarily, the .sif files are cached in Keep. This is done on a per-user basis. If it does not exist yet, a new Arvados project named .cache is automatically created in the user’s home project. Similarly, a subproject named auto-generated singularity images will be created in the .cache project. The automatically generated .sif files are stored in collections in that project, with an expiration date two weeks in the future. If the cached image exists when Crunch runs a new container, the expiration date will be pushed out, so that it is always 2 weeks in the future from the most recent start of a container using the image.

It is safe to empty out or even remove the .cache project or any of its contents; if necessary the cache projects and the .sif files will automatically be regenerated.

Notes

  • Programs running in Singularity containers may behave differently than when run in Docker, due to differences between Singularity and Docker. For example, the root (image) filesystem is read-only in a Singularity container. Programs that attempt to write outside a designated output or temporary directory are likely to fail.
  • When using Singularity as the runtime engine, the compute node needs to have a compatible Singularity executable installed, as well as the mksquashfs program used to convert Docker images to Singularity’s .sif format. The Arvados compute node image build script includes these executables since Arvados 2.3.0.

Limitations

Arvados Singularity support is a work in progress. These are the current limitations of the implementation:

  • Even when using the Singularity runtime, users’ container images are expected to be saved in Docker format. Specifying a .sif file as an image when submitting a container request is not yet supported.
  • Arvados’ Singularity implementation does not yet limit the amount of memory available in a container. Each container will have access to all memory on the host where it runs, unless memory use is restricted by Slurm/LSF.
  • The Docker ENTRYPOINT instruction is ignored.
  • Arvados is tested with Singularity version 3.10.4. Other versions may not work.
Previous: Dispatching containers to HPC Next: Permission model

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.