Crunch is the name for the Arvados system for managing computation. It provides an abstract API to various clouds and HPC resource allocation and scheduling systems, and integrates closely with Keep storage and the Arvados permission system.
Committed
state.container_uuid
field. If the same request has been submitted in the past, it may reuse an existing container. The reuse behavior can be suppressed with use_existing: false
in the container request.Queued
state and submit a container executor to the underlying work queuing system (such as Slurm).Complete
state. If the container execution was interrupted or lost due to system failure, it will go into the Cancelled
state.Final
state.output_uuid
field of the container request contains the uuid of output collection produced by container request.The runtime_constraints
section of a container specifies working RAM (ram
) and Keep cache (keep_cache_ram
). If not specified, containers get a default Keep cache (container_default_keep_cache_ram
, default 256 MiB). The total RAM requested for a container is the sum of working RAM, Keep cache, and an additional RAM reservation configured by the admin (ReserveExtraRAM
in the dispatcher configuration, default zero).
The total RAM request is used to schedule containers onto compute nodes. RAM allocation limits are enforced using kernel controls such as cgroups. A container which requests 1 GiB RAM will only be permitted to allocate up to 1 GiB of RAM, even if scheduled on a 4 GiB node. On HPC systems, a multi-core node may run multiple containers at a time.
When running on the cloud, the memory request (along with CPU and disk) is used to select (and possibly boot) an instance type with adequate resources to run the container. Instance type RAM is derated 5% from the published specification to accomodate virtual machine, kernel and system services overhead.
For example, for a 3 GiB request, default Keep cache, and no extra RAM reserved:
(3072 + 256) * 1.0526 = 3494 MiBTo run this container, the instance type must have a published RAM size of at least 3494 MiB.
For example, for a 3.75 GiB node, default Keep cache, and no extra RAM reserved:
(3840 * 0.95) – 256 = 3392 MiBTo run on this instance type, the container can request at most 3392 MiB of working RAM.
The content of this documentation is licensed under the
Creative
Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the
Apache License, Version 2.0.