Computing with Crunch

Crunch is the name for the Arvados system for managing computation. It provides an abstract API to various clouds and HPC resource allocation and scheduling systems, and integrates closely with Keep storage and the Arvados permission system.

Container API

Note: although the preferred API for Arvados going forward, the Container API may not yet be available on all installations.

1. To submit work, create a container request in the Committed state.
2. The system will fufill the container request by creating or reusing a Container object and assigning it to the container_uuid field. If the same request has been submitted in the past, it may reuse an existing container. The reuse behavior can be suppressed with use_existing: false in the container request.
3. The dispatcher process will notice a new container in Queued state and submit a container executor to the underlying work queuing system (such as SLURM).
4. The container executes. Upon termination the container goes into the Complete state. If the container execution was interrupted or lost due to system failure, it will go into the Cancelled state.
5. When the container associated with the container request is completed, the container request will go into the Final state.
6. The output_uuid field of the container request contains the uuid of output collection produced by container request.

Job API (deprecated)

1. To submit work, create a job . If the same job has been submitted in the past, it will return an existing job in Completed state.
2. The dispatcher process will notice a new job in Queued state and attempt to allocate nodes to run the job.
3. The job executes.
4. Retrieve the output field with the portable data hash of the collection with the output files of the job.

---

Previous: Storage in Keep Next: api_client_authorizations