Arvados components

Services

Located in arvados/services.

Component Description
api The API server is the core of Arvados. It is backed by a Postgres database and manages information such as metadata for storage, a record of submitted compute jobs, users, groups, and associated permissions.
arvados-dispatch-cloud Provide elastic computing by creating and destroying cloud based virtual machines on compute demand.
crunch-dispatch-local Get compute requests submitted to the API server and execute them locally.
crunch-dispatch-slurm Get compute requests submitted to the API server and submit them to slurm.
crunch-run Dispatched by crunch-dispatch, executes a single compute run: setting up a Docker container, running it, and collecting the output.
dockercleaner Daemon for cleaning up Docker containers and images.
fuse Filesystem in USErspace (FUSE) filesystem driver for Keep.
health Health check proxy, contacts configured Arvados services at their health check endpoints and reports results.
keep-balance Perform storage utilization reporting, optimization and garbage collection. Moves data blocks to their optimum location, ensures correct replication and storage class, and trashes unreferenced blocks.
keepproxy Provides low-level access to keepstore services (block-level data access) for clients outside the internal (private) network.
keepstore Provides access to underlying storage (filesystem or object storage such as Amazon S3 or Azure Blob) with Arvados permissions.
keep-web Provides high-level WebDAV access to collections (file-level data access).
login-sync Synchronize virtual machine users with Arvados users and permissions.
arvados-ws Publishes API server change events over websockets.
workbench Web application providing user interface to Arvados services.

Tools

The arv command is located in arvados/sdk/ruby, the arv-* tools are located in arvados/sdk/python, the rest are located in arvados/tools.

Component Description
arv Provides command line access to API, also provides some purpose utilities.
arv-copy Copy a collection from one cluster to another
arv-get Get files from a collection.
arv-keepdocker Upload Docker images from local Docker daemon to Keep.
arv-ls List files in a collection
arv-migrate-docker19 Migrate Docker images in Keep from v1 format (Docker 1.9 or earlier) to v2 format (Docker 1.10 or later)
arv-normalize Read manifest text on stdin and produce normalized manifest text on stdout.
arv-put Upload files to a collection.
arv-ws Print events from Arvados websocket event source.
arvbash Helpful bash macros for using Arvados at the command line.
arvbox Dockerized Arvados environment for development and testing.
crunchstat-summary Read execution metrics (cpu %, ram, network, etc) collected from a compute container and produce a report.
keep-block-check Given a list of keep block locators, check that each block exists on one of the configured keepstore servers and verify the block hash.
keep-exercise Benchmarking tool to test throughput and reliability of keepstores under various usage patterns.
keep-rsync Get lists of blocks from two clusters, copy blocks which exist on source cluster but are missing from destination cluster.
sync-groups Take a CSV file listing with (group, user, permission) records and synchronize membership in Arvados groups.

Next: Introduction to Keep

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.