Arvados components
Services
Located in arvados/services except for Workbench which is located in arvados/apps/workbench.
| Component | Description |
|---|---|
| api | The API server is the core of Arvados. It is backed by a Postgres database and manages information such as metadata for storage, a record of submitted compute jobs, users, groups, and associated permissions. |
| arv-git-httpd | Provides a git+http interface to Arvados-managed git repositories, with permissions and authentication based on an Arvados API token. |
| arvados-dispatch-cloud | Provide elastic computing by creating and destroying cloud based virtual machines on compute demand. |
| crunch-dispatch-local | Get compute requests submitted to the API server and execute them locally. |
| crunch-dispatch-slurm | Get compute requests submitted to the API server and submit them to slurm. |
| crunch-run | Dispatched by crunch-dispatch, executes a single compute run: setting up a Docker container, running it, and collecting the output. |
| dockercleaner | Daemon for cleaning up Docker containers and images. |
| fuse | Filesystem in USErspace (FUSE) filesystem driver for Keep. |
| health | Health check proxy, contacts configured Arvados services at their health check endpoints and reports results. |
| keep-balance | Perform storage utilization reporting, optimization and garbage collection. Moves data blocks to their optimum location, ensures correct replication and storage class, and trashes unreferenced blocks. |
| keepproxy | Provides low-level access to keepstore services (block-level data access) for clients outside the internal (private) network. |
| keepstore | Provides access to underlying storage (filesystem or object storage such as Amazon S3 or Azure Blob) with Arvados permissions. |
| keep-web | Provides high-level WebDAV access to collections (file-level data access). |
| login-sync | Synchronize virtual machine users with Arvados users and permissions. |
| arvados-ws | Publishes API server change events over websockets. |
| workbench | Web application providing user interface to Arvados services. |
Tools
The arv command is located in arvados/sdk/ruby, the arv-* tools are located in arvados/sdk/python, the rest are located in arvados/tools.
| Component | Description |
|---|---|
| arv | Provides command line access to API, also provides some purpose utilities. |
| arv-copy | Copy a collection from one cluster to another |
| arv-get | Get files from a collection. |
| arv-keepdocker | Upload Docker images from local Docker daemon to Keep. |
| arv-ls | List files in a collection |
| arv-migrate-docker19 | Migrate Docker images in Keep from v1 format (Docker 1.9 or earlier) to v2 format (Docker 1.10 or later) |
| arv-normalize | Read manifest text on stdin and produce normalized manifest text on stdout. |
| arv-put | Upload files to a collection. |
| arv-ws | Print events from Arvados websocket event source. |
| arvbash | Helpful bash macros for using Arvados at the command line. |
| arvbox | Dockerized Arvados environment for development and testing. |
| crunchstat-summary | Read execution metrics (cpu %, ram, network, etc) collected from a compute container and produce a report. |
| keep-block-check | Given a list of keep block locators, check that each block exists on one of the configured keepstore servers and verify the block hash. |
| keep-exercise | Benchmarking tool to test throughput and reliability of keepstores under various usage patterns. |
| keep-rsync | Get lists of blocks from two clusters, copy blocks which exist on source cluster but are missing from destination cluster. |
| sync-groups | Take a CSV file listing with (group, user, permission) records and synchronize membership in Arvados groups. |
Next: Introduction to Keep