Planning and prerequisites

Before attempting installation, you should begin by reviewing supported platforms, choosing backends for identity, storage, and scheduling, and decide how you will distribute Arvados services onto machines. You should also choose an Arvados Cluster ID, choose your hostnames, and aquire TLS certificates. It may be helpful to make notes as you go along using one of these worksheets: New cluster checklist for AWSNew cluster checklist for AzureNew cluster checklist for on premises Slurm

The installation guide describes how to set up a basic standalone Arvados instance. Additional configuration for features including federation, collection versioning, managed properties, and storage classes are described in the Admin guide.

The Arvados storage subsystem is called “keep”. The compute subsystem is called “crunch”.

  1. Supported GNU/Linux distributions
  2. Choosing which components to install
  3. Identity provider
  4. Storage backend
  5. Container compute scheduler
  6. Hardware or virtual machines
  7. Arvados Cluster ID
  8. DNS and TLS

Supported GNU/Linux distributions

Distribution State Last supported Arvados version
CentOS 7 Supported Latest
Debian 11 (“bullseye”) Supported Latest
Debian 10 (“buster”) Supported Latest
Ubuntu 20.04 (“focal”) Supported Latest
Ubuntu 18.04 (“bionic”) Supported Latest
Ubuntu 16.04 (“xenial”) EOL 2.1.2
Debian 9 (“stretch”) EOL 2.1.2
Debian 8 (“jessie”) EOL 1.4.3
Ubuntu 14.04 (“trusty”) EOL 1.4.3
Ubuntu 12.04 (“precise”) EOL 8ed7b6dd5d4df93a3f37096afe6d6f81c2a7ef6e (2017-05-03)
Debian 7 (“wheezy”) EOL 997479d1408139e96ecdb42a60b4f727f814f6c9 (2016-12-28)
CentOS 6 EOL 997479d1408139e96ecdb42a60b4f727f814f6c9 (2016-12-28)

Arvados packages are published for current Debian releases (until the EOL date), current Ubuntu LTS releases (until the end of standard support), and the latest version of CentOS.

Choosing which components to install

Arvados consists of many components, some of which may be omitted (at the cost of reduced functionality.) It may also be helpful to review the Arvados Architecture to understand how these components interact.

Core
PostgreSQL database Stores data for the API server. Required.
API server + Controller Core Arvados logic for managing users, groups, collections, containers, and enforcing permissions. Required.
Keep (storage)
Keepstore Stores content-addressed blocks in a variety of backends (local filesystem, cloud object storage). Required.
Keepproxy Gateway service to access keep servers from external networks. Required to be able to use arv-put, arv-get, or arv-mount outside the private Arvados network.
Keep-web Gateway service providing read/write HTTP and WebDAV support on top of Keep. Required to access files from Workbench.
Keep-balance Storage cluster maintenance daemon responsible for moving blocks to their optimal server location, adjusting block replication levels, and trashing unreferenced blocks. Required to free deleted data from underlying storage, and to ensure proper replication and block distribution (including support for storage classes).
User interface
Workbench, Workbench2 Primary graphical user interface for working with file collections and running containers. Optional. Depends on API server, keep-web, websockets server.
Workflow Composer Graphical user interface for editing Common Workflow Language workflows. Optional. Depends on git server (arv-git-httpd).
Additional services
Websockets server Event distribution server. Required to view streaming container logs in Workbench.
Shell server Synchronize (create/delete/configure) Unix shell accounts with Arvados users. Optional.
Git server Arvados-hosted git repositories, with Arvados-token based authentication. Optional, but required by Workflow Composer.
Crunch (running containers)
arvados-dispatch-cloud Allocate and free cloud VM instances on demand based on workload. Optional, not needed for a static Slurm cluster such as on-premises HPC.
crunch-dispatch-slurm Run analysis workflows using Docker or Singularity containers distributed across a Slurm cluster. Optional, not needed for a Cloud installation, or if you wish to use Arvados for data management only.
crunch-dispatch-lsf Run analysis workflows using Docker or Singularity containers distributed across an LSF cluster. Optional, not needed for a Cloud installation, or if you wish to use Arvados for data management only.

Identity provider

Choose which backend you will use to authenticate users.

  • Google login to authenticate users with a Google account.
  • OpenID Connect (OIDC) if you have Single-Sign-On (SSO) service that supports the OpenID Connect standard.
  • LDAP login to authenticate users by username/password using the LDAP protocol, supported by many services such as OpenLDAP and Active Directory.
  • PAM login to authenticate users by username/password according to the PAM configuration on the controller node.

PostgreSQL

Arvados works well with a standalone PostgreSQL installation. When deploying on AWS, Aurora RDS also works but Aurora Serverless is not recommended.

Storage backend

Choose which backend you will use for storing and retrieving content-addressed Keep blocks.

  • File systems storage, such as ext4 or xfs, or network file systems such as GPFS or Lustre
  • Amazon S3, or other object storage that supports the S3 API including Google Cloud Storage and Ceph.
  • Azure blob storage

You should also determine the desired replication factor for your data. A replication factor of 1 means only a single copy of a given data block is kept. With a conventional file system backend and a replication factor of 1, a hard drive failure is likely to lose data. For this reason the default replication factor is 2 (two copies are kept).

A backend may have its own replication factor (such as durability guarantees of cloud buckets) and Arvados will take this into account when writing a new data block.

Container compute scheduler

Choose which backend you will use to schedule computation.

  • On AWS EC2 and Azure, you probably want to use arvados-dispatch-cloud to manage the full lifecycle of cloud compute nodes: starting up nodes sized to the container request, executing containers on those nodes, and shutting nodes down when no longer needed.
  • For on-premises HPC clusters using slurm use crunch-dispatch-slurm to execute containers with slurm job submissions.
  • For on-premises HPC clusters using LSF use crunch-dispatch-lsf to execute containers with slurm job submissions.
  • For single node demos, use crunch-dispatch-local to execute containers directly.

Hardware (or virtual machines)

Choose how to allocate Arvados services to machines. We recommend that each machine start with a clean installation of a supported GNU/Linux distribution.

For a production installation, this is a reasonable starting point:

Function Number of nodes Recommended specs
PostgreSQL database, Arvados API server, Arvados controller, Git, Websockets, Container dispatcher 1 16+ GiB RAM, 4+ cores, fast disk for database
Workbench, Keepproxy, Keep-web, Keep-balance 1 8 GiB RAM, 2+ cores
Keepstore servers 1 2+ 4 GiB RAM
Compute worker nodes 1 0+ Depends on workload; scaled dynamically in the cloud
User shell nodes 2 0+ Depends on workload

1 Should be scaled up as needed
2 Refers to shell nodes managed by Arvados that provide ssh access for users to interact with Arvados at the command line. Optional.

Note:

For a small demo installation, it is possible to run all the Arvados services on a single node. Special considerations for single-node installs will be noted in boxes like this.

Arvados Cluster ID

Each Arvados installation is identified by a cluster identifier, which is a unique 5-character lowercase alphanumeric string. There are 36 5 = 60466176 possible cluster identifiers.

  • For automated test purposes, use “z****”
  • For experimental/local-only/private clusters that won’t ever be visible on the public Internet, use “x****”
  • For long-lived clusters, we recommend reserving a cluster id. Contact info@curii.com for more information.

Here is one way to make a random 5-character string:

~$ tr -dc 0-9a-z </dev/urandom | head -c5; echo

You may also use a different method to pick the cluster identifier. The cluster identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your ClusterID. Whenever ClusterID appears in a configuration example, replace it with your five-character cluster identifier.

DNS entries and TLS certificates

The following services are normally public-facing and require DNS entries and corresponding TLS certificates. Get certificates from your preferred TLS certificate provider. We recommend using Let’s Encrypt. You can run several services on the same node, but each distinct DNS name requires a valid, matching TLS certificate.

This guide uses the following DNS name conventions. A later part of this guide will describe how to set up Nginx virtual hosts.
It is possible to use custom DNS names for the Arvados services.

Function DNS name
Arvados API ClusterID.example.com
Arvados Git server git.ClusterID.example.com
Arvados Webshell webshell.ClusterID.example.com
Arvados Websockets endpoint ws.ClusterID.example.com
Arvados Workbench workbench.ClusterID.example.com
Arvados Workbench 2 workbench2.ClusterID.example.com
Arvados Keepproxy server keep.ClusterID.example.com
Arvados Keep-web server download.ClusterID.example.com
and
*.collections.ClusterID.example.com or
*--collections.ClusterID.example.com or
collections.ClusterID.example.com (see the keep-web install docs)

Setting up Arvados is easiest when Wildcard TLS and wildcard DNS are available. It is also possible to set up Arvados without wildcard TLS and DNS, but not having a wildcard for keep-web (i.e. not having *.collections.ClusterID.example.com) comes with a tradeoff: it will disable some features that allow users to view Arvados-hosted data in their browsers. More information on this tradeoff caused by the CORS rules applied by modern browsers is available in the keep-web URL pattern guide.

The table below lists the required TLS certificates and DNS names in each scenario.

Wildcard TLS and DNS available Wildcard TLS available Other
TLS *.ClusterID.example.com
ClusterID.example.com
*.collections.ClusterID.example.com
*.ClusterID.example.com
ClusterID.example.com
ClusterID.example.com
git.ClusterID.example.com
webshell.ClusterID.example.com
ws.ClusterID.example.com
workbench.ClusterID.example.com
workbench2.ClusterID.example.com
keep.ClusterID.example.com
download.ClusterID.example.com
collections.ClusterID.example.com
DNS ClusterID.example.com
git.ClusterID.example.com
webshell.ClusterID.example.com
ws.ClusterID.example.com
workbench.ClusterID.example.com
workbench2.ClusterID.example.com
keep.ClusterID.example.com
download.ClusterID.example.com
*.collections.ClusterID.example.com
ClusterID.example.com
git.ClusterID.example.com
webshell.ClusterID.example.com
ws.ClusterID.example.com
workbench.ClusterID.example.com
workbench2.ClusterID.example.com
keep.ClusterID.example.com
download.ClusterID.example.com
collections.ClusterID.example.com
ClusterID.example.com
git.ClusterID.example.com
webshell.ClusterID.example.com
ws.ClusterID.example.com
workbench.ClusterID.example.com
workbench2.ClusterID.example.com
keep.ClusterID.example.com
download.ClusterID.example.com
collections.ClusterID.example.com

Note:

It is also possible to create your own certificate authority, issue server certificates, and install a custom root certificate in the browser. This is out of scope for this guide.


Previous: Arvados on GKE Next: Arvados package repositories

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.