Install Keepstore servers

Keepstore provides access to underlying storage for reading and writing content-addressed blocks, with enforcement of Arvados permissions. Keepstore supports a variety of cloud object storage and POSIX filesystems for its backing store.

We recommend starting off with two Keepstore servers. Exact server specifications will be site and workload specific, but in general keepstore will be I/O bound and should be set up to maximize aggregate bandwidth with compute nodes. To increase capacity (either space or throughput) it is straightforward to add additional servers, or (in cloud environments) to increase the machine size of the existing servers.

By convention, we use the following hostname pattern:

Hostname

keep0.uuid_prefix.your.domain

keep1.uuid_prefix.your.domain

Keepstore servers should not be directly accessible from the Internet (they are accessed via keepproxy), so the hostnames only need to resolve on the private network.

Install Keepstore

On Debian-based systems:

~$ sudo apt-get install keepstore

On Red Hat-based systems:

~$ sudo yum install keepstore

Verify that Keepstore is functional:

~$ keepstore --version

Create config file

By default, keepstore will look for its configuration file at /etc/arvados/keepstore/keepstore.yml

You can override the configuration file location using the -config command line option to keepstore.

The following is a sample configuration file:

# Duration for which new permission signatures (returned in PUT
# responses) will be valid.  This should be equal to the API
# server's blob_signature_ttl configuration entry.
BlobSignatureTTL: 336h0m0s

# Local file containing the secret blob signing key (used to generate
# and verify blob signatures).  The contents of the key file must be
# identical to the API server's blob_signing_key configuration entry.
BlobSigningKeyFile: ""

# Print extra debug logging
Debug: false

# Maximum number of concurrent block deletion operations (per
# volume) when emptying trash. Default is 1.
EmptyTrashWorkers: 1

# Enable trash and delete features. If false, trash lists will be
# accepted but blocks will not be trashed or deleted.
# Keepstore does not delete data on its own.  The keep-balance
# service determines which blocks are candidates for deletion
# and instructs the keepstore to move those blocks to the trash.
EnableDelete: true

# Local port to listen on. Can be 'address:port' or ':port', where
# 'address' is a host IP address or name and 'port' is a port number
# or name.
Listen: :25107

# Format of request/response and error logs: "json" or "text".
LogFormat: json

# The secret key that must be provided by monitoring services when
# using the health check and metrics endpoints (/_health, /metrics).
ManagementToken: xyzzy

# Maximum RAM to use for data buffers, given in multiples of block
# size (64 MiB). When this limit is reached, HTTP requests requiring
# buffers (like GET and PUT) will wait for buffer space to be
# released.
#
# It should be set such that MaxBuffers * 64MiB + 10% fits
# comfortably in memory. On a host dedicated to running keepstore,
# divide total memory by 88MiB to suggest a suitable value. For example,
# if grep MemTotal /proc/meminfo reports MemTotal: 7125440 kB,
# compute 7125440 / (88 * 1024)=79 and configure MaxBuffers: 79
MaxBuffers: 128

# Maximum concurrent requests. When this limit is reached, new
# requests will receive 503 responses. Note: this limit does not
# include idle connections from clients using HTTP keepalive, so it
# does not strictly limit the number of concurrent connections. If
# omitted or zero, the default is 2 * MaxBuffers.
MaxRequests: 0

# Path to write PID file during startup. This file is kept open and
# locked with LOCK_EX until keepstore exits, so "fuser -k pidfile" is
# one way to shut down. Exit immediately if there is an error
# opening, locking, or writing the PID file.
PIDFile: ""

# Maximum number of concurrent pull operations. Default is 1, i.e.,
# pull lists are processed serially.  A pull operation copies a block
# from another keepstore server.
PullWorkers: 1

# Honor read requests only if a valid signature is provided.  This
# should be true, except for development use and when migrating from
# a very old version.
RequireSignatures: true

# Local file containing the Arvados API token used by keep-balance
# or data manager.  Delete, trash, and index requests are honored
# only for this token.
SystemAuthTokenFile: ""

# Path to server certificate file in X509 format. Enables TLS mode.
#
# Example: /var/lib/acme/live/keep0.example.com/fullchain
TLSCertificateFile: ""

# Path to server key file in X509 format. Enables TLS mode.
#
# The key pair is read from disk during startup, and whenever SIGHUP
# is received.
#
# Example: /var/lib/acme/live/keep0.example.com/privkey
TLSKeyFile: ""

# How often to check for (and delete) trashed blocks whose
# TrashLifetime has expired.
TrashCheckInterval: 24h0m0s

# Time duration after a block is trashed during which it can be
# recovered using an /untrash request.
TrashLifetime: 336h0m0s

# Maximum number of concurrent trash operations (moving a block to the
# trash, or permanently deleting it) . Default is 1, i.e., trash lists
# are processed serially.  If individual trash operations have high
# latency (eg some cloud platforms) you should increase this.
TrashWorkers: 1

Notes on storage management

On its own, a keepstore server never deletes data. The keep-balance service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash.

When a block is newly written, it is protected from deletion for the duration in BlobSignatureTTL. During this time, it cannot be trashed.

If keep-balance instructs keepstore to trash a block which is older than BlobSignatureTTL, and EnableDelete is true, the block will be moved to “trash”. A block which is in the trash is no longer accessible by read requests, but has not yet been permanently deleted. Blocks which are in the trash may be recovered using the “untrash” API endpoint. Blocks are permanently deleted after they have been in the trash for the duration in TrashLifetime.

Keep-balance is also responsible for balancing the distribution of blocks across keepstore servers by asking servers to pull blocks from other servers (as determined by their storage class and rendezvous hashing order). Pulling a block makes a copy. If a block is overreplicated (i.e. there are excess copies) after pulling, it will be subsequently trashed on the original server.

Configure storage volumes

Available storage volume types include POSIX filesystems and cloud object storage.

To use a POSIX filesystem, including both local filesystems (ext4, xfs) and network file system such as GPFS or Lustre, follow the setup instructions on Filesystem storage
If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions on S3 Object Storage
If you are using Azure Blob Storage, follow the setup instructions on Azure Blob Storage

Run keepstore as a supervised service

Install runit to supervise the keepstore daemon.

On Debian-based systems:

~$ sudo apt-get install runit

On Red Hat-based systems:

~$ sudo yum install runit

Install this script as the run script /etc/sv/keepstore/run for the keepstore service:

#!/bin/sh

exec 2>&1
GOGC=10 exec keepstore -config /etc/arvados/keepstore/keepstore.yml

Set up additional servers

Repeat the above sections to prepare volumes and bring up supervised services on each Keepstore server you are setting up.

Tell the API server about the Keepstore servers

The API server needs to be informed about the presence of your Keepstore servers.

First, if you don’t already have an admin token, create a superuser token.

On the API server, use the following commands:

~$ cd /var/www/arvados-api/current
$ sudo -u webserver-user RAILS_ENV=production bundle exec script/create_superuser_token.rb
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Configure your environment to run arv using the output of create_superuser_token.rb:

export ARVADOS_API_HOST=zzzzz.example.com
export ARVADOS_API_TOKEN=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Use this command to register each keepstore server you have installed. Make sure to update the service_host value.

~$ uuid_prefix=`arv --format=uuid user current | cut -d- -f1`
~$ echo "Site prefix is '$uuid_prefix'"
~$ read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice"
{
 "service_host":"keep0.$uuid_prefix.your.domain",
 "service_port":25107,
 "service_ssl_flag":false,
 "service_type":"disk"
}
EOF

Testing keep

Install the Python SDK

ARVADOS_API_HOST and ARVADOS_API_TOKEN must be set in the environment.

You should now be able to use arv-put to upload collections and arv-get to fetch collections:

$ echo "hello world!" > hello.txt

$ arv-put --portable-data-hash hello.txt
2018-07-12 13:35:25 arvados.arv_put[28702] INFO: Creating new cache file at /home/example/.cache/arvados/arv-put/1571ec0adb397c6a18d5c74cc95b3a2a
0M / 0M 100.0% 2018-07-12 13:35:27 arvados.arv_put[28702] INFO:

2018-07-12 13:35:27 arvados.arv_put[28702] INFO: Collection saved as 'Saved at 2018-07-12 17:35:25 UTC by example@example'
59389a8f9ee9d399be35462a0f92541c+53

$ arv-get 59389a8f9ee9d399be35462a0f92541c+53/hello.txt
hello world!

Previous: Install the controller Next: Filesystem storage