Arvados federation enables clients to transparently read, create and manipulate objects and collections across clusters in different regions or organizations. Federation supports workfows that integrate and analyze data across multiple clusters by sending computation to where the data is, reducing the technical and legal barriers to analyzing large, sensitive data sets.
This feature is under development. Support for federation is limited to certain types of requests. The behaviors described here should not be interpreted as a stable API.
Detailed configuration information is available on the federation admin section.
Clusters are identified by a five-digit alphanumeric id (numbers and lowercase letters). There are 36 5 = 60466176 possible cluster identifiers.
Cluster identifiers are mapped API server hosts one of two ways:
arvadosapi.com domain. For example, the API server for the cluster
pirca can be found at
pirca.arvadosapi.com. To register a cluster id for free under
arvadosapi.com, contact firstname.lastname@example.org
RemoteClusters section of
/etc/arvados/config.yml (for arvados-controller)
Clusters: clsr1: RemoteClusters: clsr2: Host: api.cluster2.example Proxy: true clsr3: Host: api.cluster3.example Proxy: true
In this example, the cluster
clsr1 is configured to contact
api.cluster2.example for requests involving
api.cluster3.example for requests involving
The goal is for a federated user to have a single identity across the cluster federation. This identity is a user account on a specific “home cluster”. When arvados-controller contacts a remote cluster, the remote cluster verifies the user’s identity (see below) and then creates a mirror of the user account with the same uuid of the user’s home cluster. On the remote cluster, permissions can then be granted to the federated user, and the federated user can create and own objects.
When making a request to the home cluster, authorization is established by looking up the API token in the
api_client_authorizations table to determine the user identity. When making a request to a remote cluster, we need to provide an API token which can be used to establish the user’s identity. The remote cluster will connect back to the home cluster to determine if the token valid and the user it corresponds to. However, we do not want to send along the same API token used for the original request. If the remote cluster is malicious or compromised, sending along user’s regular token would compromise the user account on the home cluster. Instead, the controller sends a “salted token”. The salted token is restricted to only to fetching the user account and group membership. The salted token consists of the uuid of the token in
api_client_authorizations and the SHA1 HMAC of the original token and the cluster id of remote cluster. To verify the token, the remote cluster contacts the home cluster and provides the token uuid, the hash, and its cluster id. The home cluster uses the uuid to look up the token re-computes the SHA1 HMAC of the original token and cluster id. If that hash matches, then the token is valid. To avoid having to re-validate the token on every request, it is cached for a short period.
The security properties of this scheme are:
In a LoginCluster federation, there is a central “home” called the LoginCluster, and one or more “satellite” clusters. The satellite clusters delegate their user management to the LoginCluster. Unlike the peer federation, satellite clusters implicitly trust the home cluster, so the “salted token” scheme is not used. Users arriving at a satellite cluster are redirected to the home cluster for login, the user token is issued by the LoginCluster, and then the user is sent back to the satellite cluster. Tokens issued by the LoginCluster are accepted by all clusters in the federation. All requests for user records on a satellite cluster is forwarded to the LoginCluster.
In the REST API, GET and PUT/PATCH requests are used to fetch and update records.
In the REST API, POST requests create new records, so there is no uuid to use for the cluster id. In this case, to create an object on a remote cluster, the request includes the
cluster_id parameter. The flow is otherwise the same as described above.
Each collection record has
manifest_text, which describes how to reassemble keep blocks into files as described in the Manifest format. Each block identifier in the manifest has an added signature which is used to confirm permission to read the block. To read a block from a keepstore server, the client must provide the block identifier, the signature, and the same API token used to retrieve the collection record.
See Federation signatures for details on how federation affects block signatures.
The content of this documentation is licensed under the
Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.