Keepstore can store data in one or more Azure Storage containers.
Before starting the configuration of individual keepstore servers is good to have an idea of the keepstores servers’ final layout. One key decision is the amount of servers and type of VM to run. Azure may change over time the bandwith capacity of each type. After conducting some empirical saturation tests, the conclusion was that the bandwith is proportional to the amount of cores with some exceptions. As a rule of thumb, is better to invest resources in more cores instead of memory or IOps.
Another decision is how many VMs should be running keepstore. For example there could be 8 VMs with one core each or one machine with 8 cores. Or anything in between. Assuming is the same cost for Cloud resources, there is always the benefit of distributing the risk of faulty VMs. The recommendation is to start with 2 VMs and expand in pairs. Having a minimum of 2 cores each. The total amount of VMs will be a function of the budget and the pipeline traffic to avoid saturation during periods of high usage. Standard D v3 family is a balanced choice, making Standard_D2_v3 the 2-core option
There are many options for storage accounts. You can read details from Azure on their documentation https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction. The type of storage and access tier will be a function of the budget and desired responsiveness. A balanced option is to have General-purpose Standard Storage account and use Blob storage, hot access tiers.
Keepstore can be configure to reflect the level of underlaying redundancy the storage will have. This is call data replication option. For example LRS (Locally Redundant Storage) saves 3 copies of the data. There desired redundancy can be chosen at the keepstore layer or at the Storage Accunt layer. The decision where the redundancy will be done and the type of Storage Account data replication (LRS, ZRS, GRS and RA-GRS) has trade-offs. Please read more on https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy and decide what is best for your needs.
Using the Azure web portal or command line tool, create or choose a storage account with a suitable redundancy profile and availability region. Use the storage account keys to create a new container.
~$ azure config mode arm
~$ az login
~$ az group create exampleGroupName eastus2
~$ az storage account create --sku Standard_LRS --kind BlobStorage --encryption-services blob --access-tier Hot --https-only true --location eastus2 --resource-group exampleGroupName --name exampleStorageAccountName
~$ az storage account keys list --resource-group exampleGroupName --account-name exampleStorageAccountName
[
{
"keyName": "key1",
"permissions": "Full",
"value": "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz=="
},
{
"keyName": "key2",
"permissions": "Full",
"value": "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy=="
}
]
~$ AZURE_STORAGE_ACCOUNT="exampleStorageAccountName" \
AZURE_STORAGE_ACCESS_KEY="zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz==" \
azure storage container create --name exampleContainerName
Note that Keepstore services may be configured to use multiple Azure Storage accounts and multiple containers within a storage account.
Volumes are configured in the Volumes section of the cluster configuration file.
Note that each volume has a UUID, like zzzzz-nyw5e-0123456789abcde. You assign these manually: replace zzzzz with your Cluster ID, and replace 0123456789abcde with an arbitrary unique string of 15 alphanumerics. Once assigned, UUIDs should not be changed.
Essential configuration values are highlighted in red. Remaining parameters are provided for documentation, with their default values.
Volumes:
ClusterID-nyw5e-000000000000000:
AccessViaHosts:
# This section determines which keepstore servers access the
# volume. In this example, keep0 has read/write access, and
# keep1 has read-only access.
#
# If the AccessViaHosts section is empty or omitted, all
# keepstore servers will have read/write access to the
# volume.
"http://keep0.ClusterID.example.com:25107": {}
"http://keep1.ClusterID.example.com:25107": {ReadOnly: true}
Driver: Azure
DriverParameters:
# Storage account name and secret key, used for
# authentication.
StorageAccountName: exampleStorageAccountName
StorageAccountKey: zzzzzzzzzzzzzzzzzzzzzzzzzz
# Storage container name.
ContainerName: exampleContainerName
# The cloud environment to use,
# e.g. "core.chinacloudapi.cn". Defaults to
# "core.windows.net" if blank or omitted.
StorageBaseURL: ""
# Time to wait for an upstream response before failing the
# request.
RequestTimeout: 10m
# Time to wait before retrying a failed "list blobs" Azure
# API call.
ListBlobsRetryDelay: 10s
# Maximum attempts at a "list blobs" Azure API call before
# giving up.
ListBlobsMaxAttempts: 12
# If non-zero, use multiple concurrent requests (each
# requesting MaxGetBytes bytes) when retrieving data. If
# zero or omitted, get the entire blob with one request.
#
# Normally this is zero but if you find that 4 small
# requests complete faster than a single large request, for
# example, you might set this to 16777216 (64 MiB รท 4).
MaxGetBytes: 0
# Time to wait for an unexpectedly empty blob to become
# non-empty. Azure's create-and-write operation is not
# atomic. The default value typically allows concurrent GET
# and PUT requests to succeed despite the race window.
WriteRaceInterval: 15s
# Time to wait between GET attempts while waiting for
# WriteRaceInterval to expire.
WriteRacePollTime: 1s
# How much replication is provided by the underlying storage
# container. This is used to inform replication decisions at
# the Keep layer.
Replication: 3
# If true, do not accept write or trash operations, even if
# AccessViaHosts.*.ReadOnly is false.
#
# If false or omitted, enable write access (subject to
# AccessViaHosts.*.ReadOnly, where applicable).
ReadOnly: false
# Storage classes to associate with this volume. See "Storage
# classes" in the "Admin" section of doc.arvados.org.
StorageClasses: null
The content of this documentation is licensed under the
Creative
Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the
Apache License, Version 2.0.