Arvados™ Docs
WARNING - you are viewing the documentation for an old version of Arvados. For the latest version, click here.
  1. Topics
    1. Arvados admin overview
  2. Users and Groups
    1. User management
    2. User management at the CLI
    3. Reassign user data ownership
    4. Link user accounts
    5. Group management
    6. Configuring federation
    7. Migrating users to federated accounts
    8. Changing upstream login providers
    9. Synchronizing external groups
    10. Securing API access with scoped tokens
    11. Automatic logout and token expiration
    12. User activity report
  3. Monitoring
    1. Logging
    2. Metrics
    3. Health checks
    4. Management token
  4. Data Management
    1. Configuring collection versioning
    2. Configuring collection's managed properties
    3. Restricting upload or download
    4. Balancing Keep servers
    5. Preventing container reuse
    6. Logs table management
    7. Metadata vocabulary
    8. Configuring storage classes
    9. Recovering data
    10. Measuring deduplication
  5. Cloud
    1. Using Preemptible instances
    2. Testing cloud configuration
  6. Other
    1. Migrating from Docker 1.9
    2. Upgrading to Containers API

Using Preemptible instances

This page describes how to enable preemptible instances. Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance.

Currently Arvados supports preemptible instances using AWS and Azure spot instances.

Configuration

First, configure some InstanceTypes that have Preemptible: true. For a preemptible instance, Price determines the maximum bid price; the actual price paid is dynamic and will likely be lower.

Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. To do this automatically, use PreemptiblePriceFactor to enable a preemptible version of each listed type, using the given factor to set the maximum bid price relative to the non-preemptible price. Alternatively, you can configure preemptible instance types explicitly. For example, the following two configurations are equivalent:

Clusters:
  ClusterID:
    Containers:
      PreemptiblePriceFactor: 0.8
    InstanceTypes:
      m4.large:
        ProviderType: m4.large
        VCPUs: 2
        RAM: 8GiB
        AddedScratch: 32GB
        Price: 0.1

Clusters:
  ClusterID:
    InstanceTypes:
      m4.large:
        ProviderType: m4.large
        VCPUs: 2
        RAM: 8GiB
        AddedScratch: 32GB
        Price: 0.1
      m4.large.preemptible:
        Preemptible: true
        ProviderType: m4.large
        VCPUs: 2
        RAM: 8GiB
        AddedScratch: 32GB
        Price: 0.08

Next, you can choose to enable automatic use of preemptible instances:

Clusters:
  ClusterID:
    Containers:
      AlwaysUsePreemptibleInstances: true

If AlwaysUsePreemptibleInstances is “true”, child containers (workflow steps) will always select preemptible instances, regardless of user option.

If AlwaysUsePreemptibleInstances is “false” (the default) or unspecified, preemptible instance are used when requested by the user.

Note that regardless of the value of AlwaysUsePreemptibleInstances, the top level workflow runner container always runs in a reserved (non-preemptible) instance, to avoid situations where the workflow runner is killed requiring the entire to be restarted.

No additional configuration is required, arvados-dispatch-cloud will now start preemptible instances where appropriate.

Cost Tracking

Preemptible instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that arvados-dispatch-cloud uses for now, as it doesn’t include any pricing data to the spot instance request.

For AWS, the real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on Amazon’s User Guide.

Preemptible instances on AWS

For general information, see using Amazon EC2 spot instances.

Permissions

When requesting spot instances, Amazon’s API may return an authorization error depending on how users and permissions are set on the account. If this is the case check logs for this error:

BaseHTTPError: AuthFailure.ServiceLinkedRoleCreationNotPermitted: The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances.

The account needs to have a service linked role created. This can be done by logging into the AWS account, go to IAM Management → Roles and create the AWSServiceRoleForEC2Spot role by clicking on the Create button, selecting EC2 service and EC2 - Spot Instances use case.

Preemptible instances on Azure

For general information, see Use Spot VMs in Azure.

When starting preemptible instances on Azure, Arvados configures the eviction policy to ‘delete’, with max price set to ‘-1’. This has the effect that preemptible VMs will not be evicted for pricing reasons. The price paid for the instance will be the current spot price for the VM type, up to a maximum of the price for a standard, non-spot VM of that type.

Please note that Azure provides no SLA for preemptible instances. Even in this configuration, preemptible instances can still be evicted for capacity reasons. If that happens and a container is aborted, Arvados will try to restart it, subject to the usual retry rules.

Spot pricing is not available on ‘B-series’ VMs, those should not be defined in the configuration file with the Preemptible flag set to true. Spot instances have a separate quota pool, make sure you have sufficient quota available.

Previous: Measuring deduplication Next: Testing cloud configuration

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.