Using Preemptible instances

This page describes how to enable preemptible instances. Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance.

Currently Arvados supports preemptible instances using AWS spot instances.

Configuration

To use preemptible instances, set UsePreemptibleInstances: true and add entries to InstanceTypes with Preemptible: true to config.yml. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The Price for preemptible instances is the maximum bid price, the actual price paid is dynamic and may be lower. For example:

Clusters:
  ClusterID: 
    Containers:
      UsePreemptibleInstances: true
    InstanceTypes:
      m4.large:
	Preemptible: false
        ProviderType: m4.large
        VCPUs: 2
        RAM: 8GiB
        AddedScratch: 32GB
        Price: 0.1
      m4.large.spot:
	Preemptible: true
        ProviderType: m4.large
        VCPUs: 2
        RAM: 8GiB
        AddedScratch: 32GB
        Price: 0.1

When UsePreemptibleInstances is enabled, child containers (workflow steps) will automatically be made preemptible. Note that because preempting the workflow runner would cancel the entire workflow, the workflow runner runs in a reserved (non-preemptible) instance.

If you are using arvados-dispatch-cloud no additional configuration is required.

If you are using the legacy Nodemanager, see below .

Preemptible instances on AWS

For general information, see using Amazon EC2 spot instances .

Permissions

When requesting spot instances, Amazon’s API may return an authorization error depending on how users and permissions are set on the account. If this is the case check logs for this error:

BaseHTTPError: AuthFailure.ServiceLinkedRoleCreationNotPermitted: The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances.

The account needs to have a service linked role created. This can be done by logging into the AWS account, go to IAM Management → Roles and create the AWSServiceRoleForEC2Spot role by clicking on the Create button, selecting EC2 service and EC2 - Spot Instances use case.

Cost Tracking

Amazon’s Spot instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that nodemanager uses for now, as it doesn’t include any pricing data to the spot instance request.

The real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on Amazon’s User Guide.

Nodemanager

If you are using the legacy Nodemanager, its config file must also declare preemptible instance sizes, which must match the API server’s InstanceTypes:

[Size m4.large]
cores = 2
scratch = 32000

[Size m4.large.spot]
cores = 2
instance_type = m4.large
preemptible = true
scratch = 32000

Previous: Configuring storage classes Next: Testing cloud configuration