Arvados API Client

The Arvados Python SDK provides a complete client interface to the Arvados API. You can use this client interface directly to send requests to your Arvados API server, and many of the higher-level interfaces in the Python SDK accept a client object in their constructor for their use. Any Arvados software you write in Python will likely use these client objects.

This document explains how to instantiate the client object, and how its methods map to the full Arvados API. Refer to the API documentation for full details about all available resources and methods. The rest of the Python SDK documentation after this covers the higher-level interfaces it provides.

Initializing the API client

In the simplest case, you can import the arvados module and call its api method with an API version number:

import arvados
arv_client = arvados.api('v1')

When called this way, the SDK gets Arvados API credentials from the first source it finds in this list:

The environment variables ARVADOS_API_HOST, ARVADOS_API_TOKEN, and ARVADOS_API_HOST_INSECURE.
The settings.conf file under the directories listed in systemd’s CONFIGURATION_DIRECTORY environment variable.
The arvados/settings.conf file under the directory in the XDG_CONFIG_HOME environment variable. This defaults to ~/.config/arvados/settings.conf if XDG_CONFIG_HOME is not set.
The arvados/settings.conf file under the directories in the XDG_CONFIG_DIRS environment variable.

You can alternatively pass these settings as arguments:

import arvados
arv_client = arvados.api(
    'v1',
    host='api.arvados.example.com',
    token='ExampleToken',
    insecure=False,
)

Either way, you can now use the arv_client object to send requests to the Arvados API server you specified, using the configured token. The client object queries the API server for its supported API version and methods, so this client object will always support the same API the server does, even when there is a version mismatch between it and the Python SDK.

Resources, methods, and requests

The API client has a method that corresponds to each type of resource supported by the Arvados API server (listed in the documentation sidebar). You call these methods without any arguments. They return a resource object you use to call a method on that resource type.

Each resource object has a method that corresponds to each API method supported by that resource type. You call these methods with the keyword arguments and values documented in the API reference. They return an API request object.

Each API request object has an execute() method. If it succeeds, it returns the kind of object documented in the API reference for that method. Usually that’s a dictionary with details about the object you requested. If there’s a problem, it raises an exception.

Putting it all together, basic API requests usually look like:

arv_object = arv_client.resource_type().api_method(
    argument=...,
    other_argument=...,
).execute()

Later sections detail how to call common resource methods in the API with more concrete examples. Additional methods may be available on specific resource types.

Retrying failed requests

If you execute an API request and it fails because of a temporary error like a network problem, the SDK waits with randomized exponential back-off, then retries the request. You can specify the maximum number of retries by passing a num_retries integer to either arvados.api or the execute() method; the SDK will use whichever number is greater. The default number of retries is 10, which means that an API request could take up to about 35 minutes if the temporary problem persists that long. To disable automatic retries, just pass num_retries=0 to arvados.api:

import arvados
arv_client = arvados.api('v1', num_retries=0, ...)

get method

To fetch a single Arvados object, call the get method of the resource type. You must pass a uuid argument string that identifies the object to fetch. The method returns a dictionary with the object’s fields.

# Get a workflow and output its Common Workflow Language definition
workflow = api.workflows().get(uuid='zzzzz-7fd4e-12345abcde67890').execute()
print(workflow['definition'])

You can pass a select argument that’s a list of field names to return in the included object. Doing this avoids the overhead of de/serializing and transmitting data that you won’t use. Skipping a large field over a series of requests can yield a noticeable performance improvement.

# Get a workflow and output its name and description.
# Don't load the workflow definition, which might be large and we're not going to use.
workflow = api.workflows().get(
    uuid='zzzzz-7fd4e-12345abcde67890',
    select=['name', 'description'],
).execute()
print(f"## {workflow['name']} ##\n\n{workflow['description']}")

# ERROR: This raises a KeyError because we didn't load this field in
# the `select` argument.
workflow['created_at']

list method

To fetch multiple Arvados objects of the same type, call the list method for that resource type. The list method takes a number of arguments. Refer to the list method API reference for details about them. The method returns a dictionary also documented at the bottom of that section. The most interesting field is 'items', which is a list of dictionaries where each one corresponds to an Arvados object that matched your search. To work with a single page of results:

# Output the exit codes of the 10 most recently run containers.
container_list = arv_client.containers().list(
    limit=10,
    order=['finished_at desc'],
).execute()
for container in container_list['items']:
    print(f"{container['uuid']}: {container['exit_code']}")

If you need to retrieve all of the results for a query, you may need to call the list method multiple times: the default limit is 100 items, and the API server will never return more than 1000. The SDK function arvados.util.keyset_list_all can help orchestrate this for you. Call it with the list method you want to query (don’t call it yourself!) and the same keyword arguments you would pass to that method. You can control ordering by passing an order_key string that names the field to use, and an ascending bool that indicates whether that key should be sorted in ascending or descending order. The function returns an iterator of dictionaries, where each dictionary corresponds to an Arvados object that matched your query.

# Output all the portable data hashes in a project.
project_data = set()
for collection in arvados.util.keyset_list_all(
    # Note we pass the `list` method without calling it
    arv_client.collections().list,
    # The UUID of the project we're searching
    filters=[['owner_uuid', '=', 'zzzzz-j7d0g-12345abcde67890']],
):
    project_data.add(collection['portable_data_hash'])
print('\n'.join(project_data))

When you list many objects, the following can help improve performance:

Call the list method with count='none' to avoid the overhead of counting all results with each request.
Call the list method with a select argument to only request the data you need. This cuts out some overhead from de/serializing and transferring data you won’t use.

create method

To create a new Arvados object, call the create method for that resource type. You must pass a body dictionary with a single item. Its key is the resource type you’re creating as a string, and its value is dictionary of data fields for that resource. The method returns a dictionary with the new object’s fields.

If the resource type has a name field, you may pass an ensure_unique_name boolean argument. If true, the method will automatically update the name of the new object to make it unique if necessary.

# Create a new project and output its UUID.
project = arv_client.groups().create(
    body={
        'group': {
            'name': 'Python SDK Test Project',
            'group_class': 'project',
        },
    },
    ensure_unique_name=True,
).execute()
print(project['uuid'])

update method

To modify an existing Arvados object, call the update method for that resource type. You must pass a uuid string argument that identifies the object to update, and a body dictionary with a single item. Its key is the resource type you’re creating as a string, and its value is dictionary of data fields to update on the resource. The method returns a dictionary with the updated object’s fields.

# Update the name of a container request,
# finalize it to submit it to Crunch for processing,
# and output its priority.
submitted_container_request = arv_client.container_requests().update(
    uuid='zzzzz-xvhdp-12345abcde67890',
    body={
        'container_request': {
            'name': 'Container Request Committed by Python SDK',
            'state': 'Committed',
        },
    },
    ensure_unique_name=True,
).execute()
print(submitted_container_request['priority'])

delete method

To delete an existing Arvados object, call the delete method for that resource type. You must pass a uuid string argument that identifies the object to delete. The method returns a dictionary with the deleted object’s fields.

# Delete a collection and output its name
deleted_collection = arv_client.collections().delete(
    uuid='zzzzz-4zz18-12345abcde67890',
).execute()
print(deleted_collection['name'])

For resource types that support being trashed, you can untrash the object by calling the resource type’s untrash method with a uuid argument identifying the object to restore.

Previous: Installation Next: Code cookbook