Arvados™ Docs
WARNING - you are viewing the documentation for an old version of Arvados. For the latest version, click here.
  1. Topics
    1. Arvados admin overview
  2. Users and Groups
    1. User management
    2. User management at the CLI
    3. Reassign user data ownership
    4. Link user accounts
    5. Group management
    6. Configuring federation
    7. Migrating users to federated accounts
    8. Changing upstream login providers
    9. Synchronizing external groups
    10. Securing API access with scoped tokens
    11. Automatic logout and token expiration
    12. User activity report
  3. Monitoring
    1. Logging
    2. Metrics
    3. Health checks
    4. Management token
  4. Data Management
    1. Configuring collection versioning
    2. Configuring collection's managed properties
    3. Restricting upload or download
    4. Balancing Keep servers
    5. Preventing container reuse
    6. Logs table management
    7. User properties vocabulary
    8. Configuring storage classes
    9. Recovering data
    10. Measuring deduplication
  5. Cloud
    1. Using Preemptible instances
    2. Testing cloud configuration
  6. Other
    1. Migrating from Docker 1.9
    2. Upgrading to Containers API

Upgrading to Containers API

The “containers” API is the recommended way to submit compute work to Arvados. It supersedes the “jobs” API, which is end-of-life in Arvados 2.0.

Benefits over the “jobs” API

  • Simpler and more robust execution with fewer points of failure
  • Automatic retry for containers that fail to run to completion due to infrastructure errors
  • Scales to thousands of simultaneous containers
  • Able to support alternate schedulers/dispatchers in addition to slurm
  • Improved logging, different streams logs/metrics stored in different files in the log collection
  • Records more upfront detail about the compute node, and additional metrics (such as available disk space over the course of the container run)
  • Better behavior when deciding whether to reuse past work — pick the oldest container that matches the criteria
  • Can reuse running containers between workflows, cancelling a workflow will not cancel containers that are shared with other workflows
  • Supports setting time-to-live on intermediate output collections for automatic cleanup
  • Supports “secret” inputs, suitable for passwords or access tokens, which are hidden from the API responses and logs, and forgotten after use
  • Does not require “git” for dispatching work

Differences from the “jobs” API

Containers cannot reuse jobs (but can reuse other containers)

Uses the service crunch-dispatch-slurm instead of crunch-dispatch.rb

Non-CWL Arvados “pipeline templates” are not supported with containers. Pipeline templates should be rewritten in CWL and registered as “Workflows”.

The containers APIs is incompatible with the jobs API, code which integrates with the “jobs” API must be updated to work with containers

Containers have network access disabled by default

The keep mount only exposes collections which are explicitly listed as inputs

Migrating to “containers” API

Run your workflows using arvados-cwl-runner --api=containers (only necessary if both the jobs and containers APIs are enabled, if the jobs API is disabled, it will use the containers API automatically)

Register your workflows so they can be run from workbench using arvados-cwl-runner --api=containers --create-workflow

Read Migrating running CWL on jobs API to containers API

Use arv:APIRequirement: {} in the requirements section of your CWL file to enable network access for the container (see Arvados CWL Extensions)

For examples on how to manage container requests with the Python SDK, see Python cookbook

Previous: Migrating from Docker 1.9

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.