Arvados CWL Extensions

Arvados provides several extensions to CWL for workflow optimization, site-specific configuration, and to enable access the Arvados API.

To use Arvados CWL extensions, add the following $namespaces section at the top of your CWL file:

$namespaces:
  arv: "http://arvados.org/cwl#"
  cwltool: "http://commonwl.org/cwltool#"

For portability, Arvados extensions should go into the hints section of your CWL file, for example:

hints:
  arv:RunInSingleContainer: {}
  arv:RuntimeConstraints:
    keep_cache: 123456
    outputDirType: keep_output_dir
  arv:PartitionRequirement:
    partition: dev_partition
  arv:APIRequirement: {}
  cwltool:LoadListingRequirement:
    loadListing: shallow_listing
  arv:IntermediateOutput:
    outputTTL: 3600
  arv:ReuseRequirement:
    enableReuse: false
  cwltool:Secrets:
    secrets: [input1, input2]
  cwltool:TimeLimit:
    timelimit: 14400
  arv:WorkflowRunnerResources:
    ramMin: 2048
    coresMin: 2

The one exception to this is arv:APIRequirement, see note below.

arv:RunInSingleContainer

Indicates that a subworkflow should run in a single container and not be scheduled as separate steps.

arv:RuntimeConstraints

Set Arvados-specific runtime hints.

Field	Type	Description
keep_cache	int	Size of file data buffer for Keep mount in MiB. Default is 256 MiB. Increase this to reduce cache thrashing in situations such as accessing multiple large (64+ MiB) files at the same time, or performing random access on a large file.
outputDirType	enum	Preferred backing store for output staging. If not specified, the system may choose which one to use. One of local_output_dir or keep_output_dir

local_output_dir: Use regular file system local to the compute node. There must be sufficient local scratch space to store entire output; specify this with outdirMin of ResourceRequirement. Files are batch uploaded to Keep when the process completes. Most compatible, but upload step can be time consuming for very large files.

keep_output_dir: Use writable Keep mount. Files are streamed to Keep as they are written. Does not consume local scratch space, but does consume RAM for output buffers (up to 192 MiB per file simultaneously open for writing.) Best suited to processes which produce sequential output of large files (non-sequential writes may produced fragmented file manifests). Supports regular files and directories, does not support special files such as symlinks, hard links, named pipes, named sockets, or device nodes.|

arv:PartitionRequirement

Select preferred compute partitions on which to run jobs.

Field	Type	Description
partition	string or array of strings

arv:APIRequirement

Indicates that process wants to access to the Arvados API. Will be granted network access and have ARVADOS_API_HOST and ARVADOS_API_TOKEN set in the environment. Tools which rely on the Arvados API being present should put arv:APIRequirement in the requirements section of the tool (rather than hints) to indicate that that it is not portable to non-Arvados CWL runners.

Use arv:APIRequirement in hints to enable general (non-Arvados-specific) network access for a tool.

cwltool:LoadListingRequirement

In CWL v1.0 documents, the default behavior for Directory objects is to recursively expand the listing for access by parameter references an expressions. For directory trees containing many files, this can be expensive in both time and memory usage. Use cwltool:LoadListingRequirement to change the behavior for expansion of directory listings in the workflow runner.

Field	Type	Description
loadListing	string	One of `no_listing`, `shallow_listing`, or `deep_listing`

no_listing: Do not expand directory listing at all. The listing field on the Directory object will be undefined.

shallow_listing: Only expand the first level of directory listing. The listing field on the toplevel Directory object will contain the directory contents, however listing will not be defined on subdirectories.

deep_listing: Recursively expand all levels of directory listing. The listing field will be provided on the toplevel object and all subdirectories.

arv:IntermediateOutput

Specify desired handling of intermediate output collections.

Field	Type	Description
outputTTL	int	If the value is greater than zero, consider intermediate output collections to be temporary and should be automatically trashed. Temporary collections will be trashed `outputTTL` seconds after creation. A value of zero means intermediate output should be retained indefinitely (this is the default behavior). Note: arvados-cwl-runner currently does not take workflow dependencies into account when setting the TTL on an intermediate output collection. If the TTL is too short, it is possible for a collection to be trashed before downstream steps that consume it are started. The recommended minimum value for TTL is the expected duration of the entire the workflow.

arv:ReuseRequirement

Enable/disable work reuse for current process. Default true (work reuse enabled).

Field	Type	Description
enableReuse	boolean	Enable/disable work reuse for current process. Default true (work reuse enabled).

cwltool:Secrets

Indicate that one or more input parameters are “secret”. Must be applied at the top level Workflow. Secret parameters are not stored in keep, are hidden from logs and API responses, and are wiped from the database after the workflow completes.

Field	Type	Description
secrets	array	Input parameters which are considered “secret”. Must be strings.

cwltool:TimeLimit

Set an upper limit on the execution time of a CommandLineTool or ExpressionTool. A tool execution which exceeds the time limit may be preemptively terminated and considered failed. May also be used by batch systems to make scheduling decisions.

Field	Type	Description
timelimit	int	Execution time limit in seconds. If set to zero, no limit is enforced.

arv:WorkflowRunnerResources

Specify resource requirements for the workflow runner process (arvados-cwl-runner) that manages a workflow run. Must be applied to the top level workflow. Will also be set implicitly when using --submit-runner-ram on the command line along with --create-workflow or --update-workflow. Use this to adjust the runner’s allocation if the workflow runner is getting “out of memory” exceptions or being killed by the out-of-memory (OOM) killer.

Field	Type	Description
ramMin	int	RAM, in mebibytes, to reserve for the arvados-cwl-runner process. Default 1 GiB
coresMin	int	Number of cores to reserve to the arvados-cwl-runner process. Default 1 core.

Previous: Best Practices for writing CWL Next: Customizing Crunch environment using Docker