In order to use the
arv command, make sure that you have a working environment.
arv create can be used to create Arvados objects from the command line. Arv create opens up the editor of your choice (set the EDITOR environment variable) and allows you to type or paste a json or yaml description. When saved the object will be created on the API server, if it passes validation.
arv create --helpOptions: --project-uuid, -p <s>: Project uuid in which to create the object --help, -h: Show this message
arv get can be used to get a textual representation of Arvados objects from the command line. The output can be limited to a subset of the object’s fields. This command can be used with only the knowledge of an object’s UUID.
arv get --helpUsage: arv [--format json|yaml] get [uuid] [fields...] Fetch the specified Arvados object, select the specified fields, and print a text representation.
arv edit can be used to edit Arvados objects from the command line. Arv edit opens up the editor of your choice (set the EDITOR environment variable) with the json or yaml description of the object. Saving the file will update the Arvados object on the API server, if it passes validation.
arv edit --helpArvados command line client Usage: arv edit [uuid] [fields...] Fetch the specified Arvados object, select the specified fields, open an interactive text editor on a text representation (json or yaml, use --format) and then update the object. Will use 'nano' by default, customize with the EDITOR or VISUAL environment variable.
arv copy can be used to copy a pipeline instance, template or collection from one Arvados instance to another. It takes care of copying the object and all its dependencies.
arv copy --helpusage: arv_copy.py [-h] [-v] [--progress] [--no-progress] [-f] --src SOURCE_ARVADOS --dst DESTINATION_ARVADOS [--recursive] [--no-recursive] [--dst-git-repo DST_GIT_REPO] [--project-uuid PROJECT_UUID] [--retries RETRIES] object_uuid Copy a pipeline instance, template or collection from one Arvados instance to another. positional arguments: object_uuid The UUID of the object to be copied. optional arguments: -h, --help show this help message and exit -v, --verbose Verbose output. --progress Report progress on copying collections. (default) --no-progress Do not report progress on copying collections. -f, --force Perform copy even if the object appears to exist at the remote destination. --src SOURCE_ARVADOS The name of the source Arvados instance (required) - points at an Arvados config file. May be either a pathname to a config file, or (for example) "foo" as shorthand for $HOME/.config/arvados/foo.conf. --dst DESTINATION_ARVADOS The name of the destination Arvados instance (required) - points at an Arvados config file. May be either a pathname to a config file, or (for example) "foo" as shorthand for $HOME/.config/arvados/foo.conf. --recursive Recursively copy any dependencies for this object. (default) --no-recursive Do not copy any dependencies. NOTE: if this option is given, the copied object will need to be updated manually in order to be functional. --dst-git-repo DST_GIT_REPO The name of the destination git repository. Required when copying a pipeline recursively. --project-uuid PROJECT_UUID The UUID of the project at the destination to which the pipeline should be copied. --retries RETRIES Maximum number of times to retry server requests that encounter temporary failures (e.g., server down). Default 3.
arv tag is used to tag Arvados objects.
arv tag --helpUsage: arv tag add tag1 [tag2 ...] --object object_uuid1 [object_uuid2...] arv tag remove tag1 [tag2 ...] --object object_uuid1 [object_uuid2...] arv tag remove --all --dry-run, -n: Don't actually do anything --verbose, -v: Print some things on stderr --uuid, -u: Return the UUIDs of the objects in the response, one per line (default) --json, -j: Return the entire response received from the API server, as a JSON object --human, -h: Return the response received from the API server, as a JSON object with whitespace added for human consumption --pretty, -p: Synonym of --human --yaml, -y: Return the response received from the API server, in YAML format --help, -e: Show this message
arv ws provides access to the websockets event stream.
arv ws --helpusage: arv-ws [-h] [-u UUID] [-f FILTERS] [--poll-interval POLL_INTERVAL | --no-poll] [-p PIPELINE | -j JOB] optional arguments: -h, --help show this help message and exit -u UUID, --uuid UUID Filter events on object_uuid -f FILTERS, --filters FILTERS Arvados query filter to apply to log events (JSON encoded) --poll-interval POLL_INTERVAL If websockets is not available, specify the polling interval, default is every 15 seconds --no-poll Do not poll if websockets are not available, just fail -p PIPELINE, --pipeline PIPELINE Supply pipeline uuid, print log output from pipeline and its jobs -j JOB, --job JOB Supply job uuid, print log output from jobs
arv keep provides access to the Keep storage service.
arv keep --helpUsage: arv keep [method] [--parameters] Use 'arv keep [method] --help' to get more information about specific methods. Available methods: ls, get, put, docker
arv keep ls --helpusage: arv-ls [-h] [--retries RETRIES] [-s] locator List contents of a manifest positional arguments: locator Collection UUID or locator optional arguments: -h, --help show this help message and exit --retries RETRIES Maximum number of times to retry server requests that encounter temporary failures (e.g., server down). Default 3. -s List file sizes, in KiB.
arv keep get --helpusage: arv-get [-h] [--retries RETRIES] [--progress | --no-progress | --batch-progress] [--hash HASH | --md5sum] [-n] [-r] [-f | --skip-existing] locator [destination] Copy data from Keep to a local file or pipe. positional arguments: locator Collection locator, optionally with a file path or prefix. destination Local file or directory where the data is to be written. Default: /dev/stdout. optional arguments: -h, --help show this help message and exit --retries RETRIES Maximum number of times to retry server requests that encounter temporary failures (e.g., server down). Default 3. --progress Display human-readable progress on stderr (bytes and, if possible, percentage of total data size). This is the default behavior when it is not expected to interfere with the output: specifically, stderr is a tty _and_ either stdout is not a tty, or output is being written to named files rather than stdout. --no-progress Do not display human-readable progress on stderr. --batch-progress Display machine-readable progress on stderr (bytes and, if known, total data size). --hash HASH Display the hash of each file as it is read from Keep, using the given hash algorithm. Supported algorithms include md5, sha1, sha224, sha256, sha384, and sha512. --md5sum Display the MD5 hash of each file as it is read from Keep. -n Do not write any data -- just read from Keep, and report md5sums if requested. -r Retrieve all files in the specified collection/prefix. This is the default behavior if the "locator" argument ends with a forward slash. -f Overwrite existing files while writing. The default behavior is to refuse to write *anything* if any of the output files already exist. As a special case, -f is not needed to write to /dev/stdout. --skip-existing Skip files that already exist. The default behavior is to refuse to write *anything* if any files exist that would have to be overwritten. This option causes even devices, sockets, and fifos to be skipped.
arv keep put --helpusage: arv-put [-h] [--max-manifest-depth N | --normalize] [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw] [--use-filename FILENAME] [--filename FILENAME] [--portable-data-hash] [--replication N] [--project-uuid UUID] [--name NAME] [--progress | --no-progress | --batch-progress] [--resume | --no-resume] [--retries RETRIES] [path [path ...]] Copy data from the local filesystem to Keep. positional arguments: path Local file or directory. Default: read from standard input. optional arguments: -h, --help show this help message and exit --max-manifest-depth N Maximum depth of directory tree to represent in the manifest structure. A directory structure deeper than this will be represented as a single stream in the manifest. If N=0, the manifest will contain a single stream. Default: -1 (unlimited), i.e., exactly one manifest stream per filesystem directory that contains files. --normalize Normalize the manifest by re-ordering files and streams after writing data. --as-stream Synonym for --stream. --stream Store the file content and display the resulting manifest on stdout. Do not write the manifest to Keep or save a Collection object in Arvados. --as-manifest Synonym for --manifest. --in-manifest Synonym for --manifest. --manifest Store the file data and resulting manifest in Keep, save a Collection object in Arvados, and display the manifest locator (Collection uuid) on stdout. This is the default behavior. --as-raw Synonym for --raw. --raw Store the file content and display the data block locators on stdout, separated by commas, with a trailing newline. Do not store a manifest. --use-filename FILENAME Synonym for --filename. --filename FILENAME Use the given filename in the manifest, instead of the name of the local file. This is useful when "-" or "/dev/stdin" is given as an input file. It can be used only if there is exactly one path given and it is not a directory. Implies --manifest. --portable-data-hash Print the portable data hash instead of the Arvados UUID for the collection created by the upload. --replication N Set the replication level for the new collection: how many different physical storage devices (e.g., disks) should have a copy of each data block. Default is to use the server-provided default (if any) or 2. --project-uuid UUID Store the collection in the specified project, instead of your Home project. --name NAME Save the collection with the specified name. --progress Display human-readable progress on stderr (bytes and, if possible, percentage of total data size). This is the default behavior when stderr is a tty. --no-progress Do not display human-readable progress on stderr, even if stderr is a tty. --batch-progress Display machine-readable progress on stderr (bytes and, if known, total data size). --resume Continue interrupted uploads from cached state (default). --no-resume Do not continue interrupted uploads from cached state. --retries RETRIES Maximum number of times to retry server requests that encounter temporary failures (e.g., server down). Default 3.
arv pipeline run can be used to start a pipeline run from the command line.
The User Guide has a page with a bit more information on using arv pipeline run.
arv pipeline run --helpOptions: --dry-run, -n: Do not start any new jobs or wait for existing jobs to finish. Just find out whether jobs are finished, queued, or running for each component. --status-text <s>: Store plain text status in given file. (Default: /dev/stdout) --status-json <s>: Store json-formatted pipeline in given file. (Default: /dev/null) --no-wait: Do not wait for jobs to finish. Just look up status, submit new jobs if needed, and exit. --no-reuse: Do not reuse existing jobs to satisfy pipeline components. Submit a new job for every component. --debug, -d: Print extra debugging information on stderr. --debug-level <i>: Set debug verbosity level. --template <s>: UUID of pipeline template, or path to local pipeline template file. --instance <s>: UUID of pipeline instance. --submit: Submit the pipeline instance to the server, and exit. Let the Crunch dispatch service satisfy the components by finding/running jobs. --run-pipeline-here: Manage the pipeline instance in-process. Submit jobs to Crunch as needed. Do not exit until the pipeline finishes (or fails). --run-jobs-here: Run jobs in the local terminal session instead of submitting them to Crunch. Implies --run-pipeline-here. Note: this results in a significantly different job execution environment, and some Crunch features are not supported. It can be necessary to modify a pipeline in order to make it run this way. --run-here: Synonym for --run-jobs-here. --description <s>: Description for the pipeline instance. --version, -v: Print version and exit --help, -h: Show this message
arv-run command creates Arvados pipelines at the command line that fan out to multiple concurrent tasks across Arvados compute nodes.
The User Guide has a page on using arv-run.
arv run --helpusage: arv-run [-h] [--retries RETRIES] [--dry-run] [--local] [--docker-image DOCKER_IMAGE] [--ignore-rcode] [--no-reuse] [--no-wait] [--project-uuid PROJECT_UUID] [--git-dir GIT_DIR] [--repository REPOSITORY] [--script-version SCRIPT_VERSION] ... positional arguments: args optional arguments: -h, --help show this help message and exit --retries RETRIES Maximum number of times to retry server requests that encounter temporary failures (e.g., server down). Default 3. --dry-run Print out the pipeline that would be submitted and exit --local Run locally using arv-run-pipeline-instance --docker-image DOCKER_IMAGE Docker image to use, otherwise use instance default. --ignore-rcode Commands that return non-zero return codes should not be considered failed. --no-reuse Do not reuse past jobs. --no-wait Do not wait and display logs after submitting command, just exit. --project-uuid PROJECT_UUID Parent project of the pipeline --git-dir GIT_DIR Git repository passed to arv-crunch-job when using --local --repository REPOSITORY repository field of component, default 'arvados' --script-version SCRIPT_VERSION script_version field of component, default 'master'
The content of this documentation is licensed under the
Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.