Package arvados :: Module util

[frames] | no frames]

Module util

source code

Functions

[hide private]

clear_tmpdir(path=None)
Ensure the given directory (or TASK_TMPDIR if none given) exists and is empty.

source code

run_command(execargs, **kwargs)

source code

git_checkout(url, version, path)

source code

tar_extractor(path, decompress_flag)

source code

tarball_extract(tarball, path)
Retrieve a tarball from Keep and extract it to a local directory.

source code

zipball_extract(zipball, path)
Retrieve a zip archive from Keep and extract it to a local directory.

source code

collection_extract(collection, path, files=[], decompress=True)
Retrieve a collection from Keep and extract it to a local directory.

source code

mkdir_dash_p(path)

source code

stream_extract(stream, path, files=[], decompress=True)
Retrieve a stream from Keep and extract it to a local directory.

source code

listdir_recursive(dirname, base=None, max_depth=None)
Return a list of file and directory names found under dirname.

source code

boolean

is_hex(s, length=..., max_length=...)
Return True if s is a string of hexadecimal digits.

source code

list_all(fn, num_retries=0, **kwargs)

source code

ca_certs_path(fallback=httplib2.CA_CERTS)
Return the path of the best available CA certs source.

source code

new_request_id()

source code

Variables

[hide private]

HEX_RE = re.compile(r'^[0-9a-fA-F]+$')

keep_locator_pattern = re.compile(r'[0-9a-f]{32}\+\d+(\+\S+)*')

signed_locator_pattern = re.compile(r'[0-9a-f]{32}\+\d+(\+\S+)...

portable_data_hash_pattern = re.compile(r'[0-9a-f]{32}\+\d+')

uuid_pattern = re.compile(r'[a-z0-9]{5}-[a-z0-9]{5}-[a-z0-9]{1...

collection_uuid_pattern = re.compile(r'[a-z0-9]{5}-4zz18-[a-z0...

group_uuid_pattern = re.compile(r'[a-z0-9]{5}-j7d0g-[a-z0-9]{1...

user_uuid_pattern = re.compile(r'[a-z0-9]{5}-tpzed-[a-z0-9]{15}')

link_uuid_pattern = re.compile(r'[a-z0-9]{5}-o0j2j-[a-z0-9]{15}')

job_uuid_pattern = re.compile(r'[a-z0-9]{5}-8i9sb-[a-z0-9]{15}')

container_uuid_pattern = re.compile(r'[a-z0-9]{5}-dz642-[a-z0-...

manifest_pattern = re.compile(r'((\S+)( +[a-f0-9]{32}(\+\d+)(\...

Function Details

[hide private]

tarball_extract(tarball, path)

source code

Retrieve a tarball from Keep and extract it to a local directory. Return the absolute path where the tarball was extracted. If the top level of the tarball contained just one file or directory, return the absolute path of that single item.

tarball -- collection locator path -- where to extract the tarball: absolute, or relative to job tmp

zipball_extract(zipball, path)

source code

Retrieve a zip archive from Keep and extract it to a local directory. Return the absolute path where the archive was extracted. If the top level of the archive contained just one file or directory, return the absolute path of that single item.

zipball -- collection locator path -- where to extract the archive: absolute, or relative to job tmp

collection_extract(collection, path, files=[], decompress=True)

source code

Retrieve a collection from Keep and extract it to a local directory. Return the absolute path where the collection was extracted.

collection -- collection locator path -- where to extract: absolute, or relative to job tmp

stream_extract(stream, path, files=[], decompress=True)

source code

Retrieve a stream from Keep and extract it to a local directory. Return the absolute path where the stream was extracted.

stream -- StreamReader object path -- where to extract: absolute, or relative to job tmp

listdir_recursive(dirname, base=None, max_depth=None)

source code

Return a list of file and directory names found under dirname.

If base is not None, prepend "{base}/" to each returned name.

If max_depth is None, descend into directories and return only the names of files found in the directory tree.

If max_depth is a non-negative integer, stop descending into directories at the given depth, and at that point return directory names instead.

If max_depth==0 (and base is None) this is equivalent to sorted(os.listdir(dirname)).

is_hex(s, length=..., max_length=...)

source code

Return True if s is a string of hexadecimal digits. If one length argument is given, the string must contain exactly that number of digits. If two length arguments are given, the string must contain a number of digits between those two lengths, inclusive. Return False otherwise.

Returns: boolean

ca_certs_path(fallback=httplib2.CA_CERTS)

source code

Return the path of the best available CA certs source.

This function searches for various distribution sources of CA certificates, and returns the first it finds. If it doesn't find any, it returns the value of `fallback` (httplib2's CA certs by default).

Variables Details

[hide private]

signed_locator_pattern

Value:

re.compile(r'[0-9a-f]{32}\+\d+(\+\S+)*\+A\S+(\+\S+)*')

uuid_pattern

Value:

re.compile(r'[a-z0-9]{5}-[a-z0-9]{5}-[a-z0-9]{15}')

collection_uuid_pattern

Value:

re.compile(r'[a-z0-9]{5}-4zz18-[a-z0-9]{15}')

group_uuid_pattern

Value:

re.compile(r'[a-z0-9]{5}-j7d0g-[a-z0-9]{15}')

container_uuid_pattern

Value:

re.compile(r'[a-z0-9]{5}-dz642-[a-z0-9]{15}')

manifest_pattern

Value:

re.compile(r'((\S+)( +[a-f0-9]{32}(\+\d+)(\+\S+)*)+( +\d+:\d+:\S+)+$)+
', flags= re.MULTILINE)