Package arvados :: Package commands :: Module put :: Class ArvPutUploadJob
[hide private]
[frames] | no frames]

Class ArvPutUploadJob

source code

builtins.object --+
                  |
                 ArvPutUploadJob

Instance Methods [hide private]
 
__init__(self, paths, resume=True, use_cache=True, reporter=None, name=None, owner_uuid=None, api_client=None, ensure_unique_name=False, num_retries=None, put_threads=None, replication_desired=None, filename=None, update_time=60.0, update_collection=None, storage_classes=None, logger=logging.getLogger('arvados.arv_put'), dry_run=False, follow_links=True, exclude_paths=[], exclude_names=None, trash_at=None) source code
 
_build_upload_list(self)
Scan the requested paths to count file sizes, excluding requested files and dirs and building the upload file list.
source code
 
start(self, save_collection)
Start supporting thread & file uploading
source code
 
_collection_trash_at(self)
Returns the trash date that the collection should use at save time.
source code
 
save_collection(self) source code
 
destroy_cache(self) source code
 
_collection_size(self, collection)
Recursively get the total size of the collection
source code
 
_update_task(self)
Periodically called support task.
source code
 
_update(self, final=False)
Update cached manifest text and report progress.
source code
 
report_progress(self) source code
 
_write_stdin(self, filename) source code
 
_check_file(self, source, filename)
Check if this file needs to be uploaded
source code
 
_upload_files(self) source code
 
_write(self, source_fd, output) source code
 
_my_collection(self) source code
 
_get_cache_filepath(self) source code
 
_setup_state(self, update_collection)
Create a new cache file or load a previously existing one.
source code
 
_cached_manifest_valid(self)
Validate the oldest non-expired block signature to check if cached manifest is usable: checking if the cached manifest was not created with a different arvados account.
source code
 
collection_file_paths(self, col, path_prefix='.')
Return a list of file paths by recursively go through the entire collection `col`
source code
 
_lock_file(self, fileobj) source code
 
_save_state(self)
Atomically save current state into cache.
source code
 
collection_name(self) source code
 
collection_trash_at(self) source code
 
manifest_locator(self) source code
 
portable_data_hash(self) source code
 
manifest_text(self, stream_name=".", strip=False, normalize=False) source code
 
_datablocks_on_item(self, item)
Return a list of datablock locators, recursively navigating through subcollections
source code
 
data_locators(self) source code
Class Variables [hide private]
  CACHE_DIR = '.cache/arvados/arv-put'
  EMPTY_STATE = {'manifest': None, 'files': {}}
Method Details [hide private]

_collection_trash_at(self)

source code 

Returns the trash date that the collection should use at save time. Takes into account absolute/relative trash_at values requested by the user.

_update_task(self)

source code 

Periodically called support task. File uploading is asynchronous so we poll status from the collection.