arvados.collection.CollectionWriter

Class CollectionWriter

builtins.object --+ | CollectionBase --+ | CollectionWriter

Deprecated, use Collection instead.

Instance Methods

[hide private]

__init__(self, api_client=None, num_retries=0, replication=None)
Instantiate a CollectionWriter.

source code

__exit__(self, exc_type, exc_value, traceback)

source code

do_queued_work(self)

source code

_work_file(self)

source code

_work_dirents(self)

source code

_work_trees(self)

source code

_queue_file(self, source, filename=None)

source code

_queue_dirents(self, stream_name, dirents)

source code

_queue_tree(self, path, stream_name, max_manifest_depth)

source code

write_file(self, source, filename=None)

source code

write_directory_tree(self, path, stream_name='.', max_manifest_depth=-1)

source code

write(self, newdata)

source code

file-like object

open(streampath, filename=...)
Pass in the path of a file to write to the Collection, either as a single string or as two separate stream name and file name arguments.

source code

flush_data(self)

source code

start_new_file(self, newfilename=None)

source code

set_current_file_name(self, newfilename)

source code

current_file_name(self)

source code

finish_current_file(self)

source code

start_new_stream(self, newstreamname='.')

source code

set_current_stream_name(self, newstreamname)

source code

current_stream_name(self)

source code

finish_current_stream(self)

source code

finish(self)
Store the manifest in Keep and return its locator.

source code

portable_data_hash(self)

source code

manifest_text(self)

source code

data_locators(self)

source code

save_new(self, name=None)

source code

Inherited from CollectionBase: __enter__, stripped_manifest

Inherited from CollectionBase (private): _my_keep

init(self, api_client=None, num_retries=0, replication=None)
(Constructor)

source code

Instantiate a CollectionWriter.

CollectionWriter lets you build a new Arvados Collection from scratch.
Write files to it.  The CollectionWriter will upload data to Keep as
appropriate, and provide you with the Collection manifest text when
you're finished.

Arguments:
* api_client: The API client to use to look up Collections.  If not
  provided, CollectionReader will build one from available Arvados
  configuration.
* num_retries: The default number of times to retry failed
  service requests.  Default 0.  You may change this value
  after instantiation, but note those changes may not
  propagate to related objects like the Keep client.
* replication: The number of copies of each block to store.
  If this argument is None or not supplied, replication is
  the server-provided default if available, otherwise 2.

exit(self, exc_type, exc_value, traceback)

source code

Overrides: CollectionBase.__exit__

open(streampath, filename=...)

source code

Pass in the path of a file to write to the Collection, either as a single string or as two separate stream name and file name arguments. This method returns a file-like object you can write to add it to the Collection.

You may only have one file object from the Collection open at a time, so be sure to close the object when you're done. Using the object in a with statement makes that easy:

 with cwriter.open('./doc/page1.txt') as outfile:
     outfile.write(page1_data)
 with cwriter.open('./doc/page2.txt') as outfile:
     outfile.write(page2_data)

Returns: file-like object

finish(self)

source code

Store the manifest in Keep and return its locator.

This is useful for storing manifest fragments (task outputs) temporarily in Keep during a Crunch job.

In other cases you should make a collection instead, by sending manifest_text() to the API server's "create collection" endpoint.

Class CollectionWriter

__init__(self, api_client=None, num_retries=0, replication=None) (Constructor)

__exit__(self, exc_type, exc_value, traceback)

open(streampath, filename=...)

finish(self)

init(self, api_client=None, num_retries=0, replication=None)
(Constructor)

exit(self, exc_type, exc_value, traceback)