Package arvados :: Module collection :: Class CollectionWriter
[hide private]
[frames] | no frames]

Class CollectionWriter

source code

builtins.object --+    
     CollectionBase --+

Deprecated, use Collection instead.

Instance Methods [hide private]
__init__(self, api_client=None, num_retries=0, replication=None)
Instantiate a CollectionWriter.
source code
__exit__(self, exc_type, exc_value, traceback) source code
do_queued_work(self) source code
_work_file(self) source code
_work_dirents(self) source code
_work_trees(self) source code
_queue_file(self, source, filename=None) source code
_queue_dirents(self, stream_name, dirents) source code
_queue_tree(self, path, stream_name, max_manifest_depth) source code
write_file(self, source, filename=None) source code
write_directory_tree(self, path, stream_name='.', max_manifest_depth=-1) source code
write(self, newdata) source code
file-like object
open(streampath, filename=...)
Pass in the path of a file to write to the Collection, either as a single string or as two separate stream name and file name arguments.
source code
flush_data(self) source code
start_new_file(self, newfilename=None) source code
set_current_file_name(self, newfilename) source code
current_file_name(self) source code
finish_current_file(self) source code
start_new_stream(self, newstreamname='.') source code
set_current_stream_name(self, newstreamname) source code
current_stream_name(self) source code
finish_current_stream(self) source code
Store the manifest in Keep and return its locator.
source code
portable_data_hash(self) source code
manifest_text(self) source code
data_locators(self) source code
save_new(self, name=None) source code

Inherited from CollectionBase: __enter__, stripped_manifest

Inherited from CollectionBase (private): _my_keep

Method Details [hide private]

__init__(self, api_client=None, num_retries=0, replication=None)

source code 
Instantiate a CollectionWriter.

CollectionWriter lets you build a new Arvados Collection from scratch.
Write files to it.  The CollectionWriter will upload data to Keep as
appropriate, and provide you with the Collection manifest text when
you're finished.

* api_client: The API client to use to look up Collections.  If not
  provided, CollectionReader will build one from available Arvados
* num_retries: The default number of times to retry failed
  service requests.  Default 0.  You may change this value
  after instantiation, but note those changes may not
  propagate to related objects like the Keep client.
* replication: The number of copies of each block to store.
  If this argument is None or not supplied, replication is
  the server-provided default if available, otherwise 2.

__exit__(self, exc_type, exc_value, traceback)

source code 
Overrides: CollectionBase.__exit__

open(streampath, filename=...)

source code 

Pass in the path of a file to write to the Collection, either as a single string or as two separate stream name and file name arguments. This method returns a file-like object you can write to add it to the Collection.

You may only have one file object from the Collection open at a time, so be sure to close the object when you're done. Using the object in a with statement makes that easy:

 with'./doc/page1.txt') as outfile:
 with'./doc/page2.txt') as outfile:
Returns: file-like object


source code 

Store the manifest in Keep and return its locator.

This is useful for storing manifest fragments (task outputs) temporarily in Keep during a Crunch job.

In other cases you should make a collection instead, by sending manifest_text() to the API server's "create collection" endpoint.