Package arvados :: Module arvfile :: Class _BlockManager
[hide private]
[frames] | no frames]

Class _BlockManager

source code

builtins.object --+
                  |
                 _BlockManager

BlockManager handles buffer blocks.

Also handles background block uploads, and background block prefetch for a Collection of ArvadosFiles.

Instance Methods [hide private]
 
__init__(self, keep, copies=None, put_threads=None, num_retries=None)
keep: KeepClient object to use
source code
 
alloc_bufferblock(self, blockid=None, starting_capacity=2**14, owner=None)
Allocate a new, empty bufferblock in WRITABLE state and return it.
source code
 
_alloc_bufferblock(self, blockid=None, starting_capacity=2**14, owner=None) source code
 
dup_block(self, block, owner)
Create a new bufferblock initialized with the content of an existing bufferblock.
source code
 
is_bufferblock(self, locator) source code
 
_commit_bufferblock_worker(self)
Background uploader thread.
source code
 
start_put_threads(self) source code
 
_block_prefetch_worker(self)
The background downloader thread.
source code
 
start_get_threads(self) source code
 
stop_threads(self)
Shut down and wait for background upload and download threads to finish.
source code
 
__enter__(self) source code
 
__exit__(self, exc_type, exc_value, traceback) source code
 
repack_small_blocks(self, force=False, sync=False, closed_file_size=0)
Packs small blocks together before uploading
source code
 
commit_bufferblock(self, block, sync)
Initiate a background upload of a bufferblock.
source code
 
get_bufferblock(self, locator) source code
 
get_padding_block(self)
Get a bufferblock 64 MB in size consisting of all zeros, used as padding when using truncate() to extend the size of a file.
source code
 
delete_bufferblock(self, locator) source code
 
_delete_bufferblock(self, locator) source code
 
get_block_contents(self, locator, num_retries, cache_only=False)
Fetch a block.
source code
 
commit_all(self)
Commit all outstanding buffer blocks.
source code
 
block_prefetch(self, locator)
Initiate a background download of a block.
source code
Class Variables [hide private]
  DEFAULT_PUT_THREADS = 2
  DEFAULT_GET_THREADS = 2
Method Details [hide private]

alloc_bufferblock(self, blockid=None, starting_capacity=2**14, owner=None)

source code 
Allocate a new, empty bufferblock in WRITABLE state and return it.

:blockid:
  optional block identifier, otherwise one will be automatically assigned

:starting_capacity:
  optional capacity, otherwise will use default capacity

:owner:
  ArvadosFile that owns this block

Decorators:
  • @synchronized

dup_block(self, block, owner)

source code 
Create a new bufferblock initialized with the content of an existing bufferblock.

:block:
  the buffer block to copy.

:owner:
  ArvadosFile that owns the new block

Decorators:
  • @synchronized

is_bufferblock(self, locator)

source code 
Decorators:
  • @synchronized

start_get_threads(self)

source code 
Decorators:
  • @synchronized

stop_threads(self)

source code 

Shut down and wait for background upload and download threads to finish.

Decorators:
  • @synchronized

repack_small_blocks(self, force=False, sync=False, closed_file_size=0)

source code 

Packs small blocks together before uploading

Decorators:
  • @synchronized

commit_bufferblock(self, block, sync)

source code 
Initiate a background upload of a bufferblock.

:block:
  The block object to upload

:sync:
  If `sync` is True, upload the block synchronously.
  If `sync` is False, upload the block asynchronously.  This will
  return immediately unless the upload queue is at capacity, in
  which case it will wait on an upload queue slot.

get_bufferblock(self, locator)

source code 
Decorators:
  • @synchronized

get_padding_block(self)

source code 

Get a bufferblock 64 MB in size consisting of all zeros, used as padding when using truncate() to extend the size of a file.

For reference (and possible future optimization), the md5sum of the padding block is: 7f614da9329cd3aebf59b91aadc30bf0+67108864

Decorators:
  • @synchronized

delete_bufferblock(self, locator)

source code 
Decorators:
  • @synchronized

get_block_contents(self, locator, num_retries, cache_only=False)

source code 

Fetch a block.

First checks to see if the locator is a BufferBlock and return that, if not, passes the request through to KeepClient.get().

commit_all(self)

source code 

Commit all outstanding buffer blocks.

This is a synchronous call, and will not return until all buffer blocks are uploaded. Raises KeepWriteError() if any blocks failed to upload.

block_prefetch(self, locator)

source code 

Initiate a background download of a block.

This assumes that the underlying KeepClient implements a block cache, so repeated requests for the same block will not result in repeated downloads (unless the block is evicted from the cache.) This method does not block.