Package arvados :: Module arvfile :: Class ArvadosFile

Class ArvadosFile

builtins.object --+
                  |
                 ArvadosFile

Represent a file in a Collection.

ArvadosFile manages the underlying representation of a file in Keep as a sequence of segments spanning a set of blocks, and implements random read/write access.

This object may be accessed from multiple threads.

Instance Methods

[hide private]

__init__(self, parent, name, stream=[], segments=[])
ArvadosFile constructor.

source code

writable(self)

source code

permission_expired(self, as_of_dt=None)
Returns True if any of the segment's locators is expired

source code

has_remote_blocks(self)
Returns True if any of the segment's locators has a +R signature

source code

_copy_remote_blocks(self, remote_blocks={})
Ask Keep to copy remote blocks and point to their local copies.

source code

segments(self)

source code

clone(self, new_parent, new_name)
Make a copy of this file.

source code

replace_contents(self, other)
Replace segments of this file with segments from another `ArvadosFile` object.

source code

__eq__(self, other)

source code

__ne__(self, other)

source code

set_segments(self, segs)

source code

set_committed(self, value=True)
Set committed flag.

source code

committed(self)
Get whether this is committed or not.

source code

add_writer(self, writer)
Add an ArvadosFileWriter reference to the list of writers

source code

remove_writer(self, writer, flush)
Called from ArvadosFileWriter.close().

source code

closed(self)
Get whether this is closed or not.

source code

truncate(self, size)
Shrink or expand the size of the file.

source code

readfrom(self, offset, size, num_retries, exact=False)
Read up to `size` bytes from the file starting at `offset`.

source code

writeto(self, offset, data, num_retries)
Write `data` to the file starting at `offset`.

source code

flush(self, sync=True, num_retries=0)
Flush the current bufferblock to Keep.

source code

add_segment(self, blocks, pos, size)
Add a segment to the end of the file.

source code

_add_segment(self, blocks, pos, size)
Internal implementation of add_segment.

source code

size(self)
Get the file size.

source code

manifest_text(self, stream_name=".", portable_locators=False, normalize=False, only_committed=False)

source code

_reparent(self, newparent, newname)

source code

Method Details

[hide private]

init(self, parent, name, stream=[], segments=[])
(Constructor)

source code


ArvadosFile constructor.

:stream:
  a list of Range objects representing a block stream

:segments:
  a list of Range objects representing segments

permission_expired(self, as_of_dt=None)

source code

Returns True if any of the segment's locators is expired

Decorators:

@synchronized

has_remote_blocks(self)

source code

Returns True if any of the segment's locators has a +R signature

Decorators:

@synchronized

_copy_remote_blocks(self, remote_blocks={})

source code

Ask Keep to copy remote blocks and point to their local copies.

This is called from the parent Collection.

:remote_blocks:
    Shared cache of remote to local block mappings. This is used to avoid
    doing extra work when blocks are shared by more than one file in
    different subdirectories.

Decorators:

@synchronized

segments(self)

source code

Decorators:

@synchronized

clone(self, new_parent, new_name)

source code

Make a copy of this file.

Decorators:

@synchronized

replace_contents(self, other)

source code

Replace segments of this file with segments from another `ArvadosFile` object.

Decorators:

@must_be_writable
@synchronized

set_segments(self, segs)

source code

Decorators:

@synchronized

set_committed(self, value=True)

source code

Set committed flag.

If value is True, set committed to be True.

If value is False, set committed to be False for this and all parents.

Decorators:

@synchronized

committed(self)

source code

Get whether this is committed or not.

Decorators:

@synchronized

add_writer(self, writer)

source code

Add an ArvadosFileWriter reference to the list of writers

Decorators:

@synchronized

remove_writer(self, writer, flush)

source code

Called from ArvadosFileWriter.close(). Remove a writer reference from the list and do some block maintenance tasks.

Decorators:

@synchronized

closed(self)

source code

Get whether this is closed or not. When the writers list is empty, the file is supposed to be closed.

truncate(self, size)

source code

Shrink or expand the size of the file.

If `size` is less than the size of the file, the file contents after `size` will be discarded. If `size` is greater than the current size of the file, it will be filled with zero bytes.

Decorators:

@must_be_writable
@synchronized

readfrom(self, offset, size, num_retries, exact=False)

source code

Read up to `size` bytes from the file starting at `offset`.

:exact:
 If False (default), return less data than requested if the read
 crosses a block boundary and the next block isn't cached.  If True,
 only return less data than requested when hitting EOF.

writeto(self, offset, data, num_retries)

source code

Write `data` to the file starting at `offset`.

This will update existing bytes and/or extend the size of the file as necessary.

Decorators:

@must_be_writable
@synchronized

flush(self, sync=True, num_retries=0)

source code

Flush the current bufferblock to Keep.

:sync:
  If True, commit block synchronously, wait until buffer block has been written.
  If False, commit block asynchronously, return immediately after putting block into
  the keep put queue.

Decorators:

@synchronized

add_segment(self, blocks, pos, size)

source code

Add a segment to the end of the file.

`pos` and `offset` reference a section of the stream described by `blocks` (a list of Range objects)

Decorators:

@must_be_writable
@synchronized

size(self)

source code

Get the file size.

Decorators:

@synchronized

manifest_text(self, stream_name=".", portable_locators=False, normalize=False, only_committed=False)

source code

Decorators:

@synchronized

_reparent(self, newparent, newname)

source code

Decorators:

@must_be_writable
@synchronized

Class ArvadosFile

__init__(self, parent, name, stream=[], segments=[]) (Constructor)

permission_expired(self, as_of_dt=None)

has_remote_blocks(self)

_copy_remote_blocks(self, remote_blocks={})

segments(self)

clone(self, new_parent, new_name)

replace_contents(self, other)

set_segments(self, segs)

set_committed(self, value=True)

committed(self)

add_writer(self, writer)

remove_writer(self, writer, flush)

closed(self)

truncate(self, size)

readfrom(self, offset, size, num_retries, exact=False)

writeto(self, offset, data, num_retries)

flush(self, sync=True, num_retries=0)

add_segment(self, blocks, pos, size)

size(self)

manifest_text(self, stream_name=".", portable_locators=False, normalize=False, only_committed=False)

_reparent(self, newparent, newname)

init(self, parent, name, stream=[], segments=[])
(Constructor)