Package arvados :: Module keep :: Class KeepClient
[hide private]
[frames] | no frames]

Class KeepClient

source code

builtins.object --+
                  |
                 KeepClient

Nested Classes [hide private]
  KeepService
Make requests to a single Keep service, and track results.
  KeepWriterQueue
  KeepWriterThreadPool
  KeepWriterThread
Instance Methods [hide private]
 
__init__(self, api_client=None, proxy=None, timeout=DEFAULT_TIMEOUT, proxy_timeout=DEFAULT_PROXY_TIMEOUT, api_token=None, local_store=None, block_cache=None, num_retries=0, session=None)
Initialize a new KeepClient.
source code
 
current_timeout(self, attempt_number)
Return the appropriate timeout to use for this client.
source code
 
_any_nondisk_services(self, service_list) source code
 
build_services_list(self, force_rebuild=False) source code
 
_service_weight(self, data_hash, service_uuid)
Compute the weight of a Keep service endpoint for a data block with a known hash.
source code
 
weighted_service_roots(self, locator, force_rebuild=False, need_writable=False)
Return an array of Keep service endpoints, in the order in which they should be probed when reading or writing data with the given hash+hints.
source code
 
map_new_services(self, roots_map, locator, force_rebuild, need_writable, headers) source code
 
get_from_cache(self, loc)
Fetch a block only if is in the cache, otherwise return None.
source code
 
head(self, loc_s, **kwargs) source code
 
get(self, loc_s, **kwargs) source code
 
_get_or_head(self, loc_s, method="GET", num_retries=None, request_id=None)
Get data from Keep.
source code
 
put(self, data, copies=2, num_retries=None, request_id=None)
Save data in Keep.
source code
 
local_store_put(self, data, copies=1, num_retries=None)
A stub for put().
source code
 
local_store_get(self, loc_s, num_retries=None)
Companion to local_store_put().
source code
 
is_cached(self, locator) source code
Static Methods [hide private]
 
_check_loop_result(result) source code
Class Variables [hide private]
  DEFAULT_TIMEOUT = 2, 256, 32768
  DEFAULT_PROXY_TIMEOUT = 20, 256, 32768
Method Details [hide private]

__init__(self, api_client=None, proxy=None, timeout=DEFAULT_TIMEOUT, proxy_timeout=DEFAULT_PROXY_TIMEOUT, api_token=None, local_store=None, block_cache=None, num_retries=0, session=None)
(Constructor)

source code 
Initialize a new KeepClient.

Arguments:
:api_client:
  The API client to use to find Keep services.  If not
  provided, KeepClient will build one from available Arvados
  configuration.

:proxy:
  If specified, this KeepClient will send requests to this Keep
  proxy.  Otherwise, KeepClient will fall back to the setting of the
  ARVADOS_KEEP_SERVICES or ARVADOS_KEEP_PROXY configuration settings.
  If you want to KeepClient does not use a proxy, pass in an empty
  string.

:timeout:
  The initial timeout (in seconds) for HTTP requests to Keep
  non-proxy servers.  A tuple of three floats is interpreted as
  (connection_timeout, read_timeout, minimum_bandwidth). A connection
  will be aborted if the average traffic rate falls below
  minimum_bandwidth bytes per second over an interval of read_timeout
  seconds. Because timeouts are often a result of transient server
  load, the actual connection timeout will be increased by a factor
  of two on each retry.
  Default: (2, 256, 32768).

:proxy_timeout:
  The initial timeout (in seconds) for HTTP requests to
  Keep proxies. A tuple of three floats is interpreted as
  (connection_timeout, read_timeout, minimum_bandwidth). The behavior
  described above for adjusting connection timeouts on retry also
  applies.
  Default: (20, 256, 32768).

:api_token:
  If you're not using an API client, but only talking
  directly to a Keep proxy, this parameter specifies an API token
  to authenticate Keep requests.  It is an error to specify both
  api_client and api_token.  If you specify neither, KeepClient
  will use one available from the Arvados configuration.

:local_store:
  If specified, this KeepClient will bypass Keep
  services, and save data to the named directory.  If unspecified,
  KeepClient will fall back to the setting of the $KEEP_LOCAL_STORE
  environment variable.  If you want to ensure KeepClient does not
  use local storage, pass in an empty string.  This is primarily
  intended to mock a server for testing.

:num_retries:
  The default number of times to retry failed requests.
  This will be used as the default num_retries value when get() and
  put() are called.  Default 0.

current_timeout(self, attempt_number)

source code 

Return the appropriate timeout to use for this client.

The proxy timeout setting if the backend service is currently a proxy, the regular timeout setting otherwise. The `attempt_number` indicates how many times the operation has been tried already (starting from 0 for the first try), and scales the connection timeout portion of the return value accordingly.

_service_weight(self, data_hash, service_uuid)

source code 

Compute the weight of a Keep service endpoint for a data block with a known hash.

The weight is md5(h + u) where u is the last 15 characters of the service endpoint's UUID.

head(self, loc_s, **kwargs)

source code 
Decorators:
  • @retry.retry_method

get(self, loc_s, **kwargs)

source code 
Decorators:
  • @retry.retry_method

_get_or_head(self, loc_s, method="GET", num_retries=None, request_id=None)

source code 
Get data from Keep.

This method fetches one or more blocks of data from Keep.  It
sends a request each Keep service registered with the API
server (or the proxy provided when this client was
instantiated), then each service named in location hints, in
sequence.  As soon as one service provides the data, it's
returned.

Arguments:
* loc_s: A string of one or more comma-separated locators to fetch.
  This method returns the concatenation of these blocks.
* num_retries: The number of times to retry GET requests to
  *each* Keep server if it returns temporary failures, with
  exponential backoff.  Note that, in each loop, the method may try
  to fetch data from every available Keep service, along with any
  that are named in location hints in the locator.  The default value
  is set when the KeepClient is initialized.

put(self, data, copies=2, num_retries=None, request_id=None)

source code 
Save data in Keep.

This method will get a list of Keep services from the API server, and
send the data to each one simultaneously in a new thread.  Once the
uploads are finished, if enough copies are saved, this method returns
the most recent HTTP response body.  If requests fail to upload
enough copies, this method raises KeepWriteError.

Arguments:
* data: The string of data to upload.
* copies: The number of copies that the user requires be saved.
  Default 2.
* num_retries: The number of times to retry PUT requests to
  *each* Keep server if it returns temporary failures, with
  exponential backoff.  The default value is set when the
  KeepClient is initialized.

Decorators:
  • @retry.retry_method

local_store_put(self, data, copies=1, num_retries=None)

source code 

A stub for put().

This method is used in place of the real put() method when using local storage (see constructor's local_store argument).

copies and num_retries arguments are ignored: they are here only for the sake of offering the same call signature as put().

Data stored this way can be retrieved via local_store_get().