For some use cases, you may want to limit the ability of users to upload or download data from outside the cluster. (By “outside” we mean from networks other than the cluster’s own private network). For example, this makes it possible to share restricted data sets with users so that they may run their own data analysis on the cluster, while preventing them from easily downloading the data set to their local workstation.
This feature exists in addition to the existing Arvados permission system. Users can only download from collections they have read access to, and can only upload to projects and collections they have write access to.
There are two services involved in accessing data from outside the cluster.
Permitting keepproxy makes it possible to use arv-put and arv-get. It works in terms of individual 64 MiB keep blocks. It prints a log line each time a user uploads or downloads an individual block. Those logs are usually stored by journald or syslog.
The default policy allows anyone to upload or download.
Collections:
KeepproxyPermission:
User:
Download: true
Upload: true
Admin:
Download: true
Upload: true
Permitting WebDAV makes it possible to use WebDAV, S3 API, and upload/download with Workbench 2. It works in terms of individual files. It prints a log each time a user uploads or downloads a file (subject to throttling discussed below). When WebDAVLogEvents (default true) is enabled, it also adds an entry into the API server logs table.
When a user attempts to upload or download from a service without permission, they will receive a 403 Forbidden response. This only applies to file content.
Denying download permission does not deny access to access to XML file listings with PROPFIND, or auto-generated HTML documents containing file listings.
Denying upload permission does not deny other operations that modify collections without directly accessing file content, such as MOVE and COPY.
The default policy allows anyone to upload or download.
Collections:
WebDAVPermission:
User:
Download: true
Upload: true
Admin:
Download: true
Upload: true
WebDAVLogEvents: true
WebDAVLogDownloadInterval: 30s
When a user or admin creates a sharing link, a custom scoped token is embedded in that link. This effectively allows anonymous user access to the associated data via that link. These custom scoped tokens are always treated as user tokens for the purposes of restricting download access, even when created by an admin user. In other words, these custom scoped tokens, when used in a sharing link, are always subject to the value of the WebDAVPermission/User/Download configuration setting.
If that custom scoped token is used with arv-get, its use will be subject to the value of the KeepproxyPermission/User/Download configuration setting.
Be aware that even when upload and download from outside the network is not allowed, a user who has access to a shell node or runs a container still has internal access to Keep. (This is necessary to be able to run workflows). From the shell node or container, a user could send data outside the network by some other method, although this requires more intent than accidentally clicking on a link and downloading a file. It is possible to set up a firewall to prevent shell and compute nodes from making connections to hosts outside the private network. Exactly how to configure firewalls is out of scope for this page, as it depends on the specific network infrastructure of your cluster.
This distinction between WebDAV and Keepproxy is important for auditing. WebDAV records ‘upload’ and ‘download’ events on the API server that are included in the User Activity Report, whereas keepproxy only logs upload and download of individual blocks, which require a reverse lookup to determine the collection(s) and file(s) a block is associated with.
You set separate permissions for WebDAV and Keepproxy, with separate policies for regular users and admin users.
These policies apply to only access from outside the cluster, using Workbench or Arvados CLI tools.
The WebDAVLogEvents option should be enabled if you intend to the run the User Activity Report. If you don’t need audits, or you are running a site that is mostly serving public data to anonymous downloaders, you can disable it to avoid the extra API server request.
For ease of access auditing, this policy prevents downloads using arv-get. Downloads through WebDAV and S3 API are permitted, but logged. Uploads are allowed.
Collections:
WebDAVPermission:
User:
Download: true
Upload: true
Admin:
Download: true
Upload: true
KeepproxyPermission:
User:
Download: false
Upload: true
Admin:
Download: false
Upload: true
WebDAVLogEvents: true
This policy prevents regular users (non-admin) from downloading data. Uploading is allowed. This supports the case where restricted data sets are shared with users so that they may run their own data analysis on the cluster, while preventing them from downloading the data set to their local workstation. Be aware that users won’t be able to download the results of their analysis, either, requiring an admin in the loop or some other process to release results.
Collections:
WebDAVPermission:
User:
Download: false
Upload: true
Admin:
Download: true
Upload: true
KeepproxyPermission:
User:
Download: false
Upload: true
Admin:
Download: true
Upload: true
WebDAVLogEvents: true
This policy is suitable for an installation where data is being shared with a group of users who are allowed to download the data, but not permitted to store their own data on the cluster.
Collections:
WebDAVPermission:
User:
Download: true
Upload: false
Admin:
Download: true
Upload: true
KeepproxyPermission:
User:
Download: true
Upload: false
Admin:
Download: true
Upload: true
WebDAVLogEvents: true
When WebDAVLogEvents is enabled, uploads and downloads of files are logged in the Arvados audit log. These events are included in the User Activity Report. The audit log can also be accessed via the API, SDKs or command line. For example, to show the 100 most recent file downloads:
arv log list --filters '[["event_type","=","file_download"]]' -o 'created_at desc' -l 100
For uploads, use the file_upload event type.
Note that this only covers upload and download activity via WebDAV, S3, and Workbench 2.
The arv-get and arv-put tools upload via Keepproxy, which does not log activity to the audit log because it operates at the block level, not the file level. Keepproxy records the uuid of the user that owns the token used in the request in its system logs. Those logs are usually stored by journald or syslog. A typical log line for such a block download looks like this:
Jul 20 15:03:38 keep.xxxx1.arvadosapi.com keepproxy[63828]: {"level":"info","locator":"abcdefghijklmnopqrstuvwxyz012345+53251584","msg":"Block download","time":"2021-07-20T15:03:38.458792300Z","user_full_name":"Albert User","user_uuid":"ce8i5-tpzed-abcdefghijklmno"}
It is possible to do a reverse lookup from the locator to find all matching collections: the manifest_text field of a collection lists all the block locators that are part of the collection. The manifest_text field also provides the relevant filename in the collection. Because this lookup is rather involved and there is no automated tool to do it, we recommend disabling KeepproxyPermission.User.Download and KeepproxyPermission.User.Upload for sites where the audit log is important and arv-get and arv-put are not essential.
If a client requests partial content past the start of a file, and a request from the same client for the same file was logged within the last time interval configured by WebDAVLogDownloadInterval, keep-web will not write a new log. This throttling applies to both printed and API server logs. The default value of 30 seconds reduces log output when clients like aws s3 cp download one file in small chunks in parallel. Administrators can set this setting to 0 to disable log throttling. This setting lets administrators choose how they want to balance full auditability against logging overhead: a shorter interval means more download requests are logged, with all the overhead that entails.
The content of this documentation is licensed under the
Creative
Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the
Apache License, Version 2.0.