Arvados™ Docs
WARNING - you are viewing the documentation for an old version of Arvados. For the latest version, click here.
  1. Overview
    1. Installation options
  2. Docker quick start
    1. Arvados-in-a-box
  3. Arvados on Kubernetes
    1. Arvados on Kubernetes
    2. Arvados on Minikube
    3. Arvados on GKE
  4. Manual installation
    1. Planning and prerequisites
    2. Arvados package repositories
    3. Upgrading Arvados and Release notes
  5. Configuration
    1. Configuration files
    2. Migrating Configuration from v1.4 to v2.0
    3. Configuration reference
  6. Core
    1. Install API server and Controller
  7. Keep
    1. Install Keepstore servers
    2. Configure filesystem storage
    3. Configure S3 object storage
    4. Configure Azure Blob storage
    5. Install Keepproxy server
    6. Install Keep-web server
    7. Install Keep-balance
  8. User interface
    1. Set up web based login
    2. Install the Single Sign On (SSO) server
    3. Install Workbench
    4. Install Workbench 2
    5. Install Composer
  9. Additional services
    1. Install the websocket server
    2. Install the Git server
    3. Set up a shell node
    4. Configure webshell
  10. Containers API
    1. Set up a compute node
    2. Install arvados/jobs image
    3. Install the cloud dispatcher
    4. Install the SLURM dispatcher
    5. Test SLURM dispatch
  11. External dependencies
    1. Install PostgreSQL 9.4+
    2. Install Ruby and Bundler
    3. Install Nginx
    4. Setting up Google auth
    5. Set up Docker

Migrating Configuration from v1.4 to v2.0

Note:

New installations of Arvados 2.0+ can skip this section

Arvados 2.0 migrates to a centralized configuration file for all components. The centralized Arvados configuration is /etc/arvados/config.yml. Components that support the new centralized configuration are listed below. During the migration period, legacy configuration files are still loaded and take precedence over the centralized configuration file.

API server

The legacy API server configuration is stored in config/application.yml and config/database.yml. After migration to /etc/arvados/config.yml, both of these files should be moved out of the way and/or deleted.

Change to the API server directory and use the following commands:

$ RAILS_ENV=production bundle exec rake config:migrate > config.yml
$ cp config.yml /etc/arvados/config.yml

This will print the contents of config.yml after merging the legacy application.yml and database.yml into the existing systemwide config.yml. It may be redirected to a file and copied to /etc/arvados/config.yml (it is safe to copy over, all configuration items from the existing /etc/arvados/config.yml will be included in the migrated output).

If you wish to update config.yml configuration by hand, or check that everything has been migrated, use config:diff to print configuration items that differ between application.yml and the system config.yml.

$ RAILS_ENV=production bundle exec rake config:diff

This command will also report if no migrations are required.

Workbench

The legacy workbench configuration is stored in config/application.yml. After migration to /etc/arvados/config.yml, this file should be moved out of the way and/or deleted.

Change to the workbench server directory and use the following commands:

$ RAILS_ENV=production bundle exec rake config:migrate > config.yml
$ cp config.yml /etc/arvados/config.yml

This will print the contents of config.yml after merging the legacy application.yml into the existing systemwide config.yml. It may be redirected to a file and copied to /etc/arvados/config.yml (it is safe to copy over, all configuration items from the existing /etc/arvados/config.yml will be included in the migrated output).

If you wish to update config.yml configuration by hand, or check that everything has been migrated, use config:diff to print configuration items that differ between application.yml and the system config.yml.

$ RAILS_ENV=production bundle exec rake config:diff

This command will also report if no migrations are required.

keepstore, keep-web, crunch-dispatch-slurm, arvados-ws, keepproxy, arv-git-httpd, keep-balance

The legacy config for each component (loaded from /etc/arvados/component/component.yml or a different location specified via the -legacy-component-config command line argument) takes precedence over the centralized config. After you migrate everything from the legacy config to the centralized config, you should delete /etc/arvados/component/component.yml and/or stop using the corresponding -legacy-component-config argument.

To migrate a component configuration, do this on each node that runs an Arvados service:

  1. Ensure that the latest config.yml is installed on the current node
  2. Install arvados-server using apt-get or yum.
  3. Run arvados-server config-check, review and apply the recommended changes to /etc/arvados/config.yml
  4. After applying changes, re-run arvados-server config-check again to check for additional warnings and recommendations.
  5. When you are satisfied, delete the legacy config file, restart the service, and check its startup logs.
  6. Copy the updated config.yml file to your next node, and repeat the process there.

After migrating and removing all legacy config files, make sure the /etc/arvados/config.yml file is identical across all system nodes — API server, keepstore, etc. — and restart all services to make sure they are using the latest configuration.

Cloud installations only: node manager

Node manager is deprecated and replaced by arvados-dispatch-cloud. No automated config migration is available. Follow the instructions to install the cloud dispatcher

Only one dispatch process should be running at a time. If you are migrating a system that currently runs Node manager and crunch-dispatch-slurm, it is safest to remove the crunch-dispatch-slurm service entirely before installing arvados-dispatch-cloud.

~$ sudo systemctl --now disable crunch-dispatch-slurm
~$ sudo apt-get remove crunch-dispatch-slurm

arvados-controller, arvados-dispatch-cloud

Already uses centralized config exclusively. No migration needed.

Previous: Configuration files Next: Configuration reference

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.