Migrating Docker images

If you have an existing Arvados installation using Docker 1.9 and wish to update to Docker 1.10+, you must migrate the Docker images stored in Keep.

The arv-migrate-docker19 tool converts Docker images stored in Arvados from image format v1 (Docker <= 1.9) to image format v2 (Docker >= 1.10).

Requires Docker running on the local host (can be either 1.9 or 1.10+). Linux kernel >= 3.18-rc6 to support overlayfs.

Usage:

  1. Install arvados/migrate-docker19 image: docker pull arvados/migrate-docker19:1.0. If you’re unable to do this, you can run arvados/docker/migrate-docker19/build.sh to create arvados/migrate-docker19 Docker image.
  2. Make sure you have the right modules installed: sudo modprobe overlayfs bridge br_netfilter nf_nat
  3. Set ARVADOS_API_HOST and ARVADOS_API_TOKEN to the cluster you want to migrate.
  4. Your temporary directory should have the size of all layers of the biggest image in the cluster, this is hard to estimate, but you can start with five times that size. You can set up a different directory by using the --tmp-dir switch. Make sure that the user running the docker daemon has permissions to write in that directory.
  5. Run arv-migrate-docker19 --dry-run from the Arvados Python SDK on the host (not in a container). This will print out some information useful for the migration.
  6. Finally to make the migration run arv-migrate-docker19 from the Arvados Python SDK on the host (not in a container).

This will query Arvados for v1 format Docker images. For each image that does not already have a corresponding v2 format image (as indicated by a docker_image_migration tag) it will perform the following process:

i) download the image from Arvados
ii) load it into Docker
iii) update the Docker version, which updates the image
iv) save the v2 format image and upload to Arvados
v) create a migration link

Once the Docker images in Keep have been migrated, upgrade the version of Docker used across the cluster. Finally, update the API server configuration from “v1” to “v2” to reflect the supported Docker image version:

docker_image_formats: ["v2"]

Previous: Cheat Sheet

The content of this documentation is licensed under the Creative Commons Attribution-Share Alike 3.0 United States licence.
Code samples in this documentation are licensed under the Apache License, Version 2.0.