Task Migration
Migrate tasks from old task store to new kv store
Expectations
During the migration tasks will exist in both systems. This migration is not transactional but it is repeatable and reversable.
We can expect to lose our run history. Migrating that data would be slow and has a much greater potential to lose data and cause the migration to fail.
It is possible for a short period of time the task could be run by both the old system and the new. This can only happen if the deployment in k8s hangs during deploy.
Required Steps
- Merge the PR that integrates the new system 3537
- Get the image of the currently running task service (in case we need to roll back).
- Get the image that needs to be pushed to the live servers.
- Establish port forward to the production etcd server.
- Ensure new task data location is empty.
etcdctl get --prefix=true tasksv1 - Run Migration tool.
- Deploy new task image
two-prod set image deployment/tasks tasks=quay.io/influxdb/tasks:<tag> - Run Migration tool again (ensure data is up to date).
- Manually confirm thats are running using a user logged into production and by viewing task service logs.
Failure Procedure
- Roll back to previous taskd image.
two-prod rollout undo deploy/task - clean out cruft created by migration tool.
etcdctl del --prefix=true tasksv1