[Cadence] Use of helm install --wait causes the jobs to not be created
drewhemm opened this issue · comments
Describe the bug
When installing this chart with the --wait
flag, because of the post-install
hook on the resources in cadence/templates/server-job.yaml
, they do not get created until Helm considers the app installed, but because the pods will not come online until the jobs have run, there is a circular dependency.
Steps to reproduce the issue:
Attempt to install the chart using helm install --wait
or helm upgrade --install --wait
Expected behavior
The chart should install correctly, waiting for all resources to be created
Here is some output from running helm with --debug
that shows the circular dependency very clearly:
client.go:108: [debug] creating 190 resource(s)
wait.go:53: [debug] beginning wait for 190 resources with timeout of 30m0s
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
Helm is going to wait for these pods to be ready, and they never will be because the jobs have not run. Here is how the debug output looks when the --wait
flag is absent:
client.go:108: [debug] creating 197 resource(s)
client.go:108: [debug] creating 1 resource(s)
client.go:463: [debug] Watching for changes to Job test-cadence-schema-setup with timeout of 10m0s
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: ADDED
client.go:530: [debug] test-cadence-schema-setup: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: MODIFIED
client.go:530: [debug] test-cadence-schema-setup: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: MODIFIED
client.go:108: [debug] creating 1 resource(s)
client.go:463: [debug] Watching for changes to Job test-cadence-schema-update with timeout of 10m0s
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: ADDED
client.go:530: [debug] test-cadence-schema-update: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: MODIFIED
client.go:530: [debug] test-cadence-schema-update: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: MODIFIED
client.go:254: [debug] Starting delete for "test-cadence-schema-setup" Job
client.go:254: [debug] Starting delete for "test-cadence-schema-update" Job
Thanks @drewhemm !
We'll look into it.