[BUG] Config dependency doesn't work when deployers are different
mecampbellsoup opened this issue · comments
I am trying to marry 2 skaffold files we have, one called cloud-app which builds our app and deploys it to k8s target cluster using helm deployer, and another called cloud-app-e2e-test-runner which is a simple playwright e2e test suite that we typically have deployed using kubectl deployer.
Here is my desired skaffold.yaml where I want to use cloud-app skaffold as a dependent configuration:
apiVersion: skaffold/v4beta9
kind: Config
metadata:
name: cloud-app-e2e-test-runner
requires:
- path: ../../../k8s-services/dev/skaffold-cloud-app.yaml
configs: ["cloud-app"]
activeProfiles:
- name: production
build:
artifacts:
- image: cloud-app-e2e-test-runner
context: ./
docker:
dockerfile: Dockerfile
noCache: false
sync:
manual:
- src: "tests/**"
dest: .
- src: "*.ts"
dest: .
- src: "package.json"
dest: .
- src: "playwright.config.ts"
dest: .
deploy:
kubectl: {}The build and deploy steps in cloud-app config succeed, but then when it comes time to do the cloud-app-e2e-test-runner skaffold config, I get an error:
Deployments stabilized in 57.127 seconds
Cleaning up...
- No resources found
release "cloud-app" uninstalled
nothing to deploy
Possibly related to #8745.
Full stdout logs
(⎈|default:cloud)mcampbell-1➜ ~/github/coreweave/cloud-app/cloud_app/e2e : mc/3524/e2e-ci ✘ :✹✚✭ ᐅ skaffold dev --default-repo registry.gitlab.com/coreweave/cloud-app/dev
Generating tags...
- api-gateway -> registry.gitlab.com/coreweave/cloud-app/dev/api-gateway:2024-05-07_17-24-03.405_UTC
- cloud-app -> registry.gitlab.com/coreweave/cloud-app/dev/cloud-app:2024-05-07_17-24-03.405_UTC
- cloud-app-e2e-test-runner -> registry.gitlab.com/coreweave/cloud-app/dev/cloud-app-e2e-test-runner:v1.50.0-4253-g47247432-dirty
Checking cache...
- api-gateway: Found. Tagging
- cloud-app: Found. Tagging
- cloud-app-e2e-test-runner: Found Remotely
Tags used in deployment:
- api-gateway -> registry.gitlab.com/coreweave/cloud-app/dev/api-gateway:2024-05-07_17-24-03.405_UTC@sha256:2dc379cc22227cecb86afce97a28c9c5b2fb188d0b8e32a3097cb9d71b6e4fb9
- cloud-app -> registry.gitlab.com/coreweave/cloud-app/dev/cloud-app:2024-05-07_17-24-03.405_UTC@sha256:c6afa54660939f9eac9b16ead2f050e0dcae770dd2d4fea5450f65731fa45002
- cloud-app-e2e-test-runner -> registry.gitlab.com/coreweave/cloud-app/dev/cloud-app-e2e-test-runner:v1.50.0-4253-g47247432-dirty@sha256:fc281427f9ebefd19ed5a23ac4e8e51df18ab2abbd04ea8d046c6a2d7868af72
Starting deploy...
Helm release cloud-app not installed. Installing...
NAME: cloud-app
LAST DEPLOYED: Tue May 7 17:24:07 2024
NAMESPACE: cloud
STATUS: deployed
REVISION: 1
TEST SUITE: None
Waiting for deployments to stabilize...
I0507 17:24:13.927584 2115767 request.go:697] Waited for 1.132951171s due to client-side throttling, not priority and fairness, request: GET:https://10.100.1.2:32500/api/v1/namespaces/cloud/pods?labelSelector=app%3Dcloud-app%2Capp.kubernetes.io%2Finstance%3Dcloud-app%2Capp.kubernetes.io%2Fname%3Dcloud-app%2Cskaffold.dev%2Frun-id%3De730be1b-03ab-487a-9721-103f3556957b
- cloud:deployment/cloud-app: waiting for rollout to finish: 0 of 1 updated replicas are available...
- cloud:deployment/cloud-app-alerts-notifier: waiting for rollout to finish: 0 of 1 updated replicas are available...
- cloud:deployment/cloud-app-kubernetes-ingress: Startup probe failed: Get "http://10.241.126.195:1042/healthz": dial tcp 10.241.126.195:1042: connect: connection refused
- cloud:pod/cloud-app-kubernetes-ingress-89b9566d6-xtnrh: Startup probe failed: Get "http://10.241.126.195:1042/healthz": dial tcp 10.241.126.195:1042: connect: connection refused
- cloud:deployment/cloud-app-metrics: waiting for rollout to finish: 0 of 1 updated replicas are available...
- cloud:deployment/cloud-app-reconciler: waiting for init container cloud-app-reconciler-pg-waiter to start
- cloud:pod/cloud-app-reconciler-59d4b98b4d-8gvgn: waiting for init container cloud-app-reconciler-pg-waiter to start
- cloud:deployment/cloud-app-worker-auth: waiting for rollout to finish: 0 of 1 updated replicas are available...
- cloud:deployment/cloud-app-worker-chargify: waiting for init container cloud-app-worker-chargify-pg-waiter to start
- cloud:pod/cloud-app-worker-chargify-65b7d9765-b669r: waiting for init container cloud-app-worker-chargify-pg-waiter to start
- cloud:deployment/cloud-app-worker-messaging: waiting for rollout to finish: 0 of 1 updated replicas are available...
- cloud:deployment/cloud-app-worker-sift: waiting for rollout to finish: 0 of 1 updated replicas are available...
- cloud:statefulset/cloud-app-postgresql: Waiting for 1 pods to be ready...
- cloud:deployment/cloud-app-worker-auth: container cloud-app-worker-auth-pg-waiter in error: &ContainerStateWaiting{Reason:CreateContainerConfigError,Message:secret "postgres-role-postgres" not found,}
- cloud:pod/cloud-app-worker-auth-65cbb6f9bd-ncr22: container cloud-app-worker-auth-pg-waiter in error: &ContainerStateWaiting{Reason:CreateContainerConfigError,Message:secret "postgres-role-postgres" not found,}
I0507 17:24:23.927790 2115767 request.go:697] Waited for 1.987295127s due to client-side throttling, not priority and fairness, request: GET:https://10.100.1.2:32500/api/v1/namespaces/cloud/events?fieldSelector=involvedObject.name%3Dcloud-app-95b66b8bc-pd4hz%2CinvolvedObject.namespace%3Dcloud%2CinvolvedObject.kind%3DPod%2CinvolvedObject.uid%3Dd5a7d1b1-4a38-436b-8ed0-d59704070cb3
- cloud:statefulset/cloud-app-postgresql is ready. [9/10 deployment(s) still pending]
- cloud:deployment/cloud-app-reconciler is ready. [8/10 deployment(s) still pending]
- cloud:deployment/cloud-app-worker-messaging is ready. [7/10 deployment(s) still pending]
- cloud:deployment/cloud-app-alerts-notifier is ready. [6/10 deployment(s) still pending]
I0507 17:24:34.127911 2115767 request.go:697] Waited for 1.302943837s due to client-side throttling, not priority and fairness, request: GET:https://10.100.1.2:32500/api/v1/namespaces/cloud/pods?labelSelector=app%3Dqueue-worker-auth%2Capp.kubernetes.io%2Finstance%3Dcloud-app-worker-auth%2Capp.kubernetes.io%2Fname%3Dcloud-app-worker-auth%2Cskaffold.dev%2Frun-id%3De730be1b-03ab-487a-9721-103f3556957b
- cloud:deployment/cloud-app-worker-chargify is ready. [5/10 deployment(s) still pending]
- cloud:deployment/cloud-app-worker-sift is ready. [4/10 deployment(s) still pending]
- cloud:deployment/cloud-app-kubernetes-ingress is ready. [3/10 deployment(s) still pending]
- cloud:deployment/cloud-app-worker-auth is ready. [2/10 deployment(s) still pending]
- cloud:deployment/cloud-app is ready. [1/10 deployment(s) still pending]
- cloud:deployment/cloud-app-metrics is ready.
Deployments stabilized in 57.127 seconds
Cleaning up...
- No resources found
release "cloud-app" uninstalled
nothing to deploy
it outputs error:
- cloud:pod/cloud-app-worker-auth-65cbb6f9bd-ncr22: container cloud-app-worker-auth-pg-waiter in error: &ContainerStateWaiting{Reason:CreateContainerConfigError,Message:secret "postgres-role-postgres" not found,}.
I think you forgot to add this role
it outputs error:
- cloud:pod/cloud-app-worker-auth-65cbb6f9bd-ncr22: container cloud-app-worker-auth-pg-waiter in error: &ContainerStateWaiting{Reason:CreateContainerConfigError,Message:secret "postgres-role-postgres" not found,}. I think you forgot to add this role
Thanks for the suggestion, but unfortunately that isn't related.
(That error happens because this app applies sealed-secrets - so when I initially helm install, there's a bit of time lag in which our sealed secrets are applied in the cluster; and then the sealed secrets operator converts them into k8s secrets. Until that process is complete (usually just a few seconds) however we get those errors, but once the secrets are ready helm can complete installation.)
For the purposes of this issue, the only relevant log output is really:
Deployments stabilized in 57.127 seconds
Cleaning up...
- No resources found