Skip checking resources when `--wait=false` is specified

Question

Skip checking resources when `--wait=false` is specified

firgavin opened this issue 2 years ago · comments

What steps did you take:
I currently use Kapp as a CI tool to manage lots of YAML files. I used --wait=false when I deleted the app because sometimes deleting custom resources will take a long time.

What happened:
kapp exits with non-zero code which makes CI fail.

$ kapp delete -a app1 --wait=false -y
Target cluster 'https://127.0.0.1:6443' (nodes: firgavin)

Changes

Namespace  Name        Kind        Age  Op      Op st.  Wait to  Rs  Ri  
default    simple-app  Deployment  22s  delete  -       -        ok  -  
^          simple-app  Service     22s  delete  -       -        ok  -  

Op:      0 create, 2 delete, 0 update, 0 noop, 0 exists
Wait to: 0 reconcile, 0 delete, 2 noop

11:18:59AM: ---- applying 2 changes [0/2 done] ----
11:18:59AM: delete deployment/simple-app (apps/v1) namespace: default
11:18:59AM: delete service/simple-app (v1) namespace: default
11:18:59AM: ---- waiting on 2 changes [0/2 done] ----
11:18:59AM: ok: noop service/simple-app (v1) namespace: default
11:18:59AM: ok: noop deployment/simple-app (apps/v1) namespace: default
11:18:59AM: ---- applying complete [2/2 done] ----
11:18:59AM: ---- waiting complete [2/2 done] ----

kapp: Error: Expected all resources to be gone, but found: endpointslice/simple-app-vp2dw (discovery.k8s.io/v1) namespace: default, pod/simple-app-64c66864f5-g9sb8 (v1) namespace: default, replicaset/simple-app-64c66864f5 (apps/v1) namespace: default

What did you expect:
Kapp could skip checking resources when --wait=false is specified.

Anything else you would like to add:
I did some research and I found that kapp checks the existence of related resources after applying changes. But resources will be deleted eventually. See https://github.com/vmware-tanzu/carvel-kapp/blob/v0.52.0/pkg/kapp/cmd/app/delete.go#L159.
It would be great if kapp could default to skipping checking resources when --wait=false is specified or add a flag to control this logic. And if that makes sense, I'd like to help implement this ;)

Environment:

kapp version (use kapp --version): v0.52.0
OS (e.g. from /etc/os-release): Ubuntu 20.04.4 LTS
Kubernetes version (use kubectl version): v1.23.6+k3s1

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Praveen Rewar · Answer 1 · Fri Aug 12 2022 15:23:28 GMT+0800 (China Standard Time)

Yeah, it seems like setting the wait flag to false would currently lead to an error while deleting recorded apps. So definitely it's a bug.

It would be great if kapp could default to skipping checking resources when --wait=false is specified or add a flag to control this logic.

It does makes sense to allow that behaviour, I am just trying to think of any side effects it could have. One obvious thing that could happen is that one or more resources are not deleted but the app itself (metadata configmap) is deleted.
@cppforlife Any thoughts?

And if that makes sense, I'd like to help implement this ;)

That would be great, we will definitely review it on priority once we finalize the approach :)

Renu Yarday · Answer 2 · Tue Aug 16 2022 17:02:07 GMT+0800 (China Standard Time)

Hey @firgavin good to see your here. Looking forward to your PR for this issue.

Soumik Majumder · Answer 3 · Tue Aug 16 2022 17:06:13 GMT+0800 (China Standard Time)

One obvious thing that could happen is that one or more resources are not deleted but the app itself

This would be a "known risk" I guess?

We might also lose out on some "retryable cases", where kapp would retry in case of a failed delete due to a retryable error.

Dmitriy Kalinin · Answer 4 · Fri Aug 19 2022 00:57:48 GMT+0800 (China Standard Time)

I did some research and I found that kapp checks the existence of related resources after applying changes. But resources will be deleted eventually.

i think additional flag would be reasonable to disable this check. may be under dangerous?

Soumik Majumder · Answer 5 · Fri Aug 19 2022 01:05:19 GMT+0800 (China Standard Time)

i think additional flag would be reasonable to disable this check. may be under dangerous?

This approach makes sense to me

firgavin · Answer 6 · Sat Aug 20 2022 11:49:02 GMT+0800 (China Standard Time)

Hi @cppforlife, @100mik, @praveenrewar - Thanks for your insights! Here's my proposal:

We can add a flag --dangerous-disable-checking-app-deletion to enable or disable the check:

The value is set to false by default, which is compatible with the current behavior.
Once the flag is specified, kapp skips this check, and users might need to manually delete related resources.

Before I work on it, I'd like to discuss the interaction between the two flags. When --dangerous-disable-checking-app-deletion=false, should we make sure that the value of --wait is overwritten to True? If not, users can still hit the same issue. Of course, we can explain the usage in the docs if we think they should be "orthogonal". Any suggestions?

Praveen Rewar · Answer 7 · Mon Aug 22 2022 12:20:44 GMT+0800 (China Standard Time)

When --dangerous-disable-checking-app-deletion=false, should we make sure that the value of --wait is overwritten to True?

I think that we should keep the working of these 2 flags independent of each other because a user should be able to use --dangerous-disable-checking-app-deletion irrespective of --wait being enabled or disabled.

If not, users can still hit the same issue. Of course, we can explain the usage in the docs if we think they should be "orthogonal". Any suggestions?

Maybe we can add a hint in the error message?