managedFields options are very verbose, making kubectl usage a bit
howardjohn opened this issue · comments
Recently (1.18?) a new field was added to configs, managedFields. This is incredibly verbose, which makes kubectl commands like -oyaml and edit painful.
For example, on a random deployment I have looked at, the entire kubectl get deployment -oyaml is 640 lines. Managed fields is over half of that, with 358 lines.
I am not sure if its feasible to somehow mitigate this without breaking server side apply or other features, but it would be useful if possible. One possible idea would be to just hide it from kubectl commands, but I am not sure the impact of that. The goal of this issue is mostly to bring awareness that this causes some user pain and see if we can improve that a bit.
Version:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T20:56:08Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
/sig cli
Hi @howardjohn
I understand that this is a voluminous field. The team has discussed that issue multiple times, but we've decided that the information provided is useful for the users.
First, feel free to read this blog-post which will explain what it does and why it's here: https://kubernetes.io/blog/2020/04/01/kubernetes-1.18-feature-server-side-apply-beta-2/
Also, I think I could have emphasized in the blog how the fields can be used to "blame" (à la git-blame) the changes made to an object. For example, you can use it to see what and when a controller adds or changes a specific field of an object. It's also a great way to learn the intricate ways of Kubernetes and understand its asynchronous behavior (the service account controller adds the service account token volume to a pod, the "neg-status" label is set asynchronously by the neg controller, etc).
I hope that helps, and thank you for the feedback!
I would like to propose that command -o yaml | json commands by default hide the managedFields from output and show it if user passes a flag like --show-managed-fields.
While I agree the output is verbose, kubectl tries very hard to be as transparent as possible when outputting what the server returns in json/yaml format. Introspecting/manipulating particular fields in the response could be done if absolutely required, but is less than ideal.
I find that everyday workflows like kubectl edit and -o yaml | json are extremely impacted by the addition of managedFields. I think every developer will have a similar shocking reaction to this field. While the blog post is a great resource, more communication is necessary for introducing this field and its usefulness.
At first I thought that hiding it by default in -o yaml | json would be nice but I agree that introspecting/manipulating particular fields should not be done by kubectl
same here please opt-in managed-fields as extra kubectl flag
I think too it could be optional to show managed fields
I think it should be possible to suppress it, so I can "export" resources as YAML and use them for other purposes. So, opt-out.
Didn't we have something like --export in the past?
I feel that this is a good place to recommend using a plugin to clean up resources: kubectl neat https://github.com/itaysk/kubectl-neat
@towolf there was a --export field which got deprecated. that has lead me to create a replacement in the form of a kubectl plugin (kubectl-neat). Since the original use case for export is not longer required this plugin goal has evolved from "exporting a resource for recreation" to "make resources more readable for end users". for reference: kubernetes-sigs/krew-index#191 (comment)
@itaysk I found your plugin and I really like it. However, to have --export back in kubectl itself to obtain "neat" yaml, that would be much more compelling. So many applications come to mind.
kubectl-neat plugin seems to solve this issue quite elegantly. Would it be possible to integrate kubectl-neat's features set into kubectl get ?
I don't believe that kubectl should decide which fields people think are important or not. kubectl is implemented to display the object as it is received from the server. What we could do is print the managedFields as a more compact, one-line json format? I suspect we could also add another format option but since this would never be the default, I don't know how useful it would be.
What we could do is print the managedFields as a more compact, one-line json format?
No, that's worse. And again unwieldy like the thing this replaced.
I don't believe that kubectl should decide which fields people think are important or not
True, but giving a way for the user to filter out the info he/she doesn't need would improve significantly the user experience.
I hear you all, do you all agree that this isn't the only field that could benefit from at least a less prominent location if not outright hiding?
The primary risk in hiding fields is in accidentally causing some workflow that eventually updates an object to then delete the hidden fields. That shouldn't be an issue for this particular field (depending on how much we obscure), but it would definitely be an issue for some other fields.
How much of the problem is in -o yaml vs edit? edit could probably obscure fields completely safely (i.e., merged the hidden fields back with the thing with the user's input).
I'm also a little reluctant to filter fields in -o yaml, as that's the easiest way to know the exact state of an object right now. If we add filtering, I think we need a -o rawyaml that doesn't filter. OTOH if we're going to end up with an additional format, why not leave -o yaml the way it is and add a short / filtered yaml output?
maybe hide it only in describe which is manipulating output anyway
Describe should definitely not show this. (except for maybe the update timestamp.) Does it right now? If so that's a bug.
sorry I didn't check..
Describe should definitely not show this. (except for maybe the update timestamp.) Does it right now? If so that's a bug.
Kubernetes 1.18: kubectl describe pod/mypod does NOT display the managedFields. kubectl get pod/mypod -o yaml does show managedFields.
Great, that's what I expected. We should probably make it show the most recent update timestamp, I don't think we added that yet.
I'm quite new to k8s. You can disable managedFields. managedField is a featuregate and it's in beta. I encourage you do not disable, because the documentation says:
Note: Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.
I've done a little test disabling it in a minikube env e it worked as expected. I did it just for learning purpose.
mini-kube start --feature-gates=ServerSideApply=false
kubectl get pods/kube-proxy-sb794 -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2020-06-05T13:40:31Z"
generateName: kube-proxy-
labels:
controller-revision-hash: 58b47d6fd4
k8s-app: kube-proxy
pod-template-generation: "1"
name: kube-proxy-sb794
namespace: kube-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: DaemonSet
name: kube-proxy
uid: 2561d935-a809-4aab-850b-540b9336ccff
resourceVersion: "447"
selfLink: /api/v1/namespaces/kube-system/pods/kube-proxy-sb794
uid: 1f8afe9e-9988-44f1-8b05-a7217ddf9d16
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchFields:
- key: metadata.name
operator: In
values:
- la2
containers:
- command:
- /usr/local/bin/kube-proxy
- --config=/var/lib/kube-proxy/config.conf
- --hostname-override=$(NODE_NAME)
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: k8s.gcr.io/kube-proxy:v1.18.3
imagePullPolicy: IfNotPresent
name: kube-proxy
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kube-proxy
name: kube-proxy
- mountPath: /run/xtables.lock
name: xtables-lock
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-proxy-token-xhkdj
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostNetwork: true
nodeName: la2
nodeSelector:
kubernetes.io/os: linux
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: kube-proxy
serviceAccountName: kube-proxy
terminationGracePeriodSeconds: 30
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- operator: Exists
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/network-unavailable
operator: Exists
volumes:
- configMap:
defaultMode: 420
name: kube-proxy
name: kube-proxy
- hostPath:
path: /run/xtables.lock
type: FileOrCreate
name: xtables-lock
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
- name: kube-proxy-token-xhkdj
secret:
defaultMode: 420
secretName: kube-proxy-token-xhkdj
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-06-05T13:40:31Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-06-05T13:40:34Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-06-05T13:40:34Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-06-05T13:40:31Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://4eb4e3ff978bba60055ad54cfea95d1f0cb88e1b0fc208705ed189827a56fe9c
image: k8s.gcr.io/kube-proxy:v1.18.3
imageID: docker-pullable://k8s.gcr.io/kube-proxy@sha256:6a093c22e305039b7bd6c3f8eab8f202ad8238066ed210857b25524443aa8aff
lastState: {}
name: kube-proxy
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2020-06-05T13:40:33Z"
hostIP: 192.168.99.141
phase: Running
podIP: 192.168.99.141
podIPs:
- ip: 192.168.99.141
qosClass: BestEffort
startTime: "2020-06-05T13:40:31Z"```
Disabling the field is not a viable option because most users do not have access to k8s feature flags to disable it, and its only a short term mitigation. It would be ideally if there was a longer term mitigation to lessen the impact of the field
There is no need of disabling any feature/function. The only thing needed is a way to filter the output. Kind of similar to the ls command. You may have many thousands files in a directory, but you can filter the output by an expression like ls mystuff*.log. Not sure how this kind of output filtering should be implemented in an idiomatic way consistent with kubectl get. From the user point of view, the idea is just to hide some info from the output.
I think the current plan is to do a -o shortyaml or something, but another option might be to add an inverse jsonpath expression-- print all fields that DON'T match, like grep -v. That'd make for longer command lines but would be more controllable.
(It appears impossible to get this behavior with the json path option, and also very difficult with jq scripts--at least I wasn't able to figure one out in a small amount of time.)
Thoughts between those options?
add an inverse jsonpath expression-- print all fields that DON'T match, like
grep -v
Hum.. with this kind of not in filter you may end up with almost the rest of the content of the universe. Like "who is not John". I think it would be clearer to use an "exclude" clause to let the user decide explicitly the piece of info to be excluded. Something like
kubectl get pod/mypod1 -o yaml --exludeKeys=[managedFields, fieldABC, keyXZY]
Yeah that's mostly what I had in mind, but I don't know if just the key is sufficiently unique. I was thinking you'd write a json path expression, and all matching fields (and their descendants) would be excluded. like -o yaml -vjp '.metadata.managedFields[].fieldsV1' to exclude the verbose parts or -o yaml -vjp '.metadata.managedFields' if you don't want to see any of it at all.
I'm concerned the trade off of document size invalidates the usefulness of this design. Addressing the UX side is tricky, but just the fact that so many objects get much larger really concerns. I know "kubectl apply" has always bloated things, but only impacted objects directly interacted with with "kubectl apply" which in many cases ends up be small. Now core builtin controllers are adding these fields in a seemly verbose way. As it stands right now, I'm not sure that there is clear obvious benefit to users of this feature. Even people writing controllers, I'm not sure the benefit is substantial, but honestly It not clear to me how to even use "server side apply" at all, so I could be wrong here.
I understand the technical merits, I've personally asked for server side apply, but the tradeoff seems very large.
Sorry for the follow up. I think the designers of this feature should really consider the user feedback of this implementation. I know as a feature is designed there is a lot of discussion during the initial design and alpha phase, but features don't really get seen by users until they hit beta. So discussions that may have already happened during the alpha phase or earlier must be revisited when beta occurs. All to often I see the comment of "yeah we already talked about this" with very little context shutdown the comments from users who are just now seeing the feature.
This specific feature has a fairly large impact on users and any ecosystem tools that needs to interact with the API as the volume is increasing. This hits home for me because as more and more users are using k8s at larger scales the biggest performance issue I'm dealing with right now is just that list/watch sizes are getting unwieldy.
We have a design which makes the field significantly smaller, at the cost of making it not human readable (illegible without processing). We thought we were going to need it to meet performance goals, but it turned out not necessary. We've also considered having the server omit it in responses (e.g., depending on a query parameter). @ibuildthecloud if you are seeing some performance problem and can share data that quantifies it we'd be happy to reevaluate the options.
The point of going beta is actually to get broader user feedback. So far the user feedback here has been about not wanting to see it all the time (not perf related), and as you can see in the comments on this issue we are working out ways to meet that request. The OP has 11 upvotes as I write this, which is decent but not enormous. So, I don't accept the charge that we're not listening to users.
@lavalamp Sorry if I seemed to indicated you aren't listening to users. You all are doing a great job. I think I reacted too strongly to a specific comment in this thread. My fault, sorry for that.
We thought we were going to need it to meet performance goals, but it turned out not necessary.
Can you point me to any more context on this. What the new design was and how it proved to not be necessary.
My specific concern atm is that I have a couple tools that watch basically the entire cluster. Every single object is cached in memory. Obviously that is a lot of data, so we use partialobjectmetadata because I don't need the body of the object. This makes this tool fairly memory light for such a big task. Managed fields has added a new thing that is increasing the memory footprint. I don't have specific data to say how much, but obviously a large number of go maps are now being created. In my specific use case I can easily work around this, so no big deal. I just worry about other tools in the ecosystem that need to do related things. Me personally am almost always an outlier so I don't expect anyone to change based on what I'm encountering. Just take what I say as most likely statistically irrelevant data point.
Sorry if I seemed to indicated...
No worries, sounds like I misread, sorry :)
Can you point me to any more context on this...
See:
Those aged out because we passed scalability tests after this sequence of PRs.
@apelisse might know if we wrote this up or not.
... partialobjectmetadata because I don't need the body ...
Yeah that one in particular we talked about dropping this from, since our own GC uses it. We (well, @jpbetz) ended up fixing some other bugs in the GC, and I think it fell off our radar when the tests stopped flaking. I think we're totally open to a query parameter or content type for omitting managed fields data from partialobjectmetadata.
I'm not so much concerned about the output of kubectl, i can use a plugin to filter that out but i am concerned about the increase in the amount of data in etcd.
The real issue is the average object size, the amount of data in etcd and the increased memory usage on all controllers
I'm personally leaning towards what @lavalamp proposes, iow. suppresing these fields on a case-by-case, namely edit and describe should not display them. get -oyaml or -ojson should be completely neutral and should always give you full dump of the resource as it exists on the server. I can totally support the exclude fields idea for get, though.
This will be discussed in a few minutes during SIG-CLI, I'll write down what was agreed on.
During the aforementioned sig-cli we agree that for now we'll continue with hiding those fields from kubectl edit as implemented in #91946 with remarks from Daniel addressed. For other commands we will not be doing anything, yet.
I'm not sure this is completely fixed just by hiding from kubectl edit. Should we keep this open until there's a convenient exclusion option for get?
If your editor is vi, you can enter :/^spec: right after kubectl edit ... in order to get your cursor on the interesting part of YAML. This solved my workflow for now.
From my point of view, operators care less about added fields to kubernetes resources.
What is really needed, is the way to get resource yaml/json EXACTLY as its been applied.
So.. the essence:
kubectl -n src get pod pod_name -o yaml >> /tmp/pod.yaml \
&& kubectl -n dst apply -f /tmp/pod.yamlI know some people are working on an export command (@seans) which seems unrelated to this issue? I can see how the managedFields might give that ability though.
in my experience it is possible to dump a pod yaml and create a new pod from it without cleaning up all the redundant fields (export), except for the case where moving between namespaces
regarding the "way to get resource yaml/json EXACTLY as its been applied" - I completely agree with @imunhatep . I have tried to add a feature to kubectl-neat that leverages managedFields to keep only fields managed by kubectl. but the problem is with derived resources, for example if the user created a deployment, this feature works on the deployment, but then the replicaset and pods are managed by controller-manager so it won't work. would be nice if there was a way to trace back to the original intent of the user.
would be nice if there was a way to trace back to the original intent of the user.
Follow the owner / controller refs back to the source object?
Yes I have thought about it. this tool was supposed to be able to work without api access, but I already compromised on that requirement.
There's also the issue that feature this needs to be individually coded for every supported scenario, meaning explicitly for the deployment->RS->pod, it wouldn't support any yaml it received.
And finally, when exploring this I found some cases where the field tracking will not go as far as cover all of the fields deep in the hierarchy (like nested in associative lists) which discouraged me from pursuing this but I don't want to pollute this thread with this discussion :) thanks for the suggestion!
for example if the user created a deployment, this feature works on the deployment, but then the replicaset and pods are managed by controller-manager so it won't work
Get yaml/json as its been applied by controller-manager, as managedFields are added later. Correct me here, but probably this added on processing the request by kubeapi.
ATM I see 2 possible approaches here:
keep the list of fields reflecting current resource state, e.g. status, managedFields e.t.c. and hide it on export.
Or to keep kind of a copy of applied resources (original request?) before "state fields" are applied, regardless the manager, but that may require to store a copy in etcd..
Before this is addressed in kubectl get, kubectl-neat works perfectly for me. I can display or hide those fields at will.
e.g.
#using "neat" as the alias for "kubectl neat"
kubectl get po abc -oyaml | neat
Would it be a viable option to allow the suppression of the managedFields data based on a preference set within a user's kubeconfig file? There's already a top-level dictionary called preferences: {} which I would assume was added for exactly this use case.
Maybe? We'd want to be careful not to make scripts behave differently on different systems, by e.g. only hiding if the output is a TTY or something like that.
The fields information sounds like it's mainly for the tool to help me when I'm editing, +1 for hiding it by default in kubectl edit (and have some option to show it in case it is the fields that I want to edit). Could even go so far as kubectl edit --spec so that only the spec is shown; I think that's the 99% use case for kubectl edit
Not that its really a solution to the core complaint above, but I had the same concern until i realized i could get the output im used to seeing by patching the output:
kubectl patch deployment foobar --type=json -p '[{"op": "remove", "path": "/metadata/managedFields"}]' -o yaml --dry-run=client
Its a little involved and not as great as just "kubectl get deploy foobar -oyaml" for example, but it does the job. Could set it as a bash alias/function.
I, too, would very much like to see the output of reading an object not include managedFields. It makes sense that you'd want "get" to always show the complete object. I like the idea of having a flag that says something like "--dont-show-me-that-stuff" that you can add in cases where you don't want to see it. (Which, for me, will be always.)
I also got to thinking about how PowerShell manages verbs. In PowerShell, they try to standardize command names to be "Verb-Noun", and the verbs are rationalized. For exmaple, there's "Get", as in "Get-Process", "Get-Help", "Get-Event", etc. It fetches an object and prints it to stdout. In kubectl the read verbs are get and describe, of course. Maybe this managedFields thing being too verbose could be handled by another verb as a first-level subcommand. So I looked at PowerShell and found Read and Show. Show-Markdown, for example, "Shows a Markdown file or string in the console in a friendly way using VT100 escape sequences or in a browser using HTML."
kubectl show TYPE NAME, showing only a defined set of human-admin-friendly fields, could work.
kubectl show TYPE NAME, showing only a defined set of human-admin-friendly fields, could work.
That is pretty much what kubectl describe is and that already omits the managedFields
Hi, @apelisse, Due to addition managedFields metadata, payload of the every object got increased by 40%. For example, for simple nginx pod, before <1.18, response was ~6KB and >=1.18, response is 10KB. Ours is monitoring application, we use k8s apis to get the response of all objects (chunked fashion) and we dont use this field. The additional payload significantly costing the cpu and memory. is there any way to express the k8s API to exclude in the response ? If not, having functionality like this will significantly helpful for the clients.
How about the following two things:
- We create a new apiserver option that says "don't give me the managed fields", and we would remove it before sending the object over the wire (@wojtek-t can we do that, or would this be difficult because of various caching mechanisms?). We could possibly teach controllers to use that flag so that their memory footprint and data transfer would decrease.
- We add a corresponding output format in kubectl, e.g.
-o short(that's a bad name, please suggest something better 😂 ) that triggers that apiserver behavior. cc @pwittrock
Thanks @apelisse for the response.
How about the following two things:
- We create a new apiserver option that says "don't give me the managed fields", and we would remove it before sending the object over the wire (@wojtek-t can we do that, or would this be difficult because of various caching mechanisms?). We could possibly teach controllers to use that flag so that their memory footprint and data transfer would decrease.
Yes, adding apiserver option certainly help the clients to make decision whether additional managed fields metadata required or not based on the client side requirements. will this option be available for both watch and non-watch streams?
- We add a corresponding output format in kubectl, e.g.
-o short(that's a bad name, please suggest something better 😂 ) that triggers that apiserver behavior. cc @pwittrock
how about --show-managed-fields/-m for kubectl and default will be false?
Yes, adding apiserver option certainly help the clients to make decision whether additional managed fields metadata required or not based on the client side requirements. will this option be available for both watch and non-watch streams?
I would think so, but that might depend on internal constraints related to how watch (and especially its cache) is implemented.
how about --show-managed-fields/-m for kubectl and default will be false?
I'm not convinced an extra field is necessary. It strictly not orthogonal to -o, since the option would be a no-op for all but -o json, -o yaml?
EDIT: Do people feel the same way about -o json?
kubectl get can have a new opt-in flag which would trip those fields, which would work both with -o json and -o yaml. This way we would not change the default behavior, and provide a way to strip them, something like --strip-managed-fields.
As I've mentioned before and as @ganga1980 mentioned as well, the main managed fields problem is not a cosmetic one of hiding them in kubectl, it's a performance one, they increase the CPU, memory and bandwidth utilisation, solving this at the client side is not enough.
We discussed this in our last meeting of wg-api-expression and providing a proper solution for this is on our list of GA Tasks for Server-Side Apply.
We hear you and will be working on it. I think we'd prefer a Server-Side flag to prevent sending the fields instead of a client-only solution.
There are some questions that need to be answered, but we want to solve this as well.
/wg api-expression
Hi, @apelisse , @kwiesmueller, Until this change implemented, is there any work around to exclude managed fields via API? do you guys have any pointers to achieve?
There is currently no API to "remove" the field.
A "use at your own risks" workaround could be:
kubectl patch deployment foobar --type=json -p '[{"op": "replace", "path": "/metadata/managedFields", "value": []}]' -o yaml --dry-run=server
Or its corresponding API call.
There is currently no API to "remove" the field.
A "use at your own risks" workaround could be:
kubectl patch deployment foobar --type=json -p '[{"op": "replace", "path": "/metadata/managedFields", "value": []}]' -o yaml --dry-run=serverOr its corresponding API call.
thanks @apelisse for work around but this work around will not work for me since I am not allowed to make patch API call considering the type of application I work.
Yeah, you unfortunately need write permissions to do that :-(
@knight42 is now working on the ability to hide managedFields, server-side, given a new accept header.
For those of us who are trying
- upgrade to 1.16 => 1.18
- Do not want to see managedFields in "get deployment" and other resources by default. [Both in kubectl(atleast not yet) and other controller/crd]
- Control Api-server settings
Setting --feature-gates=ServerSideApply=false seems like good workaround. Any pitfalls we should be aware of?
@Ramyak you will break any apps that rely on this. Many operators/controllers are starting to depend on this.
Theoretically no operators/controllers should rely on managedFields because we are migrating from 1.16 => 1.17 => 1.18 (and 1.16 and 1.17) did not have managedFields fields for most objects as per
https://kubernetes.io/blog/2020/04/01/kubernetes-1.18-feature-server-side-apply-beta-2/
Are they actually causing a problem for you or is it just a cosmetic nuisance?
The feature gate will be removed (by GA?), it'd be good to detect any problems it causes before that time, both because we'd like to fix them and so you don't get a surprise later. (not that I expect any problems, it's just good to be PROPERLY PARANOID)
One other consideration is that at this point all tests are run with the flag on, and most people run with the flag on (as far as I know). So turning the flag off means you'll be running in a less-tested configuration.
We are little paranoid because our pod object size is 40K before this change (because we have 20 sidecar containers). So we may be off the tested path.
After upgrade on our test cluster we do see managedFields on deployment but not on pods(even if created after upgrade). But comments above suggest I should see managedFields in "kubectl get pods" too.
I'm not sure what the right number of sidecars might be. But 20 does sound like a lot. :)
Yes, you should see these on pods created after 1.18. (or updated, but that won't set very many fields on a pod)
I think 40k isn't that big and you'll probably be fine unless you're already at enough churn that your etcd is just barely keeping up. Good to test in your non-prod cluster first; let us know if it somehow doesn't work. Removing them cosmetically won't help a system problem. Note that we pass the existing scalability tests with this on.
In our test setup, some pods did not get the managedFields, because our MutatingWebhook was removing it. Removing the MutatingWebhook let create/preserve the managedFields. I need to figure the MutatingWebhook. Thanks for all the advise.
@Ramyak we're tracking this problem in #96688
We are working on a way to resolve this.
The first step is to reset the managedFields if the admission chain breaks them.
So if your managedFields get reset because the admission webhook does something wrong, that should get fixed.
If it resets them on purpose you'll have to check the webhook itself.
I have file #96878 to add the flag
--strip-managed-fields.
I think this is the best approach. Fine, give me managedFields, but also give me a way to toggle them off.
you have a way to toggle them off you can create a wrapper or a kubectl plugin, the main issue is the extra load and storage that comes with it, the client side is a cosmetic issue, the server side is a real issue.
extra load and storage that comes with it
The server is going to continue to store this data no matter what, because it's necessary for correctness.
If you / users are e.g. archiving objects, it is fine to remove this field pre-archival but you don't need the server to do that for you.
The only circumstance in which it's mandatory for the server to do the removal is if it is consuming too much network bandwidth. We haven't seen evidence that this is the case yet.
We haven't seen evidence that this is the case yet.
We've seen a bunch of network bandwidth problems even without having managedFields before. So I would actually suggest that we do that server-side.
We've seen a bunch of network bandwidth problems even without having managedFields before. So I would actually suggest that we do that server-side.
It would only make a big difference if controllers could turn it off, but the latest apply client design uses this information client side to support an extremely easy transition from existing controller code.
Maybe you can argue with me but IMO when people hit this code usually there's something fundamentally wrong with how some component is scaling, and 30% is not very much extra runway, and a control plane network egress bandwidth is going to catch people by surprise either way.
It would only make a big difference if controllers could turn it off, but the latest apply client design uses this information client side to support an extremely easy transition from existing controller code.
OK - that's a bit of information I was missing? Can you point me to that design?
Maybe you can argue with me but IMO when people hit this code usually there's something fundamentally wrong with how some component is scaling, and 30% is not very much extra runway, and a control plane network egress bandwidth is going to catch people by surprise either way.
I think that 30% is actually a lot and is worth optimizing. But if controllers are going to rely on it anyway, that's a convincing argument against for me.
It's very new information: kubernetes/enhancements#2419
IMO kubectl-neat does is a pretty good workaround until we have a permanent solution.
For people making heavy use of kubectl diff, I've written a custom KUBECTL_EXTERNAL_DIFF tool to apply kubectl-neat to your diff output as well: https://github.com/sh0rez/kubectl-neat-diff
/close
@soltysh: Closing this issue.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@dougsland am I right in thinking that will only affect Kubectl's display of the managed fields? We have a load test to verify apiserver performance, which uses client-go to make lots of get pods requests and I see a ~20% regression between v1.17 and v1.18 which I believe is down to the extra overhead of these fields....
@mcginne sorry the delay.
@dougsland am I right in thinking that will only affect Kubectl's display of the managed fields?
Correct.
We have a load test to verify
apiserver performance, which uses client-go to make lots ofget podsrequests and I see a ~20% regression between v1.17 and
v1.18 which I believe is down to the extra overhead of these fields....
Do you have an issue opened with more data?
I haven't raised a separate issue yet as there was some discussion on performance impact between @lavalamp and @wojtek-t here #90066 (comment), and it sounded like this overhead was necessary for correctness. I can raise a separate issue about the performance impact if folks think there is some scope for improvement..
I've installed kubectl version v1.20.5 but the annoying managedFields output is still present. When will the fix be in a stable release?
@max-rocket-internet it is targeted for 1.21, see #96878 for details.
what is the delta between this and #96878? there is a comment in the PR that says this issue still should remain open after the merge?
what else are we missing?
managed fields are still downloaded, they are stripped by the client. We could possibly omit them server-side.
Just another data point. We were seeing our mutating webhooks take up to 15s, which is crazy - should be milliseconds typically. The root cause was managedFields. Simply removing the managedFields from our patching logic dropped the slow requests to ~500ms (the bad case was a config near etcd size limits, so much slower than normal). istio/istio#34799
Just another data point. We were seeing our mutating webhooks take up to 15s, which is crazy - should be milliseconds typically. The root cause was managedFields. Simply removing the managedFields from our patching logic dropped the slow requests to ~500ms (the bad case was a config near etcd size limits, so much slower than normal). istio/istio#34799
yeah, this is the sort of things I'm really concerned about, not the output of kubectl...
But is "My patch computation is very slow because it looks at the entire object instead of just at subset that is actually relevant" really something to reasonable blame the existence of managed fields for?
Sort of? I mean it worked fine up until managedFields were added, then it (subtly) started behaving really slowly due without changes on our side. I am not here to say "managedFields are bad" or anything, just that they negatively impacted us in this niche edge case scenario, and may negatively impact others if they have similar logic. If they do, I hope they see this comment and the fix (https://github.com/istio/istio/pull/34805/files#diff-e57c2ab4e6af0f24e7285fad8ec9fe8b688fad1350b6d6c405f75a067d65e0dcR723) and can spend less time figuring out what is going on.
really something to reasonable blame the existence of managed fields for?
It is correct to blame managed fields. Any change that increases payload sizes by at least 2x and commonly 3x will directly produce performance degradation. I assume this was clear to the engineers who designed and implemented this, and that they felt the benefits outweighed the costs.
Myself, I can't really see any tangible benefit. The downsides, such poor developer and user experience, performance impact, have been reported by dozens of users and are clear.
This is pretty frustrating.
Until waiting for Kubernetes upgrade to version 1.21+, I'm using yq to delete managed fields from the output.
$ kubectl get deploy api-foo -o yaml | yq e 'del(.metadata.managedFields)' -