"kubecfg update" fails against CRD objects: "expected kind, but got map"
seh opened this issue · comments
Observed Behavior
Using kubecfg update against a server running Kubernetes version 1.15.1 or later, attempting to update an existing CRD object (not the CRD itself, but an object of a kind defined by a CRD), kubecfg fails with the following message:
ERROR Error updating sealedsecrets my-ns.my-object: expected kind, but got map
Here we see kubecfg's attempt to update an object called "my-object" in namespace "my-ns." Note that this object's source manifest hasn't changed since the last run of kubecfg update. This message originates from api-machinery's (*util/strategicpatch.kindItem).VisitMap method.
We can create objects of kinds defined by a CRD, and we can delete them without any problems. However, we can't run kubecfg update against such an existing object.
In this case, we create the CRD object itself using kube.libsonnet's CustomResourceDefinition function:
kube.CustomResourceDefinition('bitnami.com', 'v1alpha1', 'SealedSecret')The corresponding YAML as shown by kubecfg show:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
app: sealed-secrets
name: sealedsecrets.bitnami.com
spec:
group: bitnami.com
names:
kind: SealedSecret
listKind: SealedSecretList
plural: sealedsecrets
singular: sealedsecret
scope: Namespaced
version: v1alpha1That is, we don't define its schema, in case that matters here.
Though we use this SealedSecret kind as an example here, we've experienced the same problem with similar CRDs we've defined that used to work fine with kubecfg update before Kubernetes 1.15.1 (at least with 1.14.3).
Note that pulumi/pulumi-kubernetes#630 mentions this same error, but doesn't indicate how they fixed it; it sounds like the problem just went away. We have not been so lucky with our clusters.
Software
kubecfg version
kubecfg version: v0.12.4
jsonnet version: v0.12.0
client-go version: v0.0.0-master+a27bea2b
Kubernetes version
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
I tried defining a little bit of the SealedSecret schema, and loading that into my cluster to see if that would change how kubecfg update deals with custom objects.
kube.CustomResourceDefinition('bitnami.com', 'v1alpha1', 'SealedSecret') {
spec+: {
preserveUnknownFields: true,
validation: {
openAPIV3Schema: {
type: 'object',
properties: {
spec: {
type: 'object',
properties: {
encryptedData: {
type: 'object',
},
},
},
},
},
},
},
},That didn't change the outcome. kubecfg update still fails when comparing an existing and proposed SealedSecret object.
Disabling validation by running kubecfg update with the --validate=false flag appears to work around this problem.
Disabling validation by running kubecfg update with the
--validate=falseflag appears to work around this problem.
This turns out to be incorrect. Using --validate=false usually avoids the problem, but not always.
Running with --v=8, I see that kubecfg update issues an HTTP GET request for the SealedSecret object, gets a successful response, prints the body, then prints the following (shown here with most of the object elided):
DEBUG origData: {"apiVersion":"bitnami.com/v1alpha1","kind":"SealedSecret","metadata":
// ...
}
ERROR Error updating sealedsecrets my-ns.my-object: expected kind, but got map
The failure originates in file pkg/update.go's patch function, where we call on strategicpatch.CreateThreeWayMergePatch.
@lblackstone came to similar findings in pulumi/pulumi-kubernetes#568, with code pointers here.
Per my inquiry in the "sig-api-machinery" channel in the "Kubernetes" Slack team, we can't submit a strategic merge patch for a custom resource until the server-side apply capability is available in the API server.
It sounds like rather than trying to detect which resources are custom, we should fall back from a failure to create a strategic merge patch to try creating another available patch type, like Pulumi did in pulumi/pulumi-kubernetes#622. They used JSON merge patch as the alternative. @anguslees, is it safe for us to use that patch type here?
I hacked in an adaptation of Pulumi's fix—falling back to calling jsonmergepatch.CreateThreeWayJSONMergePatch and jsonpatch.MergePatch—and it appears to work. I don't know if I'll have time to craft an acceptable PR in the next few days.
(Thanks for such a detailed report!)
kubecfg tries to do client-side strategic-merge-patches, precisely to isolate itself from sudden changes in server-side behaviour, and to be able to handle all resource types with common code. The code to do this client-side merge is relatively new to kubecfg however, and it seems there's a server-side change that triggers a bug somewhere :(
Just given the description, it's likely this is the result of kubernetes/kubernetes#79636, which shipped in k8s 1.15.1
Edit: incorrect!
It was broken by the extensive CRD schema improvements that came in with 1.15.0 - in particular kubernetes/kubernetes#79636. Some code in kube-openapi was misled by the new trivial type:object schema into thinking this portion of the schema was a Map and not a Kind, hence the eventual error message.
Fwiw, I think adding a CRD schema works around this bug.
@seh I think you tried this above and found it did not work? Could you confirm? In particular, when I tried it locally, I had to wait a few moments for the CRD schema to propagate through the apiserver. I was not able to update the schema and the CRD instance in the same kubecfg update invocation.
When I tried it, it did not work. I can probably try it again within the next day or two (being mostly away on vacation this week).
At present, most of our CRDs lack schemas, ring from projects we didn’t write. Take Sealed Secrets as an example: we want to deploy it, but we don’t have an accurate manifest available for the CRDs, so we wing it with the most minimal definition.
fwiw im seeing this with the argocd CRD in kubernetes v1.13.10