kubernetes / kubernetes

What happened:
API server panics handling requests for a CRD with OpenAPI validation with x-kubernetes-int-or-string

What you expected to happen:
CRD is applied successfully.

How to reproduce it (as minimally and precisely as possible):
kind create cluster --image kindest/node:v1.16.1
then kubectl apply:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: foos.stable.example.com
spec:
  group: stable.example.com
  versions:
  - name: v1
    served: true
    storage: true
  scope: Namespaced
  names:
    plural: foos
    singular: foo
    kind: Foo
  preserveUnknownFields: false
  validation:
    openAPIV3Schema:
      type: object
      properties:
        intorstr:
          x-kubernetes-int-or-string: true

Partial logs from API server:

I1011 13:08:40.273942       1 trace.go:116] Trace[1298882013]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane/log (started: 2019-10-11 13:08:27.326017611 +0000 UTC m=+218.345366948) (total time: 12.94787379s):
Trace[1298882013]: [12.947871359s] [12.945359346s] Transformed response object
E1011 13:08:43.565661       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 20926 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc00a2f06c0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:107 +0x107
panic(0x23ebe20, 0x734f680)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema.(*Structural).Unfold.func1(0xc00ea06a20, 0xc00b3054e0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/unfold.go:38 +0xa2
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema.(*Visitor).visitStructural(0xc00b305770, 0xc00ea06a20, 0xc00b305658)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/visitor.go:41 +0x48e
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema.(*Visitor).visitStructural(0xc00b305770, 0xc00ea06750, 0xc00ea06750)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/visitor.go:48 +0x173
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema.(*Visitor).Visit(...)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/visitor.go:35
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema.(*Structural).Unfold(0xc00ea06750, 0xc00ea06750)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/unfold.go:60 +0x58
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/openapi/builder.BuildSwagger(0xc00f81f600, 0xc00511e038, 0x2, 0x1010100, 0x0, 0x0, 0xd0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/openapi/builder/builder.go:105 +0x1ade
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.buildOpenAPIModelsForApply(0xc0006ee000, 0xc00f81f600, 0xc00511e038, 0x2, 0xc0011cf0c8, 0x0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:1239 +0x177
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.(*crdHandler).getOrCreateServingInfoFor(0xc0003fadc0, 0xc0021a5e60, 0x24, 0xc001095d80, 0x17, 0x0, 0x0, 0x0)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:647 +0x3f7
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.(*crdHandler).ServeHTTP(0xc0003fadc0, 0x7f482fd26ce0, 0xc00a8867b0, 0xc00b0eeb00)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:301 +0x2f1
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc007c41780, 0x7f482fd26ce0, 0xc00a8867b0, 0xc00b0eeb00)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x38d
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc0006afce0, 0x7f482fd26ce0, 0xc00a8867b0, 0xc00b0eeb00)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x85

Anything else we need to know?:

Environment: Kind

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-08T05:56:07Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g: cat /etc/os-release):

NAME="Linux Mint"
VERSION="18.3 (Sylvia)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 18.3"
VERSION_ID="18.3"
HOME_URL="http://www.linuxmint.com/"
SUPPORT_URL="http://forums.linuxmint.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/linuxmint/"
VERSION_CODENAME=sylvia
UBUNTU_CODENAME=xenial

/sig api-machinery

/cc @sttts

where

			if s.AnyOf == nil {

It seems that s.AnyOf was referencing some invalid address.

yes, which is very strange, given s was dereferenced earlier in the same method:

kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/unfold.go

Lines 28 to 38 in d647ddb

    
           if !s.XIntOrString { 
        
           	return false 
        
           } 
        
           skipAnyOf := isIntOrStringAnyOfPattern(s) 
        
           skipFirstAllOfAnyOf := isIntOrStringAllOfPattern(s) 
        
           if skipAnyOf || skipFirstAllOfAnyOf { 
        
           	return false 
        
           } 
        
           if s.AnyOf == nil {

@sttts is investigating

@sttts reproduced and has a fix

I had the same issue with prometheus (kube 1.16.1). Deploying is fine but when trying to delete, it crashes the api server.

kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup

For those who can't upgrade to 1.16.2+ (fixed version) there is a workaround : using etcd to delete the resources that crash the api server :

here I grep on coreos because I know it is prometheus that crashes the api server

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node.pem --key=/etc/ssl/etcd/ssl/node-key.pem get / --prefix --keys-only | grep coreos

Remove every item in the list that causes the crash :

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node.pem --key=/etc/ssl/etcd/ssl/node-key.pem del /registry/path-to-resource

Note this is also resolved in 1.15.5+ and 1.16.2+, upgrading to 1.17.x is not required to resolve this

	if !s.XIntOrString {
	return false
	}

	skipAnyOf := isIntOrStringAnyOfPattern(s)
	skipFirstAllOfAnyOf := isIntOrStringAllOfPattern(s)
	if skipAnyOf \|\| skipFirstAllOfAnyOf {
	return false
	}

	if s.AnyOf == nil {

x-kubernetes-int-or-string causes panic in apiserver 1.16.1