Shell-operator fails on handling something
gunmaden opened this issue · comments
Expected behavior (what you expected to happen): Periodically error occurred while shell-operator trying to handle something
Actual behavior (what actually happened): Error does not occur, have enough info about problem
Steps to reproduce:
- Deploy shell-operator & crd
Additional info:
Tried to enable debug log level but it doesn't help (only logs which entities are processing, each time error message meets in different places)
Environment:
- Shell-operator version: v1.0.10
- Kubernetes version: v1.22.12
- Installation type (kubectl apply, helm chart, etc.): helm chart
Anything else we should know?:
Hook script
#!/usr/bin/env bashset -e echoerr() { printf "ERROR. %s\n" "$*" >&2; } ##### HOOK ##### if [[ $1 == "--config" ]] ; then cat <<EOF configVersion: v1 onStartup: 1 kubernetes: - name: OnDiscoveryEvent apiVersion: custom.group/v1 kind: Discovery executeHookOnEvent: - Added - Deleted - name: OnServiceEvent apiVersion: "" kind: Service executeHookOnEvent: - Added - Modified - Deleted EOF else for context in $(jq -c '.[]' "$BINDING_CONTEXT_PATH"); do echo "Just log" done fi
Custom resource definition
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: # name must match the spec fields below, and be in the form: . name: discoveries.custom.group spec: # group name to use for REST API: /apis// group: custom.group # either Namespaced or Cluster scope: Namespaced names: # plural name to be used in the URL: /apis/// plural: discoveries # singular name to be used as an alias on the CLI and for display singular: discovery # kind is normally the CamelCased singular type. Your resource manifests use this. kind: Discovery # list of versions supported by this CustomResourceDefinition versions: - name: v1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true # The schema for validating custom objects schema: openAPIV3Schema: type: object
Logs
{"level":"info","msg":"shell-operator v1.0.10","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Listen on 0.0.0.0:9115","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_live_ticks","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_tasks_queue_action_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric gauge discovery_operator_tasks_queue_length","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric gauge discovery_operator_kube_snapshot_objects","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric gauge discovery_operator_kube_snapshot_bytes","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_kube_jq_filter_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_kube_event_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric gauge discovery_operator_hook_enable_kubernetes_bindings_seconds","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_hook_enable_kubernetes_bindings_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric gauge discovery_operator_hook_enable_kubernetes_bindings_success","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_hook_run_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_hook_run_user_cpu_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_hook_run_sys_cpu_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric gauge discovery_operator_hook_run_max_rss_bytes","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_hook_run_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_hook_run_allowed_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_hook_run_success_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_task_wait_in_queue_seconds_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Hooks dir: /hooks","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Use temporary dir: /tmp/shell-operator","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Debug endpoint listen on /var/run/shell-operator/debug.socket","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_kubernetes_client_request_latency_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric counter discovery_operator_kubernetes_client_request_result_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Kubernetes client is configured successfully with 'out-of-cluster' config","operator.component":"KubernetesAPIClient","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Kubernetes client is configured successfully with 'out-of-cluster' config","operator.component":"KubernetesAPIClient","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Initialize hooks manager. Search for and load all hooks.","time":"2022-10-05T12:14:24Z"} {"hook":"hook.sh","level":"info","msg":"Load config from '/hooks/hook.sh'","phase":"config","time":"2022-10-05T12:14:24Z"} {"hook":"hook.sh","level":"info","msg":"Loaded config: OnStartup:1, Watch k8s kinds: 'Service', 'Discovery'","phase":"config","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"start shell-operator","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"queue task HookRun::onStartup:hook.sh with hook hook.sh","operator.component":"initMainQueue","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"queue task EnableKubernetesBindings:::hook.sh:EnableKubernetesBindings for hook hook.sh","operator.component":"initMainQueue","time":"2022-10-05T12:14:24Z"} {"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"} {"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_hook_run_sys_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_hook_run_user_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"} {"binding":"","hook":"hook.sh","level":"info","msg":"Enable kubernetes binding for hook","queue":"main","task":"EnableKubernetesBindings","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"Create metric histogram discovery_operator_kubernetes_client_rate_limiter_latency_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"} {"level":"error","msg":"E1005 12:14:24.777262 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:24Z"} {"level":"info","msg":"I1005 12:14:25.959173 10 request.go:645] Throttling request took 1.076579751s, request: GET:https://10.96.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s\n","source":"klog","time":"2022-10-05T12:14:25Z"} {"level":"error","msg":"E1005 12:14:26.034804 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:26Z"} {"level":"error","msg":"E1005 12:14:28.694095 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:28Z"} {"level":"error","msg":"E1005 12:14:29.570104 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:29Z"} {"binding":"","hook":"hook.sh","level":"info","msg":"Kubernetes bindings for hook are enabled successfully, 2 tasks generated","queue":"main","task":"EnableKubernetesBindings","time":"2022-10-05T12:14:29Z"} {"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Unlock kubernetes.Event tasks","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Unlock kubernetes.Event tasks","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"} {"level":"error","msg":"E1005 12:14:30.629997 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:30Z"} {"level":"error","msg":"E1005 12:14:32.461660 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:32Z"} {"level":"error","msg":"E1005 12:14:33.203631 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:33Z"} {"level":"error","msg":"E1005 12:14:36.658349 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:36Z"} {"level":"error","msg":"E1005 12:14:40.836229 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:40Z"} {"level":"error","msg":"E1005 12:14:45.270768 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:45Z"} {"level":"error","msg":"E1005 12:15:04.555854 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:04Z"} {"level":"error","msg":"E1005 12:15:09.467184 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:09Z"} {"level":"error","msg":"E1005 12:15:36.220781 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:36Z"} {"level":"error","msg":"E1005 12:15:37.375321 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:37Z"} {"level":"error","msg":"E1005 12:16:17.142853 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:16:17Z"} {"level":"error","msg":"E1005 12:16:25.203734 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:16:25Z"} {"level":"error","msg":"E1005 12:17:06.718118 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:06Z"} {"level":"error","msg":"E1005 12:17:11.263395 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:11Z"} {"level":"error","msg":"E1005 12:17:46.645551 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:46Z"} {"level":"error","msg":"E1005 12:17:59.075634 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:59Z"} {"level":"error","msg":"E1005 12:18:35.905553 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:18:35Z"} {"level":"error","msg":"E1005 12:18:53.018460 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:18:53Z"} {"level":"error","msg":"E1005 12:19:25.314806 10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:19:25Z"}
Hello! Sorry, I've missed this one for so long. Do you have proper RBAC with rules for the Discovery kind?
I think root cause is that I've forgot to add watch
rules for resources. Thanks